chrisdutz opened a new pull request, #19086: URL: https://github.com/apache/iotdb/pull/19086
## Description ### Problem On machines with many CPU cores, `IoTDBMultiTAGsWithAttributesTableIT.fourTableJoinTest` fails every time: ``` 719: There is not enough memory to execute current fragment instance, current remaining free memory is 155701B, estimated memory usage for current fragment instance is 1058088B ``` It passes on CI. ### Cause The test runs with the default `AUTO` DataRegionGroup extension policy, so the number of DataRegionGroups follows the CPU cores of the DataNodes (`data_region_per_data_node_proportion = 0.5`). The fragment instances of the four-table join grow with it, and many of them reserve about 1 MB of operator memory because the test sets a 1 MB sort buffer. The DataNode heap stays at 256 MB, which gives about 16 MB of operator memory: | Cores | DataRegionGroups | Fragment instances of `fourTableJoinTest` | Fits into 16 MB | |---|---|---|---| | 4 (CI runner) | 2 | few | yes | | 12 (my machine) | 6 | 25 | no, needs about 21 MB | Limiting the DataNode JVMs to 4 cores (`-XX:ActiveProcessorCount=4`) on the same machine gives 2 regions, and all 34 test cases pass. ### Fix Scale the DataNode heap with the expected number of DataRegions instead of limiting the regions, so that a machine with more cores still runs the test with all of them: 128 MB per region (`availableProcessors() / 2`), but never less than the configured heap (`DataNodeMaxHeapSize`, default 256 MB). The test needs about 3.5 MB of operator memory per region, so this leaves a margin of about 2x. On a 4-core machine the result is the default 256 MB, so CI runs with the same settings as before. ### Verification `IoTDBMultiTAGsWithAttributesTableIT` alone, on macOS with 12 cores: | Setup | DataNode heap | Regions | Result | |---|---|---|---| | master | 256 MB | 6 | `fourTableJoinTest` fails in every run | | this change, `TableSimpleIT`, 5 runs | 768 MB | 6 | 34 of 34 pass in every run | | this change, `TableSimpleIT`, JVMs limited to 4 cores | 256 MB | 2 | 34 of 34 pass | | this change, `TableClusterIT` (1C3D) | 768 MB | 18 in total | 34 of 34 pass | `spotless:check` passes for `integration-test`. <hr> This PR has: - [x] been self-reviewed. - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. <hr> ##### Key changed/added classes (or packages if there are too many classes) in this PR - `integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBMultiTAGsWithAttributesTableIT.java` 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
