chrisdutz opened a new pull request, #19086:
URL: https://github.com/apache/iotdb/pull/19086

   ## Description
   
   ### Problem
   
   On machines with many CPU cores, 
`IoTDBMultiTAGsWithAttributesTableIT.fourTableJoinTest` fails every time:
   
   ```
   719: There is not enough memory to execute current fragment instance, 
current remaining free memory is 155701B, estimated memory usage for current 
fragment instance is 1058088B
   ```
   
   It passes on CI.
   
   ### Cause
   
   The test runs with the default `AUTO` DataRegionGroup extension policy, so 
the number of DataRegionGroups follows the CPU cores of the DataNodes 
(`data_region_per_data_node_proportion = 0.5`). The fragment instances of the 
four-table join grow with it, and many of them reserve about 1 MB of operator 
memory because the test sets a 1 MB sort buffer. The DataNode heap stays at 256 
MB, which gives about 16 MB of operator memory:
   
   | Cores | DataRegionGroups | Fragment instances of `fourTableJoinTest` | 
Fits into 16 MB |
   |---|---|---|---|
   | 4 (CI runner) | 2 | few | yes |
   | 12 (my machine) | 6 | 25 | no, needs about 21 MB |
   
   Limiting the DataNode JVMs to 4 cores (`-XX:ActiveProcessorCount=4`) on the 
same machine gives 2 regions, and all 34 test cases pass.
   
   ### Fix
   
   Scale the DataNode heap with the expected number of DataRegions instead of 
limiting the regions, so that a machine with more cores still runs the test 
with all of them: 128 MB per region (`availableProcessors() / 2`), but never 
less than the configured heap (`DataNodeMaxHeapSize`, default 256 MB). The test 
needs about 3.5 MB of operator memory per region, so this leaves a margin of 
about 2x.
   
   On a 4-core machine the result is the default 256 MB, so CI runs with the 
same settings as before.
   
   ### Verification
   
   `IoTDBMultiTAGsWithAttributesTableIT` alone, on macOS with 12 cores:
   
   | Setup | DataNode heap | Regions | Result |
   |---|---|---|---|
   | master | 256 MB | 6 | `fourTableJoinTest` fails in every run |
   | this change, `TableSimpleIT`, 5 runs | 768 MB | 6 | 34 of 34 pass in every 
run |
   | this change, `TableSimpleIT`, JVMs limited to 4 cores | 256 MB | 2 | 34 of 
34 pass |
   | this change, `TableClusterIT` (1C3D) | 768 MB | 18 in total | 34 of 34 
pass |
   
   `spotless:check` passes for `integration-test`.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious for an unfamiliar reader.
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - 
`integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBMultiTAGsWithAttributesTableIT.java`
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to