chrisdutz opened a new pull request, #19085:
URL: https://github.com/apache/iotdb/pull/19085

   ## Description
   
   ### Problem
   
   `IoTDBMultiTAGsWithAttributesTableIT` is flaky. Usually only 
`fourTableJoinTest` fails (a known issue with the small memory pool of the test 
DataNode). But in some runs 9 to 15 test cases fail, all with:
   
   ```
   719: There is not enough memory to execute current fragment instance, 
current remaining free memory is 921B, estimated memory usage for current 
fragment instance is 1960B
   ```
   
   Once this starts, every later query in the class fails, even ones that need 
less than 2 KB.
   
   ### Cause
   
   `fourTableJoinTest` plans 25 fragment instances, many of them reserving 
about 1 MB of operator memory, and the operator pool of the test DataNode has 
about 15 MB. Some instances fail in `LocalExecutionPlanner.checkMemory()`, so 
the dispatch fails and `ClusterScheduler` fails the query.
   
   The instances that were dispatched before the failure are cancelled by 
`SimpleQueryTerminator`, but only after `TERMINATION_GRACE_PERIOD_IN_MS` (1 s). 
Depending on timing, some of them have already lost their consumer and can't 
finish on their own. They keep their reservation until the cancel arrives.
   
   With DEBUG logging for `LocalExecutionPlanner` enabled, a failing run shows 
it:
   
   - `fourTableJoinTest` fails at 15:46:19,687. Six of its instances (`1.6`, 
`1.8`, `1.9`, `1.12`, `1.14`, `1.20`) log `[ConsumeMemory]` but never 
`[ReleaseMemory]`, together about 4.2 MB.
   - The next test queries arrive within milliseconds. They fail their dispatch 
the same way and each leaves its own instances holding memory for another 
second, so free memory drops to 921 B.
   - The class ends at 15:46:20,351, before the first deferred cancel would 
have run.
   
   ### Fix
   
   `ClusterScheduler` remembers that the dispatch failed, and `stop()` then 
calls the new `IQueryTerminator.terminateImmediately()`, which cancels the 
dispatched instances without the grace period.
   
   The grace period was added in #5808 so that instances flushing their last 
results to the consumer aren't cancelled while they finish normally. A query 
whose dispatch failed has no results to deliver, so it doesn't need it. Queries 
that were dispatched successfully keep the grace period.
   
   ### Verification
   
   - New `SimpleQueryTerminatorTest` checks the delay of both methods. 
`SimpleQueryTerminatorTest`, `QueryExecutionRetryTest`, `QueryOwnershipTest` 
and `AsyncPlanNodeSenderTest` pass (17 tests).
   - `spotless:check` passes for `iotdb-core/datanode`.
   - `IoTDBMultiTAGsWithAttributesTableIT` alone, 10 runs each on macOS, on 
this branch's base and with this change:
   
     | Failing test cases per run | Base | This change |
     |---|---|---|
     | 1 (`fourTableJoinTest` only) | 5 | 10 |
     | 2 (`fourTableJoinTest`, `timeColumnOuterJoinTest1`) | 1 | 0 |
     | 14 or 15 (cascade) | 4 | 0 |
   
     `fourTableJoinTest` itself still fails, because its 25 instances don't fit 
into the memory pool of the test DataNode. That's a separate issue and 
unchanged by this PR.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths.
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - 
`iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/ClusterScheduler.java`
   - 
`iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/IQueryTerminator.java`
   - 
`iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/SimpleQueryTerminator.java`
   - 
`iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/scheduler/SimpleQueryTerminatorTest.java`
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to