chrisdutz opened a new pull request, #19085:
URL: https://github.com/apache/iotdb/pull/19085
## Description
### Problem
`IoTDBMultiTAGsWithAttributesTableIT` is flaky. Usually only
`fourTableJoinTest` fails (a known issue with the small memory pool of the test
DataNode). But in some runs 9 to 15 test cases fail, all with:
```
719: There is not enough memory to execute current fragment instance,
current remaining free memory is 921B, estimated memory usage for current
fragment instance is 1960B
```
Once this starts, every later query in the class fails, even ones that need
less than 2 KB.
### Cause
`fourTableJoinTest` plans 25 fragment instances, many of them reserving
about 1 MB of operator memory, and the operator pool of the test DataNode has
about 15 MB. Some instances fail in `LocalExecutionPlanner.checkMemory()`, so
the dispatch fails and `ClusterScheduler` fails the query.
The instances that were dispatched before the failure are cancelled by
`SimpleQueryTerminator`, but only after `TERMINATION_GRACE_PERIOD_IN_MS` (1 s).
Depending on timing, some of them have already lost their consumer and can't
finish on their own. They keep their reservation until the cancel arrives.
With DEBUG logging for `LocalExecutionPlanner` enabled, a failing run shows
it:
- `fourTableJoinTest` fails at 15:46:19,687. Six of its instances (`1.6`,
`1.8`, `1.9`, `1.12`, `1.14`, `1.20`) log `[ConsumeMemory]` but never
`[ReleaseMemory]`, together about 4.2 MB.
- The next test queries arrive within milliseconds. They fail their dispatch
the same way and each leaves its own instances holding memory for another
second, so free memory drops to 921 B.
- The class ends at 15:46:20,351, before the first deferred cancel would
have run.
### Fix
`ClusterScheduler` remembers that the dispatch failed, and `stop()` then
calls the new `IQueryTerminator.terminateImmediately()`, which cancels the
dispatched instances without the grace period.
The grace period was added in #5808 so that instances flushing their last
results to the consumer aren't cancelled while they finish normally. A query
whose dispatch failed has no results to deliver, so it doesn't need it. Queries
that were dispatched successfully keep the grace period.
### Verification
- New `SimpleQueryTerminatorTest` checks the delay of both methods.
`SimpleQueryTerminatorTest`, `QueryExecutionRetryTest`, `QueryOwnershipTest`
and `AsyncPlanNodeSenderTest` pass (17 tests).
- `spotless:check` passes for `iotdb-core/datanode`.
- `IoTDBMultiTAGsWithAttributesTableIT` alone, 10 runs each on macOS, on
this branch's base and with this change:
| Failing test cases per run | Base | This change |
|---|---|---|
| 1 (`fourTableJoinTest` only) | 5 | 10 |
| 2 (`fourTableJoinTest`, `timeColumnOuterJoinTest1`) | 1 | 0 |
| 14 or 15 (cascade) | 4 | 0 |
`fourTableJoinTest` itself still fails, because its 25 instances don't fit
into the memory pool of the test DataNode. That's a separate issue and
unchanged by this PR.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
-
`iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/ClusterScheduler.java`
-
`iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/IQueryTerminator.java`
-
`iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/SimpleQueryTerminator.java`
-
`iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/scheduler/SimpleQueryTerminatorTest.java`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]