Caideyipi commented on PR #18387: URL: https://github.com/apache/iotdb/pull/18387#issuecomment-6409875846
Reviewed whether this PR fixes a real problem. **Yes: keep it open.** The focused self-review passed; no code changes were needed during this review. The production path supports the reported failure mechanism: - `FragmentInstanceManager` uses a shared notification executor with exactly four threads. - `FragmentInstanceExecution` runs driver cleanup and `releaseResourceWhenAllDriversAreClosed()` on that executor; the latter waits on `allDriversClosed`. - Before this PR, `LocalExecutionPlanner` returns its operator admission reservation through the same asynchronous state-listener path. Later fragments can already be `FAILED` while their reservation-return callbacks are queued behind earlier cleanup. Current `master` (`b652dddf41c41a86329845fac4214ec034a87575`) still has this path. I ran an isolated Java comparison using the actual state-machine and memory-block sources from each revision, plus the exact planner memory-release registration bodies extracted from those revisions. Four notification workers were held in terminal cleanup listeners with a latch, modeling the existing driver-close wait. The test admission pool was 16 KiB and each subsequent fragment reserved 4 KiB before failing. | Revision | Consecutive failed fragments completed while cleanup remained blocked | Next reservation rejected | Free admission bytes before unblocking | | --- | ---: | --- | ---: | | PR base `cec4f9b111bccf5302f7ac1e375c7401f425ef8c` | 4 | Yes | 0 | | PR head `70a43707ff2d86e917ba732bf0160cf41a03a89b` | 32 | No | 16384 | | Current `master` | 4 | Yes | 0 | After unblocking and draining the executor, all three runs returned to 16384 free bytes. This demonstrates a real delayed-reservation/admission problem, with recovery once notification work drains. The synchronous final-state listener fixes this mechanism without moving driver cleanup into the transition thread. Verification boundaries: this was a focused concurrency reproduction compiled against locally available supporting dependencies, not a full build or an end-to-end SELECT INTO test. The existing Ubuntu DataNode CI log confirms 22 passing tests across `FragmentInstanceStateMachineTest`, `FragmentInstanceContextTest`, and `LocalExecutionPlannerOperatorsMemoryTest`. The PR still has one failing Table Cluster IT job: `IoTDBTableAggregationNonStream2IT.rateFunctionsNormalTest` reports "Expected a PARTIAL aggregation in plan"; its cause was not established in this review. A useful follow-up would be to add a real four-thread executor/latch regression like this alongside the existing no-op-executor test, asserting both admission failure before the fix and successful reuse of the reservation after the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
