PDGGK opened a new pull request, #19084:
URL: https://github.com/apache/iotdb/pull/19084

   ## Description
   
   A fragment instance (FI) of a query that has already succeeded can stay in 
`FLUSHING` when its consumer closes the local channel first. It keeps its 
admission memory until `SimpleQueryTerminator` finishes it about one second 
later, so a query sent in that second can be rejected with `719: There is not 
enough memory to execute current fragment instance`.
   
   I found this through the intermittent 719s of `IoTDBLTTBTableFunctionIT` in 
#18725. With this one-line change on that PR's head, the class passed 8 runs in 
a row with `-PTableSimpleIT`, against 9 failing runs out of 12 without it. Its 
`-PTableClusterIT` failures have a second cause, which this change does not 
address.
   
   To check that it does not depend on that PR, I reproduced it on master 
`999721fcaf` with a local IT (not part of this PR), without LTTB:
   
   - one DataNode (operator pool 16,090,037 B), 4 data region groups, 
`sort_buffer_size_in_bytes = 3980000`;
   - two queries alternate 300 times, each on a fresh connection that reads its 
whole result:
     - `SELECT part, count(s1) AS c, max(s1) AS m FROM regions GROUP BY part 
ORDER BY c DESC, part`, whose root FI reserves 4,001,832 B;
     - `SELECT time, part, s1 FROM regions ORDER BY part, time`, whose four 
region FIs and root need 15,950,312 B, so it is rejected only if something else 
still holds more than 139,725 B.
   
   | | runs | runs with a 719 |
   |---|---|---|
   | master | 5 | 5 |
   | this change | 5 | 0 (1,500 of 1,500 queries succeed) |
   
   At the first 719 of every master run, the memory held by other queries was 
the 4,001,832 B root FI of the preceding, already completed query, plus at most 
9,352 B. (After that first rejection, master kept rejecting queries until the 
run ended, 89 to 295 of 300 per run; I have not looked into that part.)
   
   ### Cause
   
   1. The consumer closes first: when a finished query's result handle is 
closed, or when a `LIMIT` stops reading (`ExchangeOperator.java:144`). 
`LocalSourceHandle.close()` → `SharedTsBlockQueue.close()` → 
`LocalSinkChannel.close()` (`SharedTsBlockQueue.java:370`).
   2. `LocalSinkChannel.close()` sets `closed = true` and then calls its 
listener (`LocalSinkChannel.java:230`, `:233`).
   3. The listener calls `ShuffleSinkHandle.close()` 
(`MPPDataExchangeManager.java:629`, `:635`), which calls `channel.close()` 
again. That call returns `false` because the channel is already closed 
(`LocalSinkChannel.java:228`), so the handle skips 
`sinkListener.onFinish(this)` (`ShuffleSinkHandle.java:280`, `:291`-`:292`).
   4. `context.finished()` (`MPPDataExchangeManager.java:529`) is not called, 
and the FI stays in `FLUSHING` until `SimpleQueryTerminator`, after its 1000 ms 
grace period, calls `FragmentInstanceManager.cancelTask`, which calls 
`context.finished()` for a query that did not fail 
(`SimpleQueryTerminator.java:96`-`:97`, `:115`; 
`FragmentInstanceManager.java:411`-`:414`).
   
   The remote `SinkChannel.close()` calls `invokeOnFinished()` before it sets 
`closed = true` (`SinkChannel.java:366`-`:367`), so on that path the handle 
does finish.
   
   ### Change
   
   ```java
   selfClosed = channel.close() || (channel.isClosed() && !channel.isAborted());
   ```
   
   A channel that is already closed and not aborted now counts as closed; an 
aborted channel still does not. The FI then reaches `FINISHED` through the same 
`context.finished()` call that the terminator makes a second later. When the 
consumer stops early, as with a `LIMIT`, the FI now finishes at that point, as 
it already does on the remote path: the producer's later `send()` and 
`setNoMoreTsBlocks()` return immediately, and its driver ends at the next 
`sink.isClosed()` check (`Driver.java:224`).
   
   With several channels, `selfClosed` still takes the result of the last 
channel, as before; I left that unchanged. A mix of aborted and consumer-closed 
channels does not reach this `close()` through the channel listener, which 
skips it once any channel has aborted (`MPPDataExchangeManager.java:631`).
   
   ### Verification
   
   - **New tests in `MPPDataExchangeManagerTest`**, with the queue, channels, 
source handle and `ShuffleSinkHandle` created through `MPPDataExchangeManager` 
and only `FragmentInstanceContext` mocked:
     - the consumer closing right after no-more is set (forced with a direct 
listener on `isBlocked()`, no sleeps);
     - the consumer closing before no-more;
     - two channels, both closed by their consumers;
     - the remote consumer closing first;
     - an aborted channel, then `close()` on the handle;
     - an aborted channel next to a consumer-closed one, where the handle must 
stay open.
   - The three local consumer-close tests fail on master with 
`ShuffleSinkHandle did not finish after the consumer closed ...`; the other 
three pass on master too and guard the existing behaviour. All nine tests in 
the class pass with the change. Without `!channel.isAborted()`, 
`testAbortedLocalChannelKeepsShuffleSinkHandleFromFinishing` fails.
   - All test classes under `queryengine/execution/exchange` and 
`queryengine/execution/fragment` pass (20 classes, 84 tests), and so do 
`spotless:check` and the `-P with-zh-locale` compile.
   - ITs with the change, all passing:
     - `-PTableSimpleIT` (140 tests, 33 of them `@Ignore`d in 
`IoTDBOrderByTableIT`): `IoTDBCollectFanInIT`, `IoTDBTableAggregationIT`, 
`IoTDBOrderByTableIT`, `IoTDBOrderByLimitOffsetAlignByDeviceTableIT`, 
`IoTDBAlignedOffsetLimitPushDownTableIT`;
     - `-PSimpleIT` (96 tests): `IoTDBShuffleSink1IT`, `IoTDBShuffleSink2IT`, 
`IoTDBOrderByIT`, `IoTDBOrderByLimitOffsetAlignByDeviceIT`, 
`IoTDBAggregationIT`;
     - `-PTableClusterIT` (70 tests): `IoTDBCollectFanInIT`, 
`IoTDBTableAggregationIT`, `IoTDBOrderByLimitOffsetAlignByDeviceTableIT`.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added unit tests or modified existing tests to cover new code paths, 
ensuring the threshold for code coverage.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to