JackieTien97 opened a new pull request, #19330:
URL: https://github.com/apache/iotdb/pull/19330

   ## Description
   
   Two fixes for query executions that stay registered in the `Coordinator` 
after nobody can use them any more.
   
   ### 1. Abandoned queries were never reclaimed
   
   **Scenario.** A client runs a query and then neither fetches the rest of the 
result nor closes the statement, and no fragment instance of the query is left 
to fail it. For example, the query is sent to one DataNode, its data is on 
another one, and the rest of the result is already buffered on the DataNode 
that received the query.
   
   **Wrong behaviour.** `Coordinator.cleanUpStaleQueries` runs every 2 s to 
reclaim the queries that exceeded their timeout by more than 60 s. It skipped 
every query for which `IQueryExecution.isActive()` returned true. `isActive()` 
is documented as "there is an active RPC for this query", but `QueryExecution` 
and `ConfigExecution` returned `startTimeOfCurrentRpc == -1`, which is true 
when **no** RPC is in progress. The cleanup therefore skipped exactly the 
abandoned queries, and they stayed registered until the statement or the 
session was closed.
   
   **Corrected behaviour.**
   
   - `cleanUpStaleQueries` no longer depends on `isActive()`. Every query that 
exceeded its timeout by more than 60 s is reclaimed: it is removed from the 
`Coordinator` and cleaned up with a `QueryTimeoutRuntimeException`.
   - `isActive()` returns true iff an RPC of the query is in progress, as 
documented. `QueryExecution.cleanUpCoordinatorContextMapIfNeeded` had been 
written against the inverted value, so its check is negated and its behaviour 
does not change.
   
   The alternative is to keep the check and let the cleanup skip the queries 
with an RPC in progress, which is what the `isActive()` check was meant to do. 
I did not do that, because such queries are reclaimed today and that is relied 
on:
   
   - an RPC that is still in progress 60 s after the timeout is stuck, e.g. in 
a dispatch that does not return, and stopping the query is what unblocks it 
(`QueryTerminalCleanupTest.staleQueryCleanupDuringDispatchMustUnblockGetStatus`);
   - a caller that never reports the end of its RPC through 
`Coordinator.recordExecutionTime` would keep its query forever. The two 
handlers of the second fix were such callers.
   
   So what was reclaimed before is still reclaimed, and the queries without an 
RPC in progress are reclaimed in addition.
   
   **Example.** 1 ConfigNode and 2 DataNodes. `select s1 from root.remote.d1` 
(20000 rows stored on DataNode 2) is sent to DataNode 1 with `timeout=2000`; 
the client reads the first batch and then stays idle.
   
   | | before | after |
   | --- | --- | --- |
   | the query in the `Coordinator` of DataNode 1 | still registered 90 s later 
| removed 63.5 s after it started (`Cleaning up stale query ... timeout 
duration is: 2000ms`) |
   | the client continues reading after 90 s | reads all 20000 rows | `715: 
Query is not found, it may be killed by others, timeout or some other runtime 
errors, ...` |
   
   This is visible to a client that pauses for longer than the timeout plus 60 
s in the middle of a result: its next fetch now fails, while it used to succeed 
as long as the rest of the result was already buffered. (The same query on data 
of DataNode 1, whose fragment instance was still waiting to send its result, 
was released within 10 s before and after this change.)
   
   ### 2. Query statements of a batch were not run to their end, and not 
released
   
   **Scenario.** A statement that is a query is sent through 
`executeBatchStatement` (JDBC `Statement.executeBatch()`), for example `select 
s1 into root.bdst.d1(s1) from root.bsrc.d1` in the tree model or `INSERT INTO 
dst SELECT * FROM src` in the table model.
   
   **Wrong behaviour.** The handler only started such a statement and returned 
its status. A batch has no result set, so nobody fetched the result of the 
statement, and a query only makes progress while its result is consumed. The 
statement was therefore not run to its end, the batch still reported success, 
and the execution stayed registered (and listed by `SHOW QUERIES`) until the 
stale query cleanup removed it 60 s after its timeout.
   
   **Corrected behaviour.** The handler consumes the result of such a statement 
and then releases its execution. The statement has been run to its end when the 
batch returns, a failure while it runs is reported in the status of that 
statement, and nothing stays registered.
   
   `querySchemaTemplate` (e.g. `Session.showMeasurementsInTemplate`) had the 
second half of the problem: it returns its whole result in the response, but 
left the execution registered until the stale query cleanup removed it. It now 
releases the execution when it returns.
   
   **Example.** Same cluster, source tables and series with 2000 rows, 
`query_timeout_threshold=5000`.
   
   | statement sent in a batch | before | after |
   | --- | --- | --- |
   | tree `SELECT ... INTO`, source data on the DataNode that receives the 
batch | status 200, 0 rows written, the query is aborted at its timeout | 
status 200, 2000 rows written when the batch returns |
   | tree `SELECT ... INTO`, source data on the other DataNode | status 200, 0 
rows when the batch returns, 2000 rows about 0.5 s later | status 200, 2000 
rows written when the batch returns |
   | table `INSERT ... SELECT`, both placements | as the two rows above | 
status 200, 2000 rows written when the batch returns |
   | any of them, and a plain `SELECT` | execution still registered after the 
batch returned, until timeout + 60 s (65-66 s measured for a plain `SELECT`) | 
no execution registered when the batch returns |
   | `Session.showMeasurementsInTemplate` | execution registered for 65-66 s | 
no execution registered when the request returns |
   
   With several data regions the statement could also be executed partly: the 
new `IoTDBInsertQueryIT.testInsertQueryInBatch` found 200 of 1200 rows before 
the fix.
   
   One consequence to be aware of: a plain `SELECT` in a batch is now executed 
to its end before the batch returns, and its result is discarded. It used to be 
started and then left alone. The time this takes is bounded by 
`query_timeout_threshold`.
   
   ### Tests
   
   - Unit tests for the cleanup: `CoordinatorTest` (stale queries with and 
without an RPC in progress, inside the grace period, without a positive 
timeout, and a killed query), `ConfigExecutionTest` and 
`QueryExecutionLifecycleTest` (`isActive()`). On the main code of master, 3 of 
them fail: `testStaleQueryWithoutRpcInProgressIsCleanedUp` (`expected null, but 
was:<QueryExecution[...]>`) and the two 
`isActiveTellsWhetherAnRpcOfTheQueryIsInProgress`.
   - Integration tests, all failing before the second fix:
     - `IoTDBShowQueriesIT.queryInBatchReleaseTest` and 
`schemaTemplateQueryReleaseTest`: `expected:<[]> but 
was:<[20261119_122523_00011_1]>` (the id of the query that was left registered)
     - `IoTDBExecuteBatchIT.testJDBCExecuteBatchWithSelectInto`: `expected:<2> 
but was:<0>`
     - `IoTDBInsertQueryIT.testInsertQueryInBatch`: `expected:<1200> but 
was:<200>`
   
     With the fix, `IoTDBShowQueriesIT` and `IoTDBExecuteBatchIT` pass with the 
default `SimpleIT` profile and with `-PClusterIT`, and `IoTDBInsertQueryIT` 
with `-PTableSimpleIT` and `-PTableClusterIT`.
   - The numbers in the two tables were measured on a 1C2D cluster built from 
master and from this branch.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious
     for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, 
ensuring the threshold
     for code coverage.
   - [x] added integration tests.
   - [x] been tested in a test IoTDB cluster.
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - `iotdb-core/datanode`: `org.apache.iotdb.db.queryengine.plan.Coordinator`, 
`org.apache.iotdb.db.queryengine.plan.execution.QueryExecution`, 
`org.apache.iotdb.db.queryengine.plan.execution.config.ConfigExecution`, 
`org.apache.iotdb.db.protocol.thrift.impl.ClientRPCServiceImpl`
   - `iotdb-core/datanode` (test): `CoordinatorTest`, `ConfigExecutionTest`, 
`QueryExecutionLifecycleTest`
   - `integration-test`: `IoTDBShowQueriesIT`, `IoTDBExecuteBatchIT`, 
`IoTDBInsertQueryIT`
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to