JackieTien97 opened a new pull request, #19330:
URL: https://github.com/apache/iotdb/pull/19330
## Description
Two fixes for query executions that stay registered in the `Coordinator`
after nobody can use them any more.
### 1. Abandoned queries were never reclaimed
**Scenario.** A client runs a query and then neither fetches the rest of the
result nor closes the statement, and no fragment instance of the query is left
to fail it. For example, the query is sent to one DataNode, its data is on
another one, and the rest of the result is already buffered on the DataNode
that received the query.
**Wrong behaviour.** `Coordinator.cleanUpStaleQueries` runs every 2 s to
reclaim the queries that exceeded their timeout by more than 60 s. It skipped
every query for which `IQueryExecution.isActive()` returned true. `isActive()`
is documented as "there is an active RPC for this query", but `QueryExecution`
and `ConfigExecution` returned `startTimeOfCurrentRpc == -1`, which is true
when **no** RPC is in progress. The cleanup therefore skipped exactly the
abandoned queries, and they stayed registered until the statement or the
session was closed.
**Corrected behaviour.**
- `cleanUpStaleQueries` no longer depends on `isActive()`. Every query that
exceeded its timeout by more than 60 s is reclaimed: it is removed from the
`Coordinator` and cleaned up with a `QueryTimeoutRuntimeException`.
- `isActive()` returns true iff an RPC of the query is in progress, as
documented. `QueryExecution.cleanUpCoordinatorContextMapIfNeeded` had been
written against the inverted value, so its check is negated and its behaviour
does not change.
The alternative is to keep the check and let the cleanup skip the queries
with an RPC in progress, which is what the `isActive()` check was meant to do.
I did not do that, because such queries are reclaimed today and that is relied
on:
- an RPC that is still in progress 60 s after the timeout is stuck, e.g. in
a dispatch that does not return, and stopping the query is what unblocks it
(`QueryTerminalCleanupTest.staleQueryCleanupDuringDispatchMustUnblockGetStatus`);
- a caller that never reports the end of its RPC through
`Coordinator.recordExecutionTime` would keep its query forever. The two
handlers of the second fix were such callers.
So what was reclaimed before is still reclaimed, and the queries without an
RPC in progress are reclaimed in addition.
**Example.** 1 ConfigNode and 2 DataNodes. `select s1 from root.remote.d1`
(20000 rows stored on DataNode 2) is sent to DataNode 1 with `timeout=2000`;
the client reads the first batch and then stays idle.
| | before | after |
| --- | --- | --- |
| the query in the `Coordinator` of DataNode 1 | still registered 90 s later
| removed 63.5 s after it started (`Cleaning up stale query ... timeout
duration is: 2000ms`) |
| the client continues reading after 90 s | reads all 20000 rows | `715:
Query is not found, it may be killed by others, timeout or some other runtime
errors, ...` |
This is visible to a client that pauses for longer than the timeout plus 60
s in the middle of a result: its next fetch now fails, while it used to succeed
as long as the rest of the result was already buffered. (The same query on data
of DataNode 1, whose fragment instance was still waiting to send its result,
was released within 10 s before and after this change.)
### 2. Query statements of a batch were not run to their end, and not
released
**Scenario.** A statement that is a query is sent through
`executeBatchStatement` (JDBC `Statement.executeBatch()`), for example `select
s1 into root.bdst.d1(s1) from root.bsrc.d1` in the tree model or `INSERT INTO
dst SELECT * FROM src` in the table model.
**Wrong behaviour.** The handler only started such a statement and returned
its status. A batch has no result set, so nobody fetched the result of the
statement, and a query only makes progress while its result is consumed. The
statement was therefore not run to its end, the batch still reported success,
and the execution stayed registered (and listed by `SHOW QUERIES`) until the
stale query cleanup removed it 60 s after its timeout.
**Corrected behaviour.** The handler consumes the result of such a statement
and then releases its execution. The statement has been run to its end when the
batch returns, a failure while it runs is reported in the status of that
statement, and nothing stays registered.
`querySchemaTemplate` (e.g. `Session.showMeasurementsInTemplate`) had the
second half of the problem: it returns its whole result in the response, but
left the execution registered until the stale query cleanup removed it. It now
releases the execution when it returns.
**Example.** Same cluster, source tables and series with 2000 rows,
`query_timeout_threshold=5000`.
| statement sent in a batch | before | after |
| --- | --- | --- |
| tree `SELECT ... INTO`, source data on the DataNode that receives the
batch | status 200, 0 rows written, the query is aborted at its timeout |
status 200, 2000 rows written when the batch returns |
| tree `SELECT ... INTO`, source data on the other DataNode | status 200, 0
rows when the batch returns, 2000 rows about 0.5 s later | status 200, 2000
rows written when the batch returns |
| table `INSERT ... SELECT`, both placements | as the two rows above |
status 200, 2000 rows written when the batch returns |
| any of them, and a plain `SELECT` | execution still registered after the
batch returned, until timeout + 60 s (65-66 s measured for a plain `SELECT`) |
no execution registered when the batch returns |
| `Session.showMeasurementsInTemplate` | execution registered for 65-66 s |
no execution registered when the request returns |
With several data regions the statement could also be executed partly: the
new `IoTDBInsertQueryIT.testInsertQueryInBatch` found 200 of 1200 rows before
the fix.
One consequence to be aware of: a plain `SELECT` in a batch is now executed
to its end before the batch returns, and its result is discarded. It used to be
started and then left alone. The time this takes is bounded by
`query_timeout_threshold`.
### Tests
- Unit tests for the cleanup: `CoordinatorTest` (stale queries with and
without an RPC in progress, inside the grace period, without a positive
timeout, and a killed query), `ConfigExecutionTest` and
`QueryExecutionLifecycleTest` (`isActive()`). On the main code of master, 3 of
them fail: `testStaleQueryWithoutRpcInProgressIsCleanedUp` (`expected null, but
was:<QueryExecution[...]>`) and the two
`isActiveTellsWhetherAnRpcOfTheQueryIsInProgress`.
- Integration tests, all failing before the second fix:
- `IoTDBShowQueriesIT.queryInBatchReleaseTest` and
`schemaTemplateQueryReleaseTest`: `expected:<[]> but
was:<[20261119_122523_00011_1]>` (the id of the query that was left registered)
- `IoTDBExecuteBatchIT.testJDBCExecuteBatchWithSelectInto`: `expected:<2>
but was:<0>`
- `IoTDBInsertQueryIT.testInsertQueryInBatch`: `expected:<1200> but
was:<200>`
With the fix, `IoTDBShowQueriesIT` and `IoTDBExecuteBatchIT` pass with the
default `SimpleIT` profile and with `-PClusterIT`, and `IoTDBInsertQueryIT`
with `-PTableSimpleIT` and `-PTableClusterIT`.
- The numbers in the two tables were measured on a 1C2D cluster built from
master and from this branch.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious
for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold
for code coverage.
- [x] added integration tests.
- [x] been tested in a test IoTDB cluster.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `iotdb-core/datanode`: `org.apache.iotdb.db.queryengine.plan.Coordinator`,
`org.apache.iotdb.db.queryengine.plan.execution.QueryExecution`,
`org.apache.iotdb.db.queryengine.plan.execution.config.ConfigExecution`,
`org.apache.iotdb.db.protocol.thrift.impl.ClientRPCServiceImpl`
- `iotdb-core/datanode` (test): `CoordinatorTest`, `ConfigExecutionTest`,
`QueryExecutionLifecycleTest`
- `integration-test`: `IoTDBShowQueriesIT`, `IoTDBExecuteBatchIT`,
`IoTDBInsertQueryIT`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]