JackieTien97 opened a new pull request, #19334:
URL: https://github.com/apache/iotdb/pull/19334
## Description
Backport of #19330 to `dev/1.3`. It contains two fixes for query executions
that stay registered in the `Coordinator` after nobody can use them any more.
### 1. Abandoned queries were never reclaimed
A client runs a query and then neither fetches the rest of the result nor
closes the statement, and no fragment instance of the query is left to fail it.
`Coordinator.cleanUpStaleQueries` should reclaim such a query 60 s after its
timeout, but it skipped every query for which `IQueryExecution.isActive()`
returned true, and `isActive()` returned `startTimeOfCurrentRpc == -1`, which
is true when **no** RPC is in progress. The cleanup therefore skipped exactly
the abandoned queries.
- `cleanUpStaleQueries` no longer depends on `isActive()`: every query that
exceeded its timeout by more than 60 s is reclaimed.
- `isActive()` returns true iff an RPC of the query is in progress, as
documented. `QueryExecution.cleanUpCoordinatorContextMapIfNeeded` had been
written against the inverted value, so its check is negated and its behaviour
does not change.
What was reclaimed before is still reclaimed, and the queries without an RPC
in progress are reclaimed in addition. #19330 explains why the cleanup does not
start to skip the queries with an RPC in progress, and shows the effect on a
two-DataNode cluster: the query was still registered and fully readable after
90 s, and is now removed 63.5 s after it started with `timeout=2000`.
### 2. Query statements of a batch were not run to their end, and not
released
A statement that is a query, such as `select s1 into root.db.dst(s1) from
root.db.src`, is sent through `executeBatchStatement` (JDBC
`Statement.executeBatch()`). The handler only started it and returned its
status. A batch has no result set, so nobody fetched the result, and a query
only makes progress while its result is consumed. The statement was not run to
its end, the batch still reported success, and the execution stayed registered
until the stale query cleanup removed it.
The handler now consumes the result of such a statement and then releases
its execution. `querySchemaTemplate` (e.g.
`Session.showMeasurementsInTemplate`) returns its whole result in the response
and now releases its execution when it returns.
One consequence to be aware of: a plain `SELECT` in a batch is now executed
to its end before the batch returns, and its result is discarded. The time this
takes is bounded by `query_timeout_threshold`.
### Source and target
- Source: #19330, merged into `master` as
`90e3d967f9799086df50fd33084452b9268f25c0`. That commit is the squash of the
two commits of the PR, `78bf43b042f34433cfa48fbc47a8707dd04586b2` and
`9c3af0cfd3bcca347bba24a550a00a6a57372624`, and has the same tree as the second
one. The two commits were cherry-picked with `-x`, so the two fixes stay
separate here.
- Target base: `dev/1.3` at `86f9335302`.
### Adaptations
Main code:
- `QueryExecution.cleanUpCoordinatorContextMapIfNeeded` keeps the
`cleanupQueryExecution(queryId, null, t)` call of `dev/1.3`; only its condition
is negated, as in the source.
- `ClientRPCServiceImpl`: the surrounding code of both handlers differs on
`dev/1.3` (no table model branch, the query id is requested at one place), so
the changes were re-applied by hand on the `dev/1.3` file: the `consumeResult`
helper, consuming the result and releasing the execution per statement in
`executeBatchStatement`, and releasing the execution in
`executeTemplateQueryStatement`.
Tests:
- `CoordinatorTest`, `QueryExecutionLifecycleTest` and `IoTDBShowQueriesIT`
do not exist on `dev/1.3`. They are added with the tests of this change only,
and `IoTDBShowQueriesIT` with the small fixture these tests need. They are
adapted to the `Coordinator.execution` and `SessionInfo` signatures, the
`QueryType` values and the `ColumnHeaderConstant` package of `dev/1.3`.
- `IoTDBExecuteBatchIT` starts a cluster per test on `dev/1.3`, so the new
test has no cleanup block.
Not ported, because they depend on something `dev/1.3` does not have:
- the assertions on the UDF query scope of an execution (`dev/1.3` has no
UDF query scope);
- the assertions on the `QUERY_TIMEOUT` status of a reclaimed query:
`stopAndCleanup()` does not move the query state machine on `dev/1.3`, so that
status stays `SUCCESS` there. The ported tests assert that the query is removed
from the `Coordinator`. The next section shows what this means for a client;
- `IoTDBInsertQueryIT.testInsertQueryInBatch` (table model).
### What a client sees when its query is reclaimed on `dev/1.3`
`getStatus()` of a reclaimed execution keeps reporting `SUCCESS` on
`dev/1.3`, because `QueryExecution.stopAndCleanup()` does not move the query
state machine there (`master` does since #19224). Only the first RPC of a query
reads that status. Every later RPC looks the query up in the `Coordinator` and
reports `715` when it is gone.
Measured on a cluster with two DataNodes, with `dev/1.3` at `86f9335302` and
with this branch. The queries are sent to DataNode 1 with a timeout of 2 s:
| Query | `dev/1.3` | This branch |
| --- | --- | --- |
| 20000 rows on DataNode 2, left alone after the first batch | not
reclaimed: still registered 84 s later, and the client then reads all 20000
rows | reclaimed after 63.5 s; the client reads the 4096 rows it had received
and then gets `715: Query is not found, it may be killed by others, timeout or
some other runtime errors, ...` |
| 20000 rows on DataNode 1, left alone after the first batch | no longer
registered when the client comes back; it reads 4096 rows and then gets `715` |
the same |
| first RPC still dispatching when the cleanup runs (DataNode 2 suspended
for 75 s) | reclaimed after 63.0 s and 63.9 s (two runs); when DataNode 2
continues, the RPC returns `SUCCESS` without a result set.
`Statement.executeQuery` throws `SQLException: Cannot invoke
"java.util.List.size()" because "columnNameList" is null`,
`Session.executeQueryStatement` throws a `NullPointerException` with the same
message | the same, reclaimed after 62.4 s |
- The queries this PR starts to reclaim (first row) end with an explicit
`715` on the next fetch. No result is cut short without an error.
- The last row is where the `SUCCESS` status reaches a client. It is
existing behaviour of `dev/1.3` and is not changed here: the old check skipped
only the queries without an RPC in progress, so a query whose first RPC is in
progress was reclaimed before this PR as well. On `master` the same run ends
with `720: Current query is time out, ...` for both clients, because #19224
made `stopAndCleanup()` fail the query state machine. That change is not part
of #19330 and is not included in this backport. The run needed
`dn_connection_timeout_ms=300000` to keep the dispatch RPC waiting for 75 s
(default: 60 s).
### Validation on `dev/1.3`
All commands were run offline on this branch with JDK 21.
Unit tests:
```bash
mvn -o test -pl iotdb-core/datanode -am \
-Dtest=CoordinatorTest,QueryExecutionLifecycleTest,ConfigExecutionTest,QueryExecutionRetryTest
\
-Dsurefire.failIfNoSpecifiedTests=false
```
- With the four main files of the first commit as on `dev/1.3`: 15 tests, 3
failures:
- `CoordinatorTest.testStaleQueryWithoutRpcInProgressIsCleanedUp:97
expected null, but was:<QueryExecution[...]>`
- `ConfigExecutionTest.isActiveTellsWhetherAnRpcOfTheQueryIsInProgress:169`
-
`QueryExecutionLifecycleTest.isActiveTellsWhetherAnRpcOfTheQueryIsInProgress:48`
- With this branch: 15 tests, 0 failures, with the spotless and checkstyle
checks of the build enabled.
Integration tests:
```bash
mvn -o clean verify -DskipUTs
'-Dit.test=IoTDBShowQueriesIT,IoTDBExecuteBatchIT' \
-DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false \
-pl integration-test -am -P with-integration-tests
```
- With `ClientRPCServiceImpl` as on `dev/1.3`, the three new tests fail:
- `IoTDBExecuteBatchIT.testJDBCExecuteBatchWithSelectInto:105 expected:<2>
but was:<0>`
- `IoTDBShowQueriesIT.queryInBatchReleaseTest: expected:<[]> but
was:<[20261119_125950_00007_1]>`
- `IoTDBShowQueriesIT.schemaTemplateQueryReleaseTest: expected:<[]> but
was:<[20261119_125950_00012_1]>`
- With this branch: `IoTDBShowQueriesIT` (2 tests) and `IoTDBExecuteBatchIT`
(3 tests) pass with the default profile (1C1D) and with `-PClusterIT` (1C3D).
- `mvn spotless:check -pl iotdb-core/datanode,integration-test -P
with-integration-tests` passes.
- The table above comes from a real 1C2D cluster of each build
(`query_timeout_threshold=5000`, `dn_connection_timeout_ms=300000`), driven
through JDBC and the Session API. The batch measurements of #19330 on two
DataNodes were not repeated on `dev/1.3`; the integration tests above cover
that fix.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious
for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold
for code coverage.
- [x] added integration tests.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `iotdb-core/datanode`: `org.apache.iotdb.db.queryengine.plan.Coordinator`,
`org.apache.iotdb.db.queryengine.plan.execution.QueryExecution`,
`org.apache.iotdb.db.queryengine.plan.execution.config.ConfigExecution`,
`org.apache.iotdb.db.protocol.thrift.impl.ClientRPCServiceImpl`
- `iotdb-core/datanode` (test): `CoordinatorTest`, `ConfigExecutionTest`,
`QueryExecutionLifecycleTest`
- `integration-test`: `IoTDBShowQueriesIT`, `IoTDBExecuteBatchIT`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]