JackieTien97 opened a new pull request, #19336:
URL: https://github.com/apache/iotdb/pull/19336

   ## Description
   
   A fix for `dev/1.3` only. `master` reports an error in this situation since 
#19224 (see "Relation to `master` and #19334").
   
   ### Scenario
   
   A client sends a query to a DataNode. While the RPC is still starting the 
query, the query is removed from the `Coordinator`, and the start then succeeds.
   
   On `dev/1.3` this is what `Coordinator.cleanUpStaleQueries` does to a query 
whose first RPC lasts longer than `timeout + 60 s`: it reclaims every query 
that exceeded its timeout by more than 60 s, also one whose first RPC is in 
progress. So the dispatch of the fragment instances has to hang for that long 
and then succeed, for example because a DataNode stops responding without 
closing its connections while `dn_connection_timeout_ms` is set above that 
time. With the default of 60 s the dispatch RPC times out first and the 
statement fails with an error.
   
   ### Incorrect behaviour
   
   Removing a query stops it and releases its result handle, but it does not 
move the state machine of the query on `dev/1.3` 
(`QueryExecution.stopAndCleanup(Throwable)` is `stop(t); releaseResource(t);`). 
When the start then succeeds, `getStatus()` reports `SUCCESS_STATUS`. 
`Coordinator.execution` returns that status, the RPC handler finds no execution 
for the query id, and answers with the bare success status: no columns and no 
data.
   
   What the client makes of that depends on the call it used. The calls that do 
not expect a result set report a statement that succeeded.
   
   ### Corrected behaviour
   
   The statement fails with `QUERY_WAS_KILLED(715)` and the text that the fetch 
RPCs of this branch already use for a query that is gone: `Query is not found, 
it may be killed by others, timeout or some other runtime errors, you can see 
more details in server log.`
   
   ### Example
   
   A cluster with one ConfigNode and two DataNodes, 
`query_timeout_threshold=5000`, `dn_connection_timeout_ms=300000`. 
`root.remote.d1` is stored on DataNode 2, the statement is sent to DataNode 1:
   
   ```sql
   select s1 from root.remote.d1 where time <= 10
   ```
   
   DataNode 2 is suspended with `SIGSTOP` right before the statement is sent 
and continued 75 s later. DataNode 1 reclaims the query in the meantime 
(`Cleaning up stale query with id 64, which has been running for 62025 ms, 
timeout duration is: 2000ms`), and the dispatch succeeds once DataNode 2 
continues.
   
   | Client call (timeout) | `dev/1.3` at `86f9335302` | This branch |
   | --- | --- | --- |
   | JDBC `Statement.executeQuery` (2 s) | `SQLException: Cannot invoke 
"java.util.List.size()" because "columnNameList" is null` | 
`IoTDBSQLException`, error code 715 |
   | `Session.executeQueryStatement` (2 s) | `NullPointerException` with the 
same message, thrown from `IoTDBRpcDataSet.<init>` | 
`StatementExecutionException`, status code 715 |
   | JDBC `Statement.execute` (2 s) | returns `false`, `getResultSet()` is 
`null`, no exception | `IoTDBSQLException`, error code 715 |
   | `Session.executeNonQueryStatement` (server default, 5 s) | returns 
normally | `StatementExecutionException`, status code 715 |
   
   All four calls return when DataNode 2 continues, 75 s after they were sent, 
in both builds. With both DataNodes running, the four calls return the 10 rows 
(or succeed) in both builds.
   
   ### Design
   
   - **The check is in `Coordinator.execution`, not in the RPC handlers.** 
Every entry point that executes a statement goes through this method, and it 
holds the execution object: after the start it checks whether a statement that 
it registered as a query is still registered. A handler cannot make this 
distinction. For a handler, "the statement is not a query" and "the query is 
gone" both look like "there is no execution for this query id", and 
`Statement.isQuery()` does not tell them apart, because the `Coordinator` 
registers every execution whose query type is `READ`, which includes for 
example the `SHOW` statements.
   - **Only a success status is replaced.** If the start of a removed query 
failed, the failure is returned as before; it says more than "not found". The 
status counts as success when its code is `SUCCESS_STATUS` or 
`REDIRECTION_RECOMMEND`, which is the condition under which the handlers go on 
to look up the execution.
   - **Statements that are not queries are not affected.** They are never 
registered, so there is nothing to compare.
   - **The code is 715, not 720.** The `Coordinator` does not know why the 
query was removed, and 715 with this text is what a client of this branch 
already gets when it fetches from a query that was removed.
   - **The message** is a constant in `DataNodeQueryMessages` (`en` and `zh`), 
with the name and the texts it has on `master` (there in 
`DataNodeMiscMessages`). `ClientRPCServiceImpl.NO_QUERY_EXECUTION_ERR_MSG` now 
refers to it instead of repeating the English text, so both places use one 
definition.
   
   ### Relation to `master` and #19334
   
   - `master` does not need this change. Since #19224, 
`QueryExecution.stopAndCleanup(Throwable)` moves the state machine of the query 
to a terminal state, so the first RPC reports the failure: the experiment above 
ends with `720: Current query is time out` for JDBC and the Session API on 
`master`. This PR does not port #19224. It only stops `dev/1.3` from reporting 
success.
   - This PR is independent of #19334 (the backport of #19330). A query whose 
first RPC is in progress was reclaimed before #19334 as well, so the behaviour 
above exists with and without it. The two PRs do not touch the same lines.
   
   ### Not changed
   
   - **The query is still reclaimed.** This PR changes only the status the 
first RPC returns. Whether `cleanUpStaleQueries` should spare a query whose 
first RPC is in progress is a separate question (#19330 explains why it does 
not).
   - **What the removed query leaves behind on the DataNode is the same with 
and without this PR.** `QueryExecution.stop` runs while the scheduler of the 
query is not assigned yet (`ClusterScheduler.start()` has not returned), so it 
stops nothing, and the scheduler starts its state tracker after the dispatch 
succeeded. Counted with `jmap -histo:live` on DataNode 1 after the four 
statements above were answered and all connections were closed, in both builds: 
0 `QueryExecution`, 0 `ClusterScheduler`, 5 `FixedRateFragInsStateTracker`, and 
4 of them 60 s later (1 before the experiment). `master` handles the stop of a 
query that is still being scheduled since #19224; that part is not ported here.
   - **A query that is removed between the check and the handler's own lookup** 
(`Coordinator.execution` returns, then the handler calls `getQueryExecution`) 
still takes the old path. That window is a few instructions wide instead of the 
whole dispatch.
   - **Other callers of the `Coordinator`** get the same `715` status instead 
of a success status for such a query. For example, `ClusterSchemaFetchExecutor` 
threw `Fetch Schema failed, because queryExecution is null`, and now throws 
`Fetch Schema failed, because Query is not found, ...` with code 715.
   
   ### Tests
   
   All commands were run offline on this branch with JDK 21.
   
   New unit test `CoordinatorRemovedQueryTest`, which runs a statement through 
`Coordinator.execution` with a real `QueryExecution`:
   
   - `testQueryReclaimedWhileItStartsIsReportedAsNotFound`: the real 
`cleanUpStaleQueries()` reclaims the query while it starts; the status must be 
715 with the message above.
   - `testQueryThatStaysRegisteredIsReportedAsSuccessful`
   - `testStatementThatIsNotAQueryIsReportedAsSuccessful`
   - `testFailureOfQueryRemovedWhileItStartsIsKept`: a removed query whose 
start failed keeps its failure status.
   
   ```bash
   mvn -o test -pl iotdb-core/datanode -am \
     
-Dtest=CoordinatorRemovedQueryTest,QueryExecutionRetryTest,ConfigExecutionTest \
     -Dsurefire.failIfNoSpecifiedTests=false
   ```
   
   - With `Coordinator` as on `dev/1.3`: 1 failure, 
`CoordinatorRemovedQueryTest.testQueryReclaimedWhileItStartsIsReportedAsNotFound:117
 expected:<715> but was:<200>`.
   - With this branch: 11 tests, 0 failures, with the spotless and checkstyle 
checks of the build enabled.
   
   Further runs on this branch:
   
   - The unit tests of the datanode packages `queryengine`, `protocol` and 
`storageengine.load` (418 test classes): 999 tests, 0 failures, 7 skipped.
   - `mvn -o test-compile -DskipTests -P with-zh-locale` passes.
   - Integration tests with the default profile (1C1D), `IoTDBSimpleQueryIT`, 
`IoTDBExecuteBatchIT`, `IoTDBSelectIntoIT`, `IoTDBMetadataFetchIT` and 
`IoTDBQueryDemoIT`: 109 tests, 0 failures, 4 skipped.
   - This branch merged with the branch of #19334: no conflict, and 
`CoordinatorRemovedQueryTest`, `CoordinatorTest`, 
`QueryExecutionLifecycleTest`, `QueryExecutionRetryTest` and 
`ConfigExecutionTest` pass together (19 tests).
   - The table in "Example" and the instance counts in "Not changed" come from 
real 1C2D clusters built from `dev/1.3` at `86f9335302` and from this branch.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
       - [x] concurrent read
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious
     for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, 
ensuring the threshold
     for code coverage.
   - [x] been tested in a test IoTDB cluster.
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - `iotdb-core/datanode`: `org.apache.iotdb.db.queryengine.plan.Coordinator`, 
`org.apache.iotdb.db.protocol.thrift.impl.ClientRPCServiceImpl`, 
`org.apache.iotdb.db.i18n.DataNodeQueryMessages` (`en`, `zh`)
   - `iotdb-core/datanode` (test): `CoordinatorRemovedQueryTest`
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to