PDGGK opened a new pull request, #19195:
URL: https://github.com/apache/iotdb/pull/19195

   ## Description
   
   Fixes the `SELECT`, `SHOW DEVICES` and `COUNT DEVICES` cases of #19178. The 
`UPDATE` case is left out on purpose, see below.
   
   Since 2.0.11, a table scalar UDF fails when it runs in a schema fragment, 
which is what happens when it is applied to a TAG or ATTRIBUTE column in 
`WHERE`. It fails even if the function never uses `IoTDBLocal`. With a UDF that 
just returns its argument, and a table with devices `d1` (`attr1 = 'a'`, 2 
rows) and `d2` (`attr1 = 'b'`, 1 row):
   
   | statement | master | this PR |
   |---|---|---|
   | `SELECT count(*) FROM db.t WHERE str_f(device_id) = 'd1'` | `301: 
outerQueryDeadlineMs must be positive for UDF execution` | 2 |
   | `SELECT count(*) FROM db.t WHERE str_f(attr1) = 'a'` | same error | 2 |
   | `SHOW DEVICES FROM db.t WHERE str_f(device_id) = 'd1'` | same error | 
`d1`, `a` |
   | `COUNT DEVICES FROM db.t WHERE str_f(device_id) = 'd1'` | `301: 
IoTDBLocalFactory must not be null for UDF execution` | 1 |
   
   ### Cause
   
   `UserDefineScalarFunctionTransformer` creates its `IoTDBLocal` in the 
constructor, and `IoTDBLocalFactory#createIoTDBLocal` needs a factory, a 
fragment instance id, a query id and a positive deadline. Two places do not 
supply them:
   
   - `FragmentInstanceManager#execSchemaQueryFragmentInstance` creates the 
context without a deadline, so it stays at -1. The data path passes 
`DriverTask.computeDeadlineTimeInMs(instance.getTimeOut())`.
   - `DataNodeTableOperatorGenerator#visitTableDeviceQueryCount` builds its 
`ColumnTransformerBuilder.Context` with the shorter constructor, which leaves 
all four unset. `SHOW DEVICES` does not have this problem because its filter 
goes through the normal filter operator.
   
   ### Change
   
   - Schema fragments get a deadline computed the same way as data fragments.
   - `COUNT DEVICES` passes the four values to its context.
   
   Nothing else reads the deadline, so schema queries without a UDF behave as 
before. A UDF that does call `IoTDBLocal` now also works in these predicates, 
and its nested query runs as the current user, because the device fetch carries 
the outer query's session.
   
   ### Not in this PR
   
   - `UPDATE` with a UDF still fails. The plan in #19178 is to reject it at 
analysis time.
   - #19107 edits the same `createFragmentInstanceContext` call in 
`execSchemaQueryFragmentInstance`, so whichever of the two goes in second needs 
a small rebase.
   - On a table view over tree data, `COUNT DEVICES FROM view WHERE 
upper(device_id) = 'D1'` returns 0 on master, while `SHOW DEVICES` with the 
same condition finds the device. A UDF there no longer fails with this PR, but 
it gets the same 0. That comes from how `DevicePredicateHandler` picks the tag 
position, it doesn't depend on UDFs, so I left it out.
   
   ### Verification
   
   - **Real server, master `cd31721` against this change.** The statements 
above and `SELECT ... WHERE str_f(device_id) = 'd2' AND v > 1`, each run three 
times: right after the insert, again with the device cache warm, and after 
`CLEAR ATTRIBUTE CACHE`. Master fails every time and this change returns the 
right result every time. `SELECT count(*)` and `SHOW DEVICES` with the UDF on a 
table view also return the right result.
   - **New tests in `IoTDBLocalIT`:** a UDF that does not use `IoTDBLocal`, a 
UDF that queries through `IoTDBLocal`, and a user without permission on the 
table the UDF reads, who gets `Access Denied` from `SELECT`, `SHOW DEVICES` and 
`COUNT DEVICES`.
     - The class passes with `-PTableSimpleIT` (23 tests, one existing test is 
skipped there).
     - I ran it three times with `-PTableClusterIT` (1C3D). The three new tests 
passed every time. One run passed all 23; the other two hit the timeouts 
described below.
     - On master all three new tests fail with `outerQueryDeadlineMs must be 
positive`. With only the `COUNT DEVICES` change they fail the same way. With 
only the deadline change they fail at `COUNT DEVICES` with `IoTDBLocalFactory 
must not be null`.
   - `spotless:check` passes for datanode and integration-test.
   
   ### Cluster timeouts in `IoTDBLocalIT`
   
   This is not caused by this PR, but the cluster job may show it. With 
`-PTableClusterIT`, nine existing tests that run nested `IoTDBLocal` queries 
(`testDeviceNameWithSingleQuery`, `testLocalQueryUdafAtBeforeStart` and others) 
can time out after 120 s. I saw the same nine on master `cd31721` without this 
change, and in two of the three runs above. A thread dump taken during the 
timeout, on master, shows all six `Query-Worker` threads of one DataNode 
blocked in `QueryExecution.getBatchResult`, inside nested queries or the device 
fetch of a nested query. That looks like the worker pool running out, since 
those queries need worker threads too. I can open a separate issue with the 
dump if that helps.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added integration tests.
   - [x] been tested in a test IoTDB cluster.
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - `FragmentInstanceManager`, `FragmentInstanceContext`
   - `DataNodeTableOperatorGenerator`
   - `IoTDBLocalIT`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to