PDGGK opened a new pull request, #19195:
URL: https://github.com/apache/iotdb/pull/19195
## Description
Fixes the `SELECT`, `SHOW DEVICES` and `COUNT DEVICES` cases of #19178. The
`UPDATE` case is left out on purpose, see below.
Since 2.0.11, a table scalar UDF fails when it runs in a schema fragment,
which is what happens when it is applied to a TAG or ATTRIBUTE column in
`WHERE`. It fails even if the function never uses `IoTDBLocal`. With a UDF that
just returns its argument, and a table with devices `d1` (`attr1 = 'a'`, 2
rows) and `d2` (`attr1 = 'b'`, 1 row):
| statement | master | this PR |
|---|---|---|
| `SELECT count(*) FROM db.t WHERE str_f(device_id) = 'd1'` | `301:
outerQueryDeadlineMs must be positive for UDF execution` | 2 |
| `SELECT count(*) FROM db.t WHERE str_f(attr1) = 'a'` | same error | 2 |
| `SHOW DEVICES FROM db.t WHERE str_f(device_id) = 'd1'` | same error |
`d1`, `a` |
| `COUNT DEVICES FROM db.t WHERE str_f(device_id) = 'd1'` | `301:
IoTDBLocalFactory must not be null for UDF execution` | 1 |
### Cause
`UserDefineScalarFunctionTransformer` creates its `IoTDBLocal` in the
constructor, and `IoTDBLocalFactory#createIoTDBLocal` needs a factory, a
fragment instance id, a query id and a positive deadline. Two places do not
supply them:
- `FragmentInstanceManager#execSchemaQueryFragmentInstance` creates the
context without a deadline, so it stays at -1. The data path passes
`DriverTask.computeDeadlineTimeInMs(instance.getTimeOut())`.
- `DataNodeTableOperatorGenerator#visitTableDeviceQueryCount` builds its
`ColumnTransformerBuilder.Context` with the shorter constructor, which leaves
all four unset. `SHOW DEVICES` does not have this problem because its filter
goes through the normal filter operator.
### Change
- Schema fragments get a deadline computed the same way as data fragments.
- `COUNT DEVICES` passes the four values to its context.
Nothing else reads the deadline, so schema queries without a UDF behave as
before. A UDF that does call `IoTDBLocal` now also works in these predicates,
and its nested query runs as the current user, because the device fetch carries
the outer query's session.
### Not in this PR
- `UPDATE` with a UDF still fails. The plan in #19178 is to reject it at
analysis time.
- #19107 edits the same `createFragmentInstanceContext` call in
`execSchemaQueryFragmentInstance`, so whichever of the two goes in second needs
a small rebase.
- On a table view over tree data, `COUNT DEVICES FROM view WHERE
upper(device_id) = 'D1'` returns 0 on master, while `SHOW DEVICES` with the
same condition finds the device. A UDF there no longer fails with this PR, but
it gets the same 0. That comes from how `DevicePredicateHandler` picks the tag
position, it doesn't depend on UDFs, so I left it out.
### Verification
- **Real server, master `cd31721` against this change.** The statements
above and `SELECT ... WHERE str_f(device_id) = 'd2' AND v > 1`, each run three
times: right after the insert, again with the device cache warm, and after
`CLEAR ATTRIBUTE CACHE`. Master fails every time and this change returns the
right result every time. `SELECT count(*)` and `SHOW DEVICES` with the UDF on a
table view also return the right result.
- **New tests in `IoTDBLocalIT`:** a UDF that does not use `IoTDBLocal`, a
UDF that queries through `IoTDBLocal`, and a user without permission on the
table the UDF reads, who gets `Access Denied` from `SELECT`, `SHOW DEVICES` and
`COUNT DEVICES`.
- The class passes with `-PTableSimpleIT` (23 tests, one existing test is
skipped there).
- I ran it three times with `-PTableClusterIT` (1C3D). The three new tests
passed every time. One run passed all 23; the other two hit the timeouts
described below.
- On master all three new tests fail with `outerQueryDeadlineMs must be
positive`. With only the `COUNT DEVICES` change they fail the same way. With
only the deadline change they fail at `COUNT DEVICES` with `IoTDBLocalFactory
must not be null`.
- `spotless:check` passes for datanode and integration-test.
### Cluster timeouts in `IoTDBLocalIT`
This is not caused by this PR, but the cluster job may show it. With
`-PTableClusterIT`, nine existing tests that run nested `IoTDBLocal` queries
(`testDeviceNameWithSingleQuery`, `testLocalQueryUdafAtBeforeStart` and others)
can time out after 120 s. I saw the same nine on master `cd31721` without this
change, and in two of the three runs above. A thread dump taken during the
timeout, on master, shows all six `Query-Worker` threads of one DataNode
blocked in `QueryExecution.getBatchResult`, inside nested queries or the device
fetch of a nested query. That looks like the worker pool running out, since
those queries need worker threads too. I can open a separate issue with the
dump if that helps.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added integration tests.
- [x] been tested in a test IoTDB cluster.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `FragmentInstanceManager`, `FragmentInstanceContext`
- `DataNodeTableOperatorGenerator`
- `IoTDBLocalIT`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]