mingisme opened a new pull request, #19332:
URL: https://github.com/apache/iotdb/pull/19332
## Description
Fixes #19331.
### Problem
Tree-model queries on schema-template devices fetch the schema remotely on
every run instead of
reading the DataNode schema cache. On a cluster with
`schema_replication_factor=3`, that is a
Ratis read on the schema region per statement. In our deployment (3C3D,
2.0.8), a 40-device
`SELECT LAST` spent about 180 ms of its 575 ms average in schema fetch. With
both fixes below,
schema fetch fell to 13 ms and the average to 192 ms.
Two bugs cause it.
**1. `ClusterSchemaFetcher.fetchSchema` reads the cache with the wrong
key.** For template
devices it calls
`schemaCache.getMatchedTemplateSchema(pattern.getDevicePath())`. `pattern` is a
`PartialPath` for a full series such as `root.db.d1.s1`, and
`PartialPath.getDevicePath()`
returns the path itself, so the lookup is for device `root.db.d1.s1`. The
cache stores the
device as `root.db.d1`, so every lookup misses and the statement goes to
`fetchSchemaOfPreciseMatchOrPreciseDeviceUsingTemplate`. The normal-series
branch of the same
method already strips the measurement.
**2. `TableDeviceCacheEntry.setDeviceSchema` cannot replace a normal
schema.** This shows only
once (1) is fixed, and only on clusters:
1. DataNode A coordinates `SELECT LAST` on template device `d`, whose data
region is on
DataNode B.
2. B runs the fragment. `OperatorTreeGenerator` calls `declareLastCache`,
which reaches
`TableDeviceCacheEntry.setMeasurementSchema` and creates the entry with a
`TreeDeviceNormalSchema`.
3. B later coordinates a query on `d`. `getMatchedTemplateSchema` misses
because the entry holds
a normal schema. B fetches the template schema and calls
`putDeviceSchema`.
4. `setDeviceSchema` uses `compareAndSet(null, new
TreeDeviceTemplateSchema(...))`, which fails
because the field is not null. Steps 3 and 4 repeat for every query B
coordinates on `d`,
until restart.
A multi-device statement does the remote fetch when any one device misses,
so a few stuck
devices per node keep every multi-device `SELECT LAST` slow.
### Fix
1. Look up the template device by its device path: drop the last node of
`pattern`.
2. In `setDeviceSchema`, replace a missing schema or a
`TreeDeviceNormalSchema` with the fetched
`TreeDeviceTemplateSchema` (`getAndUpdate`), and return the size delta. A
template schema or
a table-model `TableAttributeSchema` is never replaced. The separate
`lastCache` field is
untouched, so values cached by the fragment are kept.
Alternative considered for (2): stop `declareLastCache` from creating a
normal schema for
template devices. The fragment would then need a template check on the query
execution path.
Replacing the normal schema at put time keeps the change inside the template
branch of
`setDeviceSchema`.
### Reproduction (1 ConfigNode and 3 DataNodes, local Docker, 40 template
devices)
Steady `Fetch Schema Cost` from `EXPLAIN ANALYZE`. A device counts as
"stuck" when it costs more
than 1 ms; a cache hit is about 0.07 ms.
| Build | Stuck devices on dn1 / dn2 / dn3 | 40-device `SELECT LAST` fetch
cost |
|---|---|---|
| master | 40 / 40 / 40 | 11.9 ms |
| fix 1 only | 0 / 13 / 12 (the devices whose fragments ran on that node) |
12.0 ms |
| both fixes | 0 / 0 / 0 | 0.37 ms |
Rows were identical across nodes and builds, and new writes were visible in
`SELECT LAST` on
every node.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent read
- [x] concurrent write
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious
for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths.
- [x] been tested in a test IoTDB cluster.
<hr>
##### Key changed/added classes in this PR
- `ClusterSchemaFetcher`
- `TableDeviceCacheEntry`
- `ClusterSchemaFetcherTest` (new)
- `TreeDeviceSchemaCacheManagerTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]