luoluoyuyu opened a new pull request, #19114:
URL: https://github.com/apache/iotdb/pull/19114
## Description
`TestUtils.restartCluster()` returned as soon as every node reported a
non-`Unknown` status and a
JDBC connection succeeded. A running node does not imply a serving Region:
right after a restart the
ConfigNode `LoadCache` keeps the RegionGroups of a Database `Disabled` until
all of their Regions
report `Running`, and any write issued in that window, including an active
LOAD, is rejected with
`NO_AVAILABLE_REGION_GROUP` (906).
`IoTDBLoadTsFileActiveScanIT#testIncompletePublicationSurvivesScannerRestartAndPreservesMods`
restarts the cluster and immediately publishes a canary TsFile into the
active LOAD listening
directory. `ActiveLoadDirScanner` rescans that directory every second, so
the canary was picked up
inside the window described above, the LOAD failed permanently (the file was
moved to
`ext/load/failed` and the status was rewritten to `LOAD_FILE_ERROR`) and
`awaitRows` timed out after
60 seconds.
Timeline from a failing CI run:
```
16:00:38.549 DataNode17821 "DataNode started"
16:00:39.674 active load scanner picked 712-0-0-0.tsfile
16:00:39.794 ConfigNode: No available SchemaRegion ... {4=Disabled,
2=Disabled, 0=Disabled}
16:00:39.800 Failed to auto load ... File will be moved to fail directory
16:00:39.878 same RegionGroups: Disabled -> Running
```
The window is ~100 ms wide, so the test passes most of the time and fails
rarely.
### Fix
`restartCluster()` now also calls the new
`TestUtils.awaitAllRegionsRunning(env)`, which waits
(bounded by 2 minutes, polled every second) until every Region listed by
`SHOW REGIONS` is
`Running`.
`SHOW REGIONS` reports `LoadManager.getRegionStatus`, read from the same
`LoadCache` that computes
`RegionGroupStatus` (`RegionGroupCache#caculateRegionGroupStatus`: a group
is `Running` only when all
of its Regions are `Running`). "All Regions Running" is therefore exactly
the condition that
guarantees the next write cannot hit `NO_AVAILABLE_REGION_GROUP` — it waits
on the same state that
rejects the write, not on an approximation, and it is an observable
condition instead of a fixed
sleep. No assertion of the affected tests is changed, and the active LOAD
test itself is untouched.
### Alternatives considered
- Probing the Database with an `INSERT` inside the affected test: works, but
only covers one
Database and has to be repeated in every restart-based IT.
- Waiting for all Regions inside `checkClusterStatusWithoutUnknown()`: would
also cover the initial
cluster start, but that helper is used by many tests that do not need the
extra wait, so the wait
is scoped to `restartCluster()` here.
- Making active LOAD retry instead of failing the file (the product-side
counterpart of this race):
kept out of this PR. A DataNode cannot observe the ConfigNode `LoadCache`
view during startup, and
blocking active LOAD globally until every Region is ready would stall
healthy Databases whenever a
single RegionGroup is unavailable.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious
for an unfamiliar reader.
- [x] added integration tests. (the existing active LOAD IT covers the
restart path; this PR makes
its precondition deterministic)
Not applicable: documentation, version/license information, concurrency of
the changed helper (it
runs in the test JVM only, single-threaded via Awaitility).
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
-
`integration-test/src/test/java/org/apache/iotdb/db/it/utils/TestUtils.java`:
`restartCluster()` calls the new `awaitAllRegionsRunning()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]