luoluoyuyu opened a new pull request, #19114:
URL: https://github.com/apache/iotdb/pull/19114

   ## Description
   
   `TestUtils.restartCluster()` returned as soon as every node reported a 
non-`Unknown` status and a
   JDBC connection succeeded. A running node does not imply a serving Region: 
right after a restart the
   ConfigNode `LoadCache` keeps the RegionGroups of a Database `Disabled` until 
all of their Regions
   report `Running`, and any write issued in that window, including an active 
LOAD, is rejected with
   `NO_AVAILABLE_REGION_GROUP` (906).
   
   
`IoTDBLoadTsFileActiveScanIT#testIncompletePublicationSurvivesScannerRestartAndPreservesMods`
   restarts the cluster and immediately publishes a canary TsFile into the 
active LOAD listening
   directory. `ActiveLoadDirScanner` rescans that directory every second, so 
the canary was picked up
   inside the window described above, the LOAD failed permanently (the file was 
moved to
   `ext/load/failed` and the status was rewritten to `LOAD_FILE_ERROR`) and 
`awaitRows` timed out after
   60 seconds.
   
   Timeline from a failing CI run:
   
   ```
   16:00:38.549 DataNode17821 "DataNode started"
   16:00:39.674 active load scanner picked 712-0-0-0.tsfile
   16:00:39.794 ConfigNode: No available SchemaRegion ... {4=Disabled, 
2=Disabled, 0=Disabled}
   16:00:39.800 Failed to auto load ... File will be moved to fail directory
   16:00:39.878 same RegionGroups: Disabled -> Running
   ```
   
   The window is ~100 ms wide, so the test passes most of the time and fails 
rarely.
   
   ### Fix
   
   `restartCluster()` now also calls the new 
`TestUtils.awaitAllRegionsRunning(env)`, which waits
   (bounded by 2 minutes, polled every second) until every Region listed by 
`SHOW REGIONS` is
   `Running`.
   
   `SHOW REGIONS` reports `LoadManager.getRegionStatus`, read from the same 
`LoadCache` that computes
   `RegionGroupStatus` (`RegionGroupCache#caculateRegionGroupStatus`: a group 
is `Running` only when all
   of its Regions are `Running`). "All Regions Running" is therefore exactly 
the condition that
   guarantees the next write cannot hit `NO_AVAILABLE_REGION_GROUP` — it waits 
on the same state that
   rejects the write, not on an approximation, and it is an observable 
condition instead of a fixed
   sleep. No assertion of the affected tests is changed, and the active LOAD 
test itself is untouched.
   
   ### Alternatives considered
   
   - Probing the Database with an `INSERT` inside the affected test: works, but 
only covers one
     Database and has to be repeated in every restart-based IT.
   - Waiting for all Regions inside `checkClusterStatusWithoutUnknown()`: would 
also cover the initial
     cluster start, but that helper is used by many tests that do not need the 
extra wait, so the wait
     is scoped to `restartCluster()` here.
   - Making active LOAD retry instead of failing the file (the product-side 
counterpart of this race):
     kept out of this PR. A DataNode cannot observe the ConfigNode `LoadCache` 
view during startup, and
     blocking active LOAD globally until every Region is ready would stall 
healthy Databases whenever a
     single RegionGroup is unavailable.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious
     for an unfamiliar reader.
   - [x] added integration tests. (the existing active LOAD IT covers the 
restart path; this PR makes
     its precondition deterministic)
   
   Not applicable: documentation, version/license information, concurrency of 
the changed helper (it
   runs in the test JVM only, single-threaded via Awaitility).
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - 
`integration-test/src/test/java/org/apache/iotdb/db/it/utils/TestUtils.java`:
     `restartCluster()` calls the new `awaitAllRegionsRunning()`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to