hongzhi-gao opened a new pull request, #19268:
URL: https://github.com/apache/iotdb/pull/19268

   ## Description
   
   Reconstructing a replica in a two-replica IoTConsensus region temporarily 
leaves one replica. That transition permits WAL reclamation, but registering 
the replacement channel does not immediately restore its retention boundary. 
Cleanup can also keep using the file boundary calculated before registration. 
Incremental WAL needed after the snapshot can therefore disappear. The 
dispatcher currently skips missing indexes, allowing replication progress to 
advance across the hole.
   
   For example, the new regression scenario pauses snapshot transmission, 
acknowledges 12,000 more writes, and runs WAL cleanup. With the original 
production code at `ed9504f3ae`, isolating the reconstructed replica returns 
only the snapshot's 1 row instead of the expected 12,001 rows. With this fix, 
each isolated replica returns all 12,001 timestamps and values.
   
   ### Changes
   
   - Hold the existing write lock while capturing the initial replication 
index, registering the channel, and publishing WAL retention. Serialize 
retention calculation/publication so a stale calculation cannot overwrite a 
newly lowered boundary.
   - Coordinate retention publication with each WAL file's eligibility check 
and actual deletion. Cache the binary-search result while the boundary remains 
unchanged, and refresh it whenever the boundary or the cleanup file list 
changes.
   - Notify the LOAD retention listener only when the boundary advances, 
matching its contract. Lowering a boundary cannot make additional directories 
reclaimable; task registration and recovery already evaluate retention 
explicitly.
   - Fail the replication channel on a missing required WAL index rather than 
sending later entries across the gap. After activation, wait for the 
coordinator's outbound channel to acknowledge a fixed local frontier before 
reporting add-peer success, with failure propagation and a 10-minute timeout.
   - Release the dispatcher monitor before retention callbacks and provide 
matching English/Chinese exception templates.
   
   No on-disk or RPC format changes are introduced. Validation targets two data 
replicas; the catch-up wait checks the coordinator's outbound channel.
   
   ### Validation
   
   - `IoTConsensusServerImplTest`: 5 tests passed, including immediate 
retention publication, catch-up timeout, and failure on a WAL gap. Re-run after 
the localization change.
   - Focused `WALNodeTest`: 3 tests passed, including a deterministic 
cleanup-initialization / boundary-lowering / deletion sequence and listener 
notifications.
   - `LoadTsFileManagerTest`: 30 tests passed.
   - `IoTDBRegionReconstructWALIT`: passed with 1 ConfigNode and 3 DataNodes, 
data RF=2. Uses the existing debugger fixture to pause only snapshot 
transmission, exceeds the dispatcher queue capacity, invokes real WAL cleanup, 
and verifies both replicas separately by stopping the other replica. The 
old-code regression run failed with 1/12,001 rows.
   - `mvn test-compile -o -DskipTests -Ddevelocity.off=true` and the same 
full-reactor command with `-Pwith-zh-locale`: passed after localization. 
Compiled Chinese constants and the missing-WAL exception were inspected.
   - Spotless, locale key/placeholder parity, raw-literal checks for the five 
added failures, and `git diff --check`: passed.
   
   The WAL/LOAD tests and IT ran before the final message-only localization 
change. Process-kill crash tests, I/O fault injection, and cross-host 
performance tests were not run.
   
   <details>
   <summary>Focused test commands</summary>
   
   ```sh
   mvn test -o -pl iotdb-core/consensus -am -Dtest=IoTConsensusServerImplTest 
-Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false 
-Ddevelocity.off=true
   
   mvn verify -o 
'-Dtest=WALNodeTest#testRetentionListenerOnlyRunsWhenBoundaryAdvances+testCleanupRechecksLoweredReplicationBoundary+testDeleteOutdatedFiles,LoadTsFileManagerTest'
 -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false 
-Dfailsafe.failIfNoSpecifiedTests=false -Dit.test=IoTDBRegionReconstructWALIT 
-pl integration-test -am -Pwith-integration-tests,ClusterIT 
-Ddevelocity.off=true
   ```
   
   </details>
   
   ### Performance observations
   
   Compared the previous correctness-fix implementation (per-file binary search 
/ unconditional listener notification) with the optimized implementation, not 
with unfixed upstream. Three alternating repetitions per version used a shared 
HDD host, colocated 1C2D with RF=2, ASYNC WAL, 8 clients, and 10-row tablets 
with 3 INT64 measurements per row. Each run used fresh directories, a 10-second 
warmup, a 30-second steady window, and a 60-second reconstruction window. All 
six runs checked acknowledged write counts and confirmed reconstruction success 
before the window ended.
   
   | Median | Before optimization | After optimization |
   | --- | ---: | ---: |
   | Steady rows/s | 339,149 | 337,837 |
   | Steady tablet-write P99 | 0.397 ms | 0.410 ms |
   | Reconstruction-window rows/s | 199,884 | 198,832 |
   | Reconstruction-window tablet-write P99 | 0.392 ms | 0.384 ms |
   
   Throughput medians changed by -0.39% and -0.53%; steady P99 increased by 
3.27%. These measurements do not establish zero regression. Existing 
approximately 30-second remove-channel stalls occurred in five runs and are 
outside this change.
   
   A separate synthetic-metadata benchmark of the real eligibility predicate 
reduced the median 100,000-file scan at a stable finite boundary from 307.45 ms 
to 0.391 ms; the MAX_VALUE fast branch remained about 1.65 ms versus 1.62 ms. 
This excludes file deletion and lock contention and is not an end-to-end 
speedup claim.
   
   This PR has:
   
   - [x] Been self-reviewed and independently reviewed for lock ordering and 
lifecycle behavior.
   - [x] Added unit/component regression coverage.
   - [x] Added and executed a targeted cluster integration test.
   - [x] Provided English and Chinese messages.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to