Caideyipi opened a new pull request, #18818:
URL: https://github.com/apache/iotdb/pull/18818

   ## Description
   
   Continuous subscription WAL replay could permanently skip unreadable 
retained files while `subscription_consensus_wal_gap` remained zero. The 
detailed warning only ran when the iterator was exhausted, and clients received 
a noncritical error containing only a file count.
   
   This change makes file skips observable and diagnosable:
   
   - Add queue/region metrics `subscription_consensus_wal_file_gap` and 
`subscription_consensus_wal_file_gap_entries`, plus queue diagnostics. Counters 
accumulate across iterator resets. File counts measure skip occurrences; entry 
counts include only known skipped WAL entries/fragments, including discarded 
incomplete request fragments. An unreadable footer leaves its entry count 
explicitly unknown.
   - Log each permanent skip immediately with its absolute filename, version 
ID, file-level searchIndex bounds, entry offsets and exception. Send a critical 
poll error with queue/region context and up to eight recent file details; the 
server log contains every skipped file.
   - Retry EOF at the original file/entry offset while preserving incomplete 
requests. Only a repeated EOF at the same offset in an unchanged sealed 
retained file confirms truncation. Near-live EOF and ordinary I/O failures wait 
for another replay attempt. Preserve the existing escape path for entries above 
the supported WAL size limit, with an explicit gap signal.
   - Preserve a critical error from parallel consumer poll tasks even when 
sibling topics return data. Healthy messages are returned once, and the next 
poll raises the saved critical exception instead of permanently reducing it to 
a warning.
   
   `subscription_consensus_wal_gap` retains its local searchIndex unit. 
Monitoring and regression checks must require both it and 
`subscription_consensus_wal_file_gap` to be zero; a zero known-entry count 
alone does not exclude a file gap. Filename searchIndex bounds describe the 
whole file and are not an inferred row-loss count.
   
   ## Validation
   
   - Full English reactor build through `test`: 88 tests in eight targeted 
subscription/WAL classes, zero failures/errors, one optional performance test 
skipped.
   - Full Chinese reactor `test-compile -DskipTests -P with-zh-locale`: passed 
(53 modules).
   - Spotless applied to DataNode, Node Commons and Subscription Client; new 
English/Chinese message keys and format specifiers match.
   
   This PR has:
   
   - [x] been self-reviewed.
   - [x] added comments explaining the retry, counting and error-delivery 
behavior.
   - [x] added or updated unit tests for transient/sealed/live EOF, 
preservation of incomplete fragments, immediate file warnings, bounded 
diagnostics, cumulative/region-scoped gauges, and delivery of a critical error 
after healthy parallel poll results.
   
   The supplied regression logs confirm six noncritical skip errors, zero 
detailed iterator warnings and 5,400 zero-valued WAL-gap samples. Those logs 
cannot identify the skipped files or establish a row-loss count; the original 
high-load benchmark has not been rerun.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to