jt2594838 opened a new pull request, #18693:
URL: https://github.com/apache/iotdb/pull/18693

   ## Description
   
   When a WAL footer is missing or corrupted, metadata readers currently fail 
even if complete entries remain readable. Recover metadata from the readable 
entry prefix, recognize empty/header-only WAL files, and quarantine nonempty 
files with no recoverable entries as `.broken` (with a numbered suffix on 
collision).
   
   - Validate footer magic, lengths, and counts before deserialization. 
Recovery scans ignore the damaged footer and retain original encoded entry 
sizes, including legacy V1 entries and complete entries in a truncated 
uncompressed segment.
   - Ordinary readers reconstruct metadata in memory without replacing a file 
held by other readers. Startup repair writes complete entries and a new footer 
to a temporary file, forces and closes it, then replaces the original. Failed 
rewrites preserve the original file.
   - Keep unknown memTable IDs distinct from an empty set, close metadata 
channels, preserve quarantined files during startup cleanup, and avoid 
overwriting earlier quarantine files.
   
   V3 writer progress stored only in a damaged footer cannot be reconstructed 
from entry bodies; recovery uses the existing unknown/default progress values. 
Incomplete compressed segments cannot be decoded, so recovery retains complete 
readable entries before them.
   
   ## Validation
   
   - 29 tests passed across `WALFileTest`, `WALRepairWriterTest`, 
`WALMetaDataV3CompatibilityTest`, and `ProgressWALReaderTest`, using freshly 
compiled affected sources and existing local dependency artifacts in an 
isolated output directory.
   - Coverage includes V1/V2/V3, invalid footer lengths/content, LZ4, truncated 
uncompressed segments, empty files, quarantine collisions, failed repair 
preserving the original, and startup cleanup retaining quarantined files.
   - Spotless, Checkstyle (zero violations), and `git diff --check` passed.
   - The full Maven reactor test attempt on source base `34349e3ad8b` stopped 
before WAL tests at pre-existing `BuiltinPipePlugin` references to missing 
`OPC_UA_CONNECTOR` and `OPC_UA_SINK`. Upstream has since fixed those references 
in #18687; a full reactor run on that newer base has not yet been performed. 
Targeted validation is not a successful full reactor build.
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining non-obvious recovery and resource-lifecycle 
decisions.
   - [x] added unit tests and updated existing tests for the changed behavior.
   
   Key changed classes: WALMetaData, WALInputStream, WALReader, 
WALByteBufReader, WALFileVersion, WALWriter, WALBuffer, WALNode, 
WALNodeRecoverTask, WALRepairWriter.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to