Aleksander Nek created HDFS-18023:
-------------------------------------

             Summary: Async edit tailing can save rolling-upgrade rollback 
images inside an edit-log segment
                 Key: HDFS-18023
                 URL: https://issues.apache.org/jira/browse/HDFS-18023
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: namenode
    Affects Versions: 3.6.0
            Reporter: Aleksander Nek


h2. Problem

With dfs.ha.tail-edits.in-progress=true, standby replay can advance across the 
segment boundary rolled by rollingUpgrade(PREPARE), or yield before 
END_LOG_SEGMENT when the replay batch is small. StandbyCheckpointer then saves 
fsimage_rollback at a TXID inside a segment. Rollback subsequently calls 
discardSegments(imageTxId+1), which requires a segment boundary and can fail.

h2. Reproduction

# Start two HA NameNodes and three JournalNodes with RPC edit tailing enabled.
# Write a namespace entry, then invoke rollingUpgrade(PREPARE).
# Wait for rollback images and compare the image TXID with finalized journal 
segment ends.
# Repeat with dfs.ha.tail-edits.max-txns-per-lock=1.

Observed before the fix: finalized segment ends at TXID 4, but the rollback 
image is saved at 5 (ordinary batch) or 3 (single-transaction batch). None of 
the three journals has a matching finalized boundary. The old-only baseline 
reproducer fails discard starting at 6 for segment [5,7].

h2. Proposed fix and validation

Periodic standby replay stops immediately after applying the first 
END_LOG_SEGMENT after PREPARE, and preserves that exact namespace until the 
local rollback image exists. The checkpointer defers a rollback image until 
this boundary is ready. Failover catch-up must bypass the pause; observers and 
non-checkpointing standbys must continue replaying. Startup replay remains 
unrestricted.

Both before-fix regressions fail with zero of three aligned journals. A focused 
baseline overlay containing only this fix passes both batch sizes, real 
rollback and subsequent restart, without Multi-DC code or the separate 
JournalNode cache fix. The standalone rollback test uses finalized streaming 
during rollback to isolate the two defects; PREPARE itself remains 
asynchronous. On current trunk all six focused tests pass: both replay batch 
sizes with real rollback and restart, failover and restart while the rollback 
image is pending, Observer replay and disabled-checkpointer replay. Existing 
HA/checkpoint regression suites are also being run before the PR is published.

The defect is present in the baseline e66f40daf3cc4ebbb5164cec1ce54b71eddec8a4 
and the corresponding source paths on current trunk 
c815ec8e1e98f20e3f0ae2637665d4adb9196660. No new configuration, wire or image 
format is needed. This report concerns HA checkpoint boundary selection, rather 
than the non-HA replay retry in HDFS-17932.

Contains content generated by OpenAI Codex.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to