Aleksander Nek created HDFS-18024:
-------------------------------------

             Summary: Preserve copy-truncate source during last-snapshot 
deletion on standby
                 Key: HDFS-18024
                 URL: https://issues.apache.org/jira/browse/HDFS-18024
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: namenode, snapshots
    Affects Versions: 3.6.0
            Reporter: Aleksander Nek


h2. Problem and impact

The copy-on-truncate source must survive until its replacement is completed, 
even if the snapshot that retained the source is deleted. During OP_TRUNCATE 
replay, a standby leaves the new target UNDER_CONSTRUCTION, whereas the active 
uses UNDER_RECOVERY. FileDiffList.combineAndCollectSnapshotBlocks() only 
protects a source whose target is UNDER_RECOVERY. Replaying deletion of the 
last snapshot can therefore remove a still-needed source from the standby block 
map.

After promotion, the missing metadata can cause the source replicas to be 
treated as unreferenced during block reports and invalidated before recovery 
creates a replacement. The regression directly proves premature metadata loss; 
invalidation of every physical replica is a possible consequence, not an 
assertion made by this test.

h2. Ordinary HA reproduction

# Start two NameNodes, three JournalNodes and three DataNodes with 
MiniQJMHACluster; no feature-specific configuration.
# Create a 1536-byte file, block size 1024, replication 3, and its only 
snapshot.
# Disable DataNode heartbeats to hold recovery pending; truncate the file by 
one byte.
# Roll and manually tail finalized edits on standby. Verify UNDER_RECOVERY on 
active versus UNDER_CONSTRUCTION on standby, both referencing the source block.
# Delete the snapshot and replay on standby.
# Expected: source remains in both block maps until recovery/CLOSE. Actual: 
source is absent on standby.

h2. Fix

Protect the source whenever the target is incomplete. Before OP_CLOSE replay 
completes the target and clears its under-construction feature, capture the 
source. After replay, remove a distinct copy source only when it has no 
latest-snapshot reference. Resolve the canonical stored block and detach its 
inode owner before BlockManager removal.

No configuration, protocol, layout or fsimage format change. FSNamesystem 
active-side cleanup is outside this patch.

This extends the active-side protection introduced by HDFS-7903 
(6acb7f2110897264241df44d564db2f85260348f), which checks UNDER_RECOVERY only; 
it covers the remaining HA replay case. HDFS-12638 concerns deleting the file 
itself, which is not done here. The issue remains reproducible on current trunk 
6c8dc8f2216cab134d5546f3327ad5907ef4b7d3.

h2. Validation

JDK 17.0.20.1, Maven 3.9.16, macOS ARM64, pure Java MiniClusters, one fork, 
isolated Maven repository. All runs use -ea, -DenableAssertions=true and 
-Dmaven.test.failure.ignore=false; the new fixture also checks 
INodeFile.class.desiredAssertionStatus().

* Current trunk with the new test only: 3 cases, expected standby 
source-retention failure 1, controls pass 2, errors/skips 0.
* Retention-only patch on c815ec8e (the two affected source files are unchanged 
in 6c8dc8f): 3 cases, expected unreferenced-source leak after CLOSE failure 1, 
controls pass 2, errors/skips 0.
* Complete patch on current trunk: TestTruncateSnapshotSource 3, 
TestFileTruncate 24, TestFSEditLogLoader 26 (13 sync + 13 async): all 53 pass, 
failures/errors/skips 0. Tests check failover, lease recovery, exact truncated 
bytes, CLOSE replay cleanup on both NameNodes, and retaining a snapshot through 
CLOSE until its later deletion.
* Checkstyle 11.1.0: zero diagnostics in the new test or on added runtime 
lines; 30 existing diagnostics in the selected runtime files.
* Targeted SpotBugs 4.9.7 before/after for the two runtime classes: no new 
findings (same 11 findings).

Log caveat: an active-side BlocksMap.removeBlock assertion is logged by 
recovery in the new HA case and existing TestFileTruncate, despite their 
successful JUnit results. A separate unmodified-trunk 
TestFileTruncate#testUpgradeAndRestart control also logs this assertion (1/1 
JUnit pass). FSNamesystem is unchanged and that active-side behavior is not 
repaired or claimed fixed here. Physical source-GC completion is not claimed. 
Native-library and shell/Bats suites were not run; Bats is unavailable locally.

Contains content generated by OpenAI Codex.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to