iskenderurs opened a new pull request, #8942: URL: https://github.com/apache/hadoop/pull/8942
### Description of PR [HDFS-18023](https://issues.apache.org/jira/browse/HDFS-18023) With asynchronous edit tailing, a rollback image created for rollingUpgrade(PREPARE) can fall inside an edit-log segment: the RPC batch may include the next START, or a small replay limit may yield before END. Subsequent rollback requires a segment boundary and can fail to discard the remaining journal segments. Pin periodic standby replay immediately after END_LOG_SEGMENT while the rollback image is pending. Save the image only at that boundary, including when an ordinary checkpoint already has the same TXID. Failover catch-up bypasses the pause, and Observer/non-checkpointing NameNodes continue replaying. After restart, a standby requests a future segment roll if necessary to obtain a valid boundary. Checkpoint retries release the checkpoint lock and sleep to let replay progress. No configuration, protocol, image format or new dependency is introduced. This PR is independent of HDFS-18022, which addresses stale JournalNode RPC caches after discard. ### How was this patch tested? Based on Apache Hadoop trunk `c815ec8e1e98f20e3f0ae2637665d4adb9196660`, JDK 17 on macOS, pure Java MiniQJMHACluster (two NameNodes and three JournalNodes). `TestAsyncPrepareRollbackBoundary`: **6 passed, 0 failures/errors/skips**: * Both default and single-transaction replay batches: image TXID matches finalized END on all three journals; real rollback removes the later namespace entry, and new writes survive restart. * Failover while saving a rollback image is deliberately paused: cancellation and catch-up preserve later edits and permit writes on the new active. * Restart during the paused save: cancellation, a new aligned rollback image and successful promotion. * Observer replay continues beyond PREPARE without waiting for an image. * A standby with checkpoints disabled continues replay and promotes successfully. PREPARE uses asynchronous tailing. During actual rollback the two boundary tests deliberately use finalized streaming, so the separate cache defect in HDFS-18022 cannot affect their outcome. No Multi-DC patch is included. Before the fix, independent reproducers observed image TXID 5 or 3 against finalized END 4, with zero of three aligned journals; rollback failed when asked to discard from inside a segment. ```sh mvn -pl hadoop-hdfs-project/hadoop-hdfs -am test \ -Dtest=TestAsyncPrepareRollbackBoundary \ -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false \ -Dmaven.test.failure.ignore=false ``` Existing regression suites on the same clean branch: **38 passed** (15 TestStandbyCheckpoints, 20 TestEditLogTailer across both parameter sets, 1 TestRollingUpgrade#testEditLogTailerRollingUpgrade, and 2 TestRollingUpgradeRollback QJM/HA-QJM cases). Together with the six added tests: **44 passed**. Surefire reports the tailer parameter sets as 10+10 in the Maven log but overwrites its final XML with an empty container; the two successful parameterized run summaries are retained. Checkstyle 11.1.0 using the repository configuration found no violations on changed lines or in the new test; pre-existing selected-file violations are not part of this patch. These are namespace/checkpoint tests with no DataNodes. Block durability, native/Linux behavior and distributed-cluster qualification are not claimed. ### For code changes: - [x] Does the title of this PR start with the corresponding JIRA issue id? - Object storage integration tests: not applicable. - New dependencies / dependency licensing: not applicable; none added. - LICENSE / NOTICE updates: not applicable. ### AI Tooling Contains content generated by OpenAI Codex. - [x] The PR includes the required AI tooling disclosure. - [x] My use of AI contributions follows the ASF legal policy https://www.apache.org/legal/generative-tooling.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
