smengcl opened a new pull request, #10722: URL: https://github.com/apache/ozone/pull/10722
Generated-by: Claude Code (Opus 4.8, 1M context) ## What changes were proposed in this pull request? The `ozonesecure-ha` debug replicas acceptance suite (`Ozone-Debug-Tests :: Test ozone debug CLI`) is flaky. Its `Suite Setup` (`Write keys`) intermittently fails with `255 != 0`, which marks every test in the suite as failed even though no test logic ran. Root cause is in `hadoop-ozone/dist/src/main/compose/common/replicas-test.sh`. The container.db backup flow introduced by HDDS-14225 (the RocksDB 7.7.3 to 10.10.1 upgrade, #9813) now stops all datanodes for a consistent backup, restarts them, and waits only for `wait_for_datanode HEALTHY` before running `debug/ozone-debug-tests.robot`. `wait_for_datanode` confirms the SCM heartbeat state (`HEALTHY`) but not that write pipelines have been re-established. `Write keys` then writes a default RATIS THREE key, a RATIS ONE key, a RATIS THREE key, and an EC rs-3-2 key; if it runs before an OPEN pipeline is available after the mass restart, the `ozone sh key put` exits non-zero (255) and the suite setup fails. Before HDDS-14225 the test backed up only `datanode1` via `docker cp` without stopping any datanode, and did not restart datanodes before `Write keys`, so this race did not exist. ### Change After the datanode restart and before running the debug test, wait (bounded at 180s) for at least one OPEN RATIS THREE pipeline via `ozone admin pipeline list --state OPEN --type RATIS --replication THREE --json`. RATIS ONE and EC writes are covered by all five datanodes already being `HEALTHY`; the RATIS THREE pipeline being OPEN is the gap this closes. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-15824 ## How was this patch tested? `bash -n` passes on the modified script. The change is exercised by the existing `acceptance (tools)` job, which runs `ozonesecure-ha/test-debug-tools.sh`. Not reproduced locally (requires a full `-Pdist` build plus a docker-compose acceptance run). Frequency was assessed from master history: `test-debug-tools.sh` passed in all 14 post-upgrade `acceptance (tools)` master runs (2026-07-08 to 07-10); the only archived `acceptance-tools` master failures in that window were `test-repair-tools.sh`, a separate flake. The failure was observed on an unrelated PR run (2026-07-10) that does not touch this write path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
