[
https://issues.apache.org/jira/browse/HDDS-15824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng reassigned HDDS-15824:
---------------------------------
Assignee: Siyao Meng
> Flaky debug replicas acceptance test: wait for RATIS THREE pipeline before
> writing keys
> ---------------------------------------------------------------------------------------
>
> Key: HDDS-15824
> URL: https://issues.apache.org/jira/browse/HDDS-15824
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> h2. Problem
> The {{ozonesecure-ha}} debug replicas acceptance suite ({{Ozone-Debug-Tests
> :: Test ozone debug CLI}} in {{smoketest/debug/ozone-debug-tests.robot}}) is
> flaky. Its Suite Setup ({{Write keys}}) intermittently fails with {{255 !=
> 0}}, which marks every test in the suite as failed even though no test logic
> ran.
> Root cause is in
> {{hadoop-ozone/dist/src/main/compose/common/replicas-test.sh}}. The
> container.db backup flow introduced by HDDS-14225 (the RocksDB 7.7.3 to
> 10.10.1 upgrade) stops all datanodes for a consistent backup, restarts them,
> and waits only for {{wait_for_datanode HEALTHY}} before running
> {{debug/ozone-debug-tests.robot}}. {{wait_for_datanode}} confirms the SCM
> heartbeat state (HEALTHY) but not that write pipelines have been
> re-established. {{Write keys}} then writes a default RATIS THREE key, a RATIS
> ONE key, a RATIS THREE key, and an EC rs-3-2 key; if it runs before an OPEN
> pipeline is available after the mass restart, the {{ozone sh key put}} exits
> non-zero (255) and the suite setup fails.
> Before HDDS-14225 the test backed up only datanode1 via {{docker cp}} without
> stopping any datanode, and did not restart datanodes before {{Write keys}},
> so this race did not exist.
> h2. Fix
> After the datanode restart and before running the debug test, wait (bounded
> at 180s) for at least one OPEN RATIS THREE pipeline via {{ozone admin
> pipeline list --state OPEN --type RATIS --replication THREE --json}}. RATIS
> ONE and EC writes are covered by all five datanodes already being HEALTHY;
> the RATIS THREE pipeline being OPEN is the gap this closes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
