Siyao Meng created HDDS-17236:
---------------------------------
Summary: Intermittent failure in ozonesecure public key and
certificate recovery test
Key: HDDS-17236
URL: https://issues.apache.org/jira/browse/HDDS-17236
Project: Apache Ozone
Issue Type: Sub-task
Components: docker-compose, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
Port 9888 is available on recon
find /data/metadata/recon/keys/public.pem -delete && find
/data/metadata/recon/certs/*.crt -delete succeed
Ncat: Could not resolve hostname "recon": Name or service not known. QUITTING.
Port 9888 is not available on recon yet
...
Timed out waiting on recon 9888 to become available
ERROR: Test execution of ozonesecure/test.sh is FAILED!!!!
{code}
h3. Root cause
{{ozonesecure/public-key-cert-recovery-test.sh}} restarts Recon (and later the
datanodes) with {{docker-compose stop recon}} followed by {{docker-compose
start recon}}. The Recon container log shows the SIGTERM and a clean shutdown,
but no second {{STARTUP_MSG}}, and the hostname stays unresolvable for the
whole 120s wait. So the container was never started again, yet {{docker-compose
start}} exited 0 (the script runs with {{set -e}} and kept going).
{{docker compose start}} (v2.38.2 on the runner) lists the project containers
once and skips any whose state is {{running}}. The Docker daemon wakes up the
{{docker stop}} caller in {{handleContainerExit}} via {{SetStopped}} before it
writes the new state to the replica that {{ContainerList}} reads
({{CheckpointTo(containersReplica)}}). If {{start}} lists containers in that
window, it still sees Recon as running and silently does nothing.
The same failure appears on master on 2026-07-13, 2026-07-15 and 2026-08-31
(ozone-build-results), always with the identical log pattern.
h3. Fix
Use {{docker-compose restart}} instead of {{stop}} + {{start}}. It calls
{{ContainerRestart}} for each container unconditionally, so the stop and start
happen inside the daemon with no stale state check in between.
- https://github.com/apache/ozone/actions/runs/39561350318/job/119114972849
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
