Ethan Rose created HDDS-17618:
---------------------------------
Summary: Recon ContainerKeyMapper FSO/OBS tasks can truncate each
other's data during reprocess
Key: HDDS-17618
URL: https://issues.apache.org/jira/browse/HDDS-17618
Project: Apache Ozone
Issue Type: Bug
Reporter: Ethan Rose
Attachments: junit-ozone-integration-test-recon.zip
This RCA was determined by AI while examining an [intermittent test
timeout|https://github.com/sodonnel/hadoop-ozone/actions/runs/41122784678/job/124310237831]
in TestReconContainerEndpoint.
----
TestReconContainerEndpoint.testContainerEndpointForOBSBucket fails
intermittently in CI with a TimeoutException in
ReconOmMetaManagerTestUtils.waitUntilReconKeyCounts. The test waits 90s for
Recon's container-key index to show the key it wrote, and the index never does.
It passes when run on its own.
Observed in the CI logs
- Recon's OM DB is bootstrapped from a full OM snapshot, which triggers
reInitializeTasks (reason MANUAL_TRIGGER).
- ContainerKeyMapperTaskFSO and ContainerKeyMapperTaskOBS both log "Reprocess
completed" and the reinitialization succeeds.
- Recon's OM DB then stays at the snapshot's sequence number. Every later OM
sync returns 0 updates, so there is no delta event that could re-add the key.
- In the failing run, OBS finished its reprocess first, hit the "last task"
branch, and logged "Cleared shared map and reset initialization flag" at
21:19:19,159. FSO's initializeKeyContainerTable log line followed at
21:19:19,160, after that reset.
Proposed root cause (hypothesis, not directly confirmed)
ContainerKeyMapperHelper guards a one-time init (truncating the container-key
tables and clearing the shared count map) with the static flag
ReconConstants.CONTAINER_KEY_MAPPER_INITIALIZED. Two tasks (FSO and OBS) share
that flag and an ACTIVE_TASK_COUNT.
The task that brings ACTIVE_TASK_COUNT to 0 resets the flag at the end of
reprocess(). Whether that is safe depends on both tasks having incremented the
count before either finishes. If one task finishes before the other has started
its init, then:
1. The finishing task sees the count reach 0 and resets the flag.
2. The slower task starts, sees the flag as false, and runs the init again.
3. That init truncates the container-key tables and wipes the data the first
task already wrote.
I did not find a log line that shows the second truncation directly. The timing
a Which task ends up with its data wiped depends on scheduling, which fits an
intermittent failure.
Proposed fix
In ContainerKeyMapperHelper.reprocess, stop resetting
CONTAINER_KEY_MAPPER_INITIALIZED from the last-finishing task. The flag is
already reset once per reprocess cycle by
ReconConstants.resetTableTruncatedFlags(), which
ReconTaskControllerImpl.reInitiamitting tasks. The test-only
clearSharedContainerCountMap() still resets it as
well. The shared count map is still cleared when the last task finishes.
Risk to review
Code that calls the FSO or OBS task's reprocess() repeatedly, outside
reInitializing resetTableTruncatedFlags(), would now skip truncation on later
runs. I found no
such caller in main code. Existing unit tests that do this still pass.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
