Aswin Shakil created HDDS-15791:
-----------------------------------
Summary: EC reconstructed RECOVERING container while still idle
can be deleted by SCM and recreated as new partial OPEN container
Key: HDDS-15791
URL: https://issues.apache.org/jira/browse/HDDS-15791
Project: Apache Ozone
Issue Type: Bug
Reporter: Aswin Shakil
Assignee: Aswin Shakil
This issue has been identified by [~deveshsingh] and [~umamahesh]
During EC offline reconstruction, the coordinator creates a *RECOVERING*
replica on a spare DataNode and writes blocks over time. If reconstruction
stalls (e.g. slow block-group processing) longer than
`ozone.recovering.container.timeout` (~20m, measured from creation, not last
write), the DN scrubber marks the replica *UNHEALTHY* and SCM force-deletes it.
The coordinator may still be running. The next `WriteChunk` or `PutBlock` finds
no container; `HddsDispatcher` auto-creates a new OPEN container instead of
failing. The coordinator continues and closes that replica shortly after,
leaving a partial container (fewer blocks/chunks than the reference) that SCM
can accept as *CLOSED* — silent data loss.
*Logic flow*
{code:java}
Coordinator Phase 1
createRecoveringContainer → RECOVERING on spare DN
(slow / sleep — no writes for extended period)
StaleRecoveringContainerScrubbingService
creationTime + timeout exceeded → mark UNHEALTHY
SCM / Replication Manager
force-delete UNHEALTHY replica → DELETED on spare DN
Coordinator resumes writes
container == null
HddsDispatcher auto-create → OPEN (not RECOVERING)
Coordinator completes
closeContainer → CLOSING / CLOSED
Result: CLOSED replica with partial blocks/chunks at EC index → data loss
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
