[
https://issues.apache.org/jira/browse/HDDS-15791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aswin Shakil updated HDDS-15791:
--------------------------------
Description:
This issue has been identified by [~deveshsingh] and [~umamahesh]
During EC offline reconstruction, the coordinator creates a *RECOVERING*
replica on a spare DataNode and writes blocks over time. If reconstruction
stalls (e.g. slow block-group processing) longer than
`ozone.recovering.container.timeout` (~20m, measured from creation, not last
write), the DN scrubber marks the replica *UNHEALTHY* and SCM force-deletes it.
The coordinator may still be running. The next `WriteChunk` finds no container;
`HddsDispatcher` auto-creates a new OPEN container instead of failing. The
coordinator continues and closes that replica shortly after, leaving a partial
container (fewer blocks/chunks than the reference) that SCM can accept as
*CLOSED* — silent data loss.
*Logic flow*
{code:java}
Coordinator Phase 1
createRecoveringContainer → RECOVERING on spare DN
(slow / sleep — no writes for extended period)
StaleRecoveringContainerScrubbingService
creationTime + timeout exceeded → mark UNHEALTHY
SCM / Replication Manager
force-delete UNHEALTHY replica → DELETED on spare DN
Coordinator resumes writes
container == null
HddsDispatcher auto-create → OPEN (not RECOVERING)
Coordinator completes
closeContainer → CLOSING / CLOSED
Result: CLOSED replica with partial blocks/chunks at EC index → data loss
{code}
was:
This issue has been identified by [~deveshsingh] and [~umamahesh]
During EC offline reconstruction, the coordinator creates a *RECOVERING*
replica on a spare DataNode and writes blocks over time. If reconstruction
stalls (e.g. slow block-group processing) longer than
`ozone.recovering.container.timeout` (~20m, measured from creation, not last
write), the DN scrubber marks the replica *UNHEALTHY* and SCM force-deletes it.
The coordinator may still be running. The next `WriteChunk` or `PutBlock` finds
no container; `HddsDispatcher` auto-creates a new OPEN container instead of
failing. The coordinator continues and closes that replica shortly after,
leaving a partial container (fewer blocks/chunks than the reference) that SCM
can accept as *CLOSED* — silent data loss.
*Logic flow*
{code:java}
Coordinator Phase 1
createRecoveringContainer → RECOVERING on spare DN
(slow / sleep — no writes for extended period)
StaleRecoveringContainerScrubbingService
creationTime + timeout exceeded → mark UNHEALTHY
SCM / Replication Manager
force-delete UNHEALTHY replica → DELETED on spare DN
Coordinator resumes writes
container == null
HddsDispatcher auto-create → OPEN (not RECOVERING)
Coordinator completes
closeContainer → CLOSING / CLOSED
Result: CLOSED replica with partial blocks/chunks at EC index → data loss
{code}
> EC reconstructed RECOVERING container while still idle can be deleted by SCM
> and recreated as new partial OPEN container
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-15791
> URL: https://issues.apache.org/jira/browse/HDDS-15791
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Aswin Shakil
> Assignee: Aswin Shakil
> Priority: Major
> Labels: pull-request-available
>
> This issue has been identified by [~deveshsingh] and [~umamahesh]
> During EC offline reconstruction, the coordinator creates a *RECOVERING*
> replica on a spare DataNode and writes blocks over time. If reconstruction
> stalls (e.g. slow block-group processing) longer than
> `ozone.recovering.container.timeout` (~20m, measured from creation, not last
> write), the DN scrubber marks the replica *UNHEALTHY* and SCM force-deletes
> it.
> The coordinator may still be running. The next `WriteChunk` finds no
> container; `HddsDispatcher` auto-creates a new OPEN container instead of
> failing. The coordinator continues and closes that replica shortly after,
> leaving a partial container (fewer blocks/chunks than the reference) that SCM
> can accept as *CLOSED* — silent data loss.
> *Logic flow*
> {code:java}
> Coordinator Phase 1
> createRecoveringContainer → RECOVERING on spare DN
> (slow / sleep — no writes for extended period)
> StaleRecoveringContainerScrubbingService
> creationTime + timeout exceeded → mark UNHEALTHY
> SCM / Replication Manager
> force-delete UNHEALTHY replica → DELETED on spare DN
> Coordinator resumes writes
> container == null
> HddsDispatcher auto-create → OPEN (not RECOVERING)
> Coordinator completes
> closeContainer → CLOSING / CLOSED
> Result: CLOSED replica with partial blocks/chunks at EC index → data loss
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
