[
https://issues.apache.org/jira/browse/HDDS-17360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reassigned HDDS-17360:
--------------------------------------
Assignee: Andrey Yarovoy
> EC pipelines remain eligible for allocation after writes fail with
> CONTAINER_MISSING
> ------------------------------------------------------------------------------------
>
> Key: HDDS-17360
> URL: https://issues.apache.org/jira/browse/HDDS-17360
> Project: Apache Ozone
> Issue Type: Bug
> Components: EC
> Reporter: Wei-Chiu Chuang
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h2. Description
> An EC write can fail because a datanode has marked the target container as
> missing:
> {code}
> EC stripe write failed: S S F S S
> StorageContainerException: ContainerID ... has been lost and cannot be
> recreated on this DataNode
> {code}
> The EC client excludes the failed pipeline and applicable failed datanodes,
> then retries with another block group. However, this failure path does not
> initiate closure of the affected container or pipeline in SCM. If SCM still
> considers them OPEN, other writers can receive allocations targeting the same
> affected container and encounter the same failure.
> This was observed while writing keys with RS(3,2). The original cause of the
> missing container replicas has not been established.
> h2. Expected behavior
> A confirmed CONTAINER_MISSING response for an OPEN EC container should lead
> to reconciliation in SCM so that the affected container/pipeline stops
> receiving new allocations. SCM should initiate the existing closure workflow
> when appropriate.
> h2. Actual behavior
> The client avoids the failed pipeline through its exclusion list, but the
> pipeline can remain available to other writers until a separate SCM lifecycle
> or health-handling path retires it.
> h2. Comparison with Ratis
> In the investigated source, Ratis treats CONTAINER_MISSING during
> state-machine writes as a fatal error, marking the state machine unhealthy
> and closing the local Raft division. Ratis also has server-side paths that
> report pipeline-close actions to SCM.
> EC writes use the gRPC container service and bypass this Ratis failure
> handling. The error is returned to the client without an equivalent
> pipeline-close action. This distinction does not apply to every Ratis write
> exception: expected container-closure responses are exempted from the fatal
> state-machine path.
> h2. Relevant code
> * ECKeyOutputStream.excludePipelineAndFailedDN()
> * WritableECContainerProvider.getContainer()
> * HddsDispatcher.dispatch()
> * GrpcXceiverService
> * ContainerStateMachine.handleCommandResult() and write()
> h2. Impact
> A container-loss event affecting many OPEN EC containers may cause repeated
> write failures across clients, additional block allocations and retries, and
> increased write latency or eventual write failure.
> h2. Suggested scope
> Handle confirmed missing-container failures specifically, rather than closing
> EC pipelines for every write exception. Validate the affected
> container/pipeline state in SCM and reuse the existing closure workflow.
> Validation should cover concurrent writers, repeated failure reports,
> allocation/closure races, SCM failover, and readability and reconstruction of
> previously acknowledged data.
> h2. Environment and reproduction
> * Replication: EC RS(3,2)
> * Deployed Ozone version: not yet established
> * Observed in deployment; controlled reproduction pending
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
