[
https://issues.apache.org/jira/browse/HDDS-17325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17325:
------------------------------
Epic Link: HDDS-17321
> GrpcOmTransport has no call deadline and ignores interrupts, so a stuck OM
> blocks the client indefinitely
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17325
> URL: https://issues.apache.org/jira/browse/HDDS-17325
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Client
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> When {{ozone.om.transport.class}} selects the gRPC transport (not the default
> since HDDS-9446, but used for S3 Gateway in some deployments),
> {{GrpcOmTransport}}:
> * calls {{submitRequest}} on a blocking stub without a deadline
> ({{withDeadlineAfter}}), and there is no config key for a call timeout. If
> the OM accepts the request but never answers (e.g. a stuck handler), the call
> blocks with no limit, and the failover and retry logic is never reached. A
> half-open connection is only closed by TCP, after 15 minutes or more.
> * sleeps between retries inside {{catch (Exception e)}}, which only logs, so
> an interrupt is lost and the retry continues.
> For S3 Gateway, each request that reaches the stuck OM holds a gateway thread
> until the gateway is restarted.
> Proposed fix: set a per-call deadline from a new config key (a
> {{StatusRuntimeException}} already goes through the retry and failover
> policy), and restore the interrupt flag and stop retrying when the thread is
> interrupted.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
