[
https://issues.apache.org/jira/browse/HDDS-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18117823#comment-18117823
]
Mohammadhani Fouladgar edited comment on HDDS-16418 at 9/21/26 6:10 PM:
------------------------------------------------------------------------
*Proposed config keys*
All under the existing ozone.scm.ha.grpc.* prefix, defined as constants in
ScmConfigKeys.java next to OZONE_SCM_HA_GRPC_DEADLINE_INTERVAL
(hadoop-hdds/common/.../ScmConfigKeys.java), read with
conf.getTimeDuration(...) / conf.getBoolean(...) following the existing
deadline pattern.
*Client channel* (InterSCMGrpcClient):
* ozone.scm.ha.grpc.client.keepalive.time = 5m
* ozone.scm.ha.grpc.client.keepalive.timeout = 30s
*Server* (InterSCMGrpcProtocolService):
* ozone.scm.ha.grpc.server.max.connection.idle = 15m
* ozone.scm.ha.grpc.server.keepalive.time = 5m
* ozone.scm.ha.grpc.server.keepalive.timeout = 30s
* ozone.scm.ha.grpc.server.permit.keepalive.time = 5m
* ozone.scm.ha.grpc.server.permit.keepalive.without.calls = true
Rationale for the "permit" keys: a gRPC server rejects client pings it
considers too frequent (GOAWAY). permitKeepAliveTime set to the client
keepalive cadence (5m) ensures the client's pings are accepted;
permitKeepAliveWithoutCalls=true avoids tearing the channel down between the
request and the first data frame. maxConnectionIdle only counts when no RPC is
active, so it does not abort an in-progress streaming download.
was (Author: JIRAUSER313318):
*Proposed config keys*
All under the existing ozone.scm.ha.grpc.* prefix, defined as constants in
ScmConfigKeys.java next to OZONE_SCM_HA_GRPC_DEADLINE_INTERVAL
(hadoop-hdds/common/.../ScmConfigKeys.java), read with
conf.getTimeDuration(...) / conf.getBoolean(...) following the existing
deadline pattern.
*Client channel* (InterSCMGrpcClient):
* ozone.scm.ha.grpc.client.keepalive.time = 5m
* ozone.scm.ha.grpc.client.keepalive.timeout = 30s
*Server* (InterSCMGrpcProtocolService):
* ozone.scm.ha.grpc.server.max.connection.idle = 15m
* ozone.scm.ha.grpc.server.keepalive.time = 5m
* ozone.scm.ha.grpc.server.keepalive.timeout = 30s
* ozone.scm.ha.grpc.server.permit.keepalive.time = 5m
* ozone.scm.ha.grpc.server.permit.keepalive.without.calls = true
Rationale for the "permit" keys: a gRPC server rejects client pings it
considers too frequent (GOAWAY). permitKeepAliveTime set to the client
keepalive cadence (5m) ensures the client's pings are accepted;
permitKeepAliveWithoutCalls=true avoids tearing the channel down between the
request and the first data frame. maxConnectionIdle only counts when no RPC is
active, so it does not abort an in-progress streaming download.
> Make Inter-SCM gRPC timeouts and keepalive operator-tunable
> -----------------------------------------------------------
>
> Key: HDDS-16418
> URL: https://issues.apache.org/jira/browse/HDDS-16418
> Project: Apache Ozone
> Issue Type: Improvement
> Components: SCM HA
> Reporter: Wei-Chiu Chuang
> Assignee: Mohammadhani Fouladgar
> Priority: Major
> Labels: configuration, grpc, pull-request-available, scm-ha
>
> h3. Problem
> SCM HA followers (and Recon) download the leader SCM RocksDB checkpoint over
> Inter-SCM gRPC during install snapshot. Transfers can run for a long time and
> depend on network stability, {{scm.db}} size, and disk throughput.
> Today operators have almost no way to tune gRPC timeout and connection
> behavior for this path:
> * *Client* ({{InterSCMGrpcClient}}): only
> {{ozone.scm.ha.grpc.deadline.interval}} exists for the stub RPC deadline. No
> documented client keepalive / idle settings on the channel. (See also
> HDDS-16417 for incorrect application of the deadline value.)
> * *Server* ({{InterSCMGrpcProtocolService}}): only port
> ({{ozone.scm.grpc.port}}) and {{maxInboundMessageSize}} are set. Keepalive,
> max connection idle, and related Netty server options use gRPC-java defaults
> and are not exposed in {{ozone-default.xml}}.
> By contrast, datanode container gRPC in {{XceiverServerGrpc}} sets explicit
> {{maxConnectionIdle}}, {{keepAliveTime}}, and {{keepAliveTimeout}} (currently
> hardcoded, not config either, but documented in code comments).
> Operators troubleshooting snapshot download stalls, proxy idle timeouts, or
> load-balancer disconnects cannot align Inter-SCM gRPC behavior with their
> environment without code changes.
> h3. Proposal
> Introduce *operator-tunable* Inter-SCM gRPC timeout and keepalive settings
> under a consistent {{ozone.scm.ha.grpc.*}} prefix (names to be finalized in
> review), for example:
> * Client RPC deadline (retain / clarify
> {{ozone.scm.ha.grpc.deadline.interval}})
> * Server: {{maxConnectionIdle}}, {{keepAliveTime}}, {{keepAliveTimeout}} (and
> any other knobs needed for long-lived streaming RPCs)
> * Client channel: matching keepalive / idle settings where applicable so
> client and server policies are coherent
> Defaults should remain safe for large checkpoint downloads (generous
> idle/deadline vs datanode chunk RPCs). Document each property in
> {{ozone-default.xml}} with units and interaction with TLS.
> h3. Affected code
> *
> {{hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcClient.java}}
> *
> {{hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcProtocolService.java}}
> *
> {{hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java}}
> * {{hadoop-ozone/recon/.../StorageContainerServiceProviderImpl.java}} (Recon
> uses {{InterSCMGrpcClient}})
> h3. Acceptance criteria
> * New config keys wired on both Inter-SCM gRPC client and server builders.
> * Defaults preserve current effective behavior (after fixing HDDS-16417
> deadline units).
> * Unit or integration test coverage that config values are read and applied
> (mock or small local server/client test acceptable).
> * Release note if defaults or operator-visible behavior changes.
> h3. Related
> * HDDS-16417 — deadline interval TimeUnit bug on the existing client property.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
