[
https://issues.apache.org/jira/browse/HDDS-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115774#comment-18115774
]
Mohammadhani Fouladgar commented on HDDS-16418:
-----------------------------------------------
Before I start implementing, I'd like to align on two design points that the
description leaves open ("names to be finalized in review", "release notes if
defaults change"). Would appreciate input from folks more familiar with the SCM
HA / Inter-SCM path.
Context: Recon and follower SCMs download the leader's RocksDB checkpoint over
a single, potentially long-lived Inter-SCM gRPC connection. Today the only
tunable is {{{}ozone.scm.ha.grpc.deadline.interval{}}}; there is no keepalive
on either side, and the server's max-connection-idle is effectively infinite.
The reported symptoms (stalls, proxy idle timeouts, LB disconnects) come from
that idle, ping-less connection being dropped by intermediaries.
Question 1 — Default behavior of the new keepalive/idle settings
- (A) Preserve current behavior: new keys default to today's behavior (no
keepalive, infinite server idle); operators opt in. Purely additive, no
upgrade-time behavior change, no release note.
- (B) Opinionated defaults: ship non-zero defaults (e.g. keepalive ~30s) tuned
for long transfers. Fixes disconnects out of the box, but changes runtime
behavior on upgrade and needs a release note.
My preference: (A) — matches the "defaults preserve current behavior"
acceptance criterion; operators who hit the issue still get the knobs to fix it.
Question 2 — Config surface
- (A) Full, coherent set: client {{keepAliveTime}} / {{{}keepAliveTimeout{}}};
server {{maxConnectionIdle}} / {{keepAliveTime}} / {{keepAliveTimeout}} plus
{{permitKeepAliveTime}} / {{{}permitKeepAliveWithoutCalls{}}}. The "permit"
settings matter because a gRPC server rejects client pings it considers too
frequent (GOAWAY), so client keepalive isn't safe without them.
- (B) Minimal: only server {{maxConnectionIdle}} + client
{{{}keepAliveTime{}}}. Smaller, but it's the incomplete combination — enabling
client keepalive without the server permit settings can itself cause the server
to drop the connection.
My preference: (A) — the full set is the internally-consistent unit; the
minimal version isn't safe to enable on its own.
Proposed direction: (A) + (A) — preserve-current defaults with the full
client/server setting set, all under the ozone.scm.ha.grpc.* prefix. Naming
suggestions welcome. Does this sound right, or is there a preference for
opinionated defaults / a narrower surface?
> Make Inter-SCM gRPC timeouts and keepalive operator-tunable
> -----------------------------------------------------------
>
> Key: HDDS-16418
> URL: https://issues.apache.org/jira/browse/HDDS-16418
> Project: Apache Ozone
> Issue Type: Improvement
> Components: SCM HA
> Reporter: Wei-Chiu Chuang
> Assignee: Mohammadhani Fouladgar
> Priority: Major
> Labels: configuration, grpc, scm-ha
>
> h3. Problem
> SCM HA followers (and Recon) download the leader SCM RocksDB checkpoint over
> Inter-SCM gRPC during install snapshot. Transfers can run for a long time and
> depend on network stability, {{scm.db}} size, and disk throughput.
> Today operators have almost no way to tune gRPC timeout and connection
> behavior for this path:
> * *Client* ({{InterSCMGrpcClient}}): only
> {{ozone.scm.ha.grpc.deadline.interval}} exists for the stub RPC deadline. No
> documented client keepalive / idle settings on the channel. (See also
> HDDS-16417 for incorrect application of the deadline value.)
> * *Server* ({{InterSCMGrpcProtocolService}}): only port
> ({{ozone.scm.grpc.port}}) and {{maxInboundMessageSize}} are set. Keepalive,
> max connection idle, and related Netty server options use gRPC-java defaults
> and are not exposed in {{ozone-default.xml}}.
> By contrast, datanode container gRPC in {{XceiverServerGrpc}} sets explicit
> {{maxConnectionIdle}}, {{keepAliveTime}}, and {{keepAliveTimeout}} (currently
> hardcoded, not config either, but documented in code comments).
> Operators troubleshooting snapshot download stalls, proxy idle timeouts, or
> load-balancer disconnects cannot align Inter-SCM gRPC behavior with their
> environment without code changes.
> h3. Proposal
> Introduce *operator-tunable* Inter-SCM gRPC timeout and keepalive settings
> under a consistent {{ozone.scm.ha.grpc.*}} prefix (names to be finalized in
> review), for example:
> * Client RPC deadline (retain / clarify
> {{ozone.scm.ha.grpc.deadline.interval}})
> * Server: {{maxConnectionIdle}}, {{keepAliveTime}}, {{keepAliveTimeout}} (and
> any other knobs needed for long-lived streaming RPCs)
> * Client channel: matching keepalive / idle settings where applicable so
> client and server policies are coherent
> Defaults should remain safe for large checkpoint downloads (generous
> idle/deadline vs datanode chunk RPCs). Document each property in
> {{ozone-default.xml}} with units and interaction with TLS.
> h3. Affected code
> *
> {{hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcClient.java}}
> *
> {{hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcProtocolService.java}}
> *
> {{hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java}}
> * {{hadoop-ozone/recon/.../StorageContainerServiceProviderImpl.java}} (Recon
> uses {{InterSCMGrpcClient}})
> h3. Acceptance criteria
> * New config keys wired on both Inter-SCM gRPC client and server builders.
> * Defaults preserve current effective behavior (after fixing HDDS-16417
> deadline units).
> * Unit or integration test coverage that config values are read and applied
> (mock or small local server/client test acceptable).
> * Release note if defaults or operator-visible behavior changes.
> h3. Related
> * HDDS-16417 — deadline interval TimeUnit bug on the existing client property.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
