[
https://issues.apache.org/jira/browse/HDDS-16373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Meng-Shuan Tsai updated HDDS-16373:
-----------------------------------
Status: Patch Available (was: Open)
> InterSCMGrpcClient treats the checkpoint download deadline as seconds instead
> of milliseconds
> ---------------------------------------------------------------------------------------------
>
> Key: HDDS-16373
> URL: https://issues.apache.org/jira/browse/HDDS-16373
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Meng-Shuan Tsai
> Assignee: Meng-Shuan Tsai
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> {{InterSCMGrpcClient}} reads {{ozone.scm.ha.grpc.deadline.interval}} in
> milliseconds, then passes the same numeric value to gRPC as seconds. The
> default 30-minute deadline therefore becomes ~20 days (1000x).
> This is a bug in SCM HA snapshot catch-up and Recon SCM DB sync. Successful
> downloads that finish within 30 minutes are unaffected.
> h2. Root cause
> Config default is 30 minutes in both places:
> * {{{}ozone-default.xml{}}}: {{ozone.scm.ha.grpc.deadline.interval=30m}}
> * {{ScmConfigKeys.OZONE_SCM_HA_GRPC_DEADLINE_INTERVAL_DEFAULT = 30 * 60 *
> 1000L}}
> The client converts that to milliseconds correctly:
> {code:java}
> timeout = conf.getTimeDuration(
> ScmConfigKeys.OZONE_SCM_HA_GRPC_DEADLINE_INTERVAL,
> ScmConfigKeys.OZONE_SCM_HA_GRPC_DEADLINE_INTERVAL_DEFAULT,
> TimeUnit.MILLISECONDS); // 1,800,000 ms
> {code}
> Then applies it with the wrong unit:
> {code:java}
> client = InterSCMProtocolServiceGrpc.newStub(channel)
> .withDeadlineAfter(timeout, TimeUnit.SECONDS); // 1,800,000 seconds ≈ 20
> days
> {code}
> {{timeout}} is {{final}} and is not converted between the two calls.
> Source:
> *
> [InterSCMGrpcClient.java|https://github.com/apache/ozone/blob/master/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcClient.java]
> *
> [ScmConfigKeys.java|https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java]
> h2. Impact
> Two production callers create this client and wait with
> {{CompletableFuture.get()}} (no timeout):
> # SCM follower install-snapshot: {{SCMStateMachine}} → {{SCMHAManagerImpl}}
> → {{SCMSnapshotProvider}}
> # Recon SCM DB sync: {{StorageContainerServiceProviderImpl}}
> If the gRPC stream stay pending, the 30-minute deadline does not fire.
> Catch-up / sync is delayed until some other completion signal, an interrupt,
> or the inflated gRPC deadline (~20 days).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
