[
https://issues.apache.org/jira/browse/HDDS-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125834#comment-18125834
]
Adweta Ojha commented on HDDS-16712:
------------------------------------
Hi, I’m interested in working on this issue.
I went through the failure path and the discussion above. My understanding is
that the Kubernetes timing change would only hide the underlying problem, and
that the fix should address how SCM handles this transient pipeline churn
during datanode re-registration.
I’d like to investigate the SCM-side behavior, particularly the case where
{{WritableRatisContainerProvider}} is waiting on ALLOCATED pipeline IDs that
are subsequently removed while replacement pipelines are being created.
Is this issue available for me to pick up? Also, should I focus on a bounded
SCM-side recovery/re-evaluation here, or would you prefer the
{{KEY_ALLOCATION_ERROR}} / client-retry direction mentioned above?
> Key creation fails intermittently in k8s after datanodes restart
> ----------------------------------------------------------------
>
> Key: HDDS-16712
> URL: https://issues.apache.org/jira/browse/HDDS-16712
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Client, Ozone Manager, SCM
> Reporter: Attila Doroszlai
> Priority: Major
>
> Intermittent failure of key generator after datanodes are restarted in
> {{kubernetes}}:
> {code}
> Ozone Client Key Generator | FAIL |
> 255 != 0
> {code}
> {code}
> 2026-10-04 10:57:23 INFO BaseFreonGenerator:317 - Executing test with prefix
> ockgpost and number-of-tests 1
> 2026-10-04 10:57:23 INFO ProgressBar:167 - Progress: 0.00 % (0 out of 1)
> 2026-10-04 10:57:24 ERROR BaseFreonGenerator:231 - Error on executing task 0
> INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Unable to
> allocate a container to the block of size: 268435456, replicationConfig:
> RATIS/THREE. Waiting for one of pipelines to be OPEN failed. The input
> pipeline IDs 521df264-010e-43d8-a0e2-dbbe8ac0f757 cannot be found
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:805)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleSubmitRequestAndSCMSafeModeRetry(OzoneManagerProtocolClientSideTranslatorPB.java:2531)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:792)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.openOutputStream(RpcClient.java:1520)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:1435)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:1420)
> at
> org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:500)
> at
> org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:480)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.lambda$createKeyWithData$0(OzoneClientKeyGenerator.java:136)
> at com.codahale.metrics.Timer.time(Timer.java:101)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKeyWithData(OzoneClientKeyGenerator.java:135)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKey(OzoneClientKeyGenerator.java:129)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:225)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.runTaskLoop(BaseFreonGenerator.java:206)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:180)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:171)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
> at java.base/java.lang.Thread.run(Thread.java:1474)
> 2026-10-04 10:57:24 INFO ProgressBar:167 - Progress: 100.00 % (1 out of 1)
> One or more freon test is failed.
> 2026-10-04 10:57:24 INFO metrics:373 - type=TIMER, name=key-create, count=1,
> min=137.43811, max=137.43811, mean=137.43811, stddev=0.0, median=137.43811,
> p75=137.43811, p95=137.43811, p98=137.43811, p99=137.43811, p999=137.43811,
> mean_rate=4.5610858427462215, m1=0.0, m5=0.0, m15=0.0,
> rate_unit=events/second, duration_unit=milliseconds
> 2026-10-04 10:57:24 INFO BaseFreonGenerator:75 - Total execution time (sec):
> 1
> 2026-10-04 10:57:24 INFO BaseFreonGenerator:75 - Failures: 1
> {code}
> - Datanodes register after the restart one by one, triggering pipeline
> creation and closure. State becomes stable after the third DN is updated.
> - If there are no candidate containers, WritableRatisContainerProvider checks
> if any pipelines are in ALLOCATED state, and waits for them to become OPEN.
> If they are then closed, new ones will be allocated. If they are closed in
> the meantime, SCM gives up.
> - Client only retries write, but not {{openKey}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
