[
https://issues.apache.org/jira/browse/HDDS-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124856#comment-18124856
]
Siyao Meng commented on HDDS-16712:
-----------------------------------
Hit this again on master:
https://github.com/apache/ozone/actions/runs/37613766011/job/112771340382
I think the cause is that both waits after the restart are satisfied before the
new datanode pods register with SCM:
* {{wait_for_startup}} matches the old "SCM exiting safe mode." line, since SCM
was not restarted.
* {{wait_for_pipeline}} matches the RATIS/THREE pipeline from before the
restart. SCM still lists it as OPEN until the datanodes re-register with their
new pod IP.
Each re-registration then makes {{NodeAddressUpdateHandler}} close all
pipelines of that datanode, including ones created after the previous
re-registration. From the SCM log of this run:
{code}
11:42:45 NodeAddressUpdateHandler:55 - Closing stale pipelines for datanode:
38ca123c... (datanode-0 ... /10.42.0.79)
11:42:48 BackgroundPipelineCreator:227 - Created new pipeline Pipeline{ Id:
34f043ee-16e8-42ca-8d00-8ba27779289e, ...
11:42:48 NodeAddressUpdateHandler:55 - Closing stale pipelines for datanode:
e05e1448... (datanode-2 ... /10.42.0.81)
11:42:48 PipelineManagerImpl:539 - Closing the stale pipeline:
Pipeline-34f043ee-16e8-42ca-8d00-8ba27779289e
11:42:49 WritableRatisContainerProvider:113 - Waiting for one of pipelines
[Pipeline-34f043ee-16e8-42ca-8d00-8ba27779289e] to be OPEN failed.
{code}
{{ockg}} for {{ockgpost}} started at 11:42:47, in the middle of this.
Restarting datanodes one at a time (as in your HDDS-16712 branch) seems to
leave the same window per iteration: {{wait_for_pipeline}} can still pass on
the old pipeline before the restarted datanode re-registers.
A possible fix: record OPEN RATIS/THREE pipeline IDs before deleting the pods,
then wait for one not in that set. A new pipeline can only become OPEN after
all of its members reported it, i.e. after the new pods re-registered, so it is
not closed by a later address update.
https://github.com/smengcl/hadoop-ozone/commit/7c0b234988b
(Claude did the analysis)
> Key creation fails intermittently in k8s after datanodes restart
> ----------------------------------------------------------------
>
> Key: HDDS-16712
> URL: https://issues.apache.org/jira/browse/HDDS-16712
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test
> Reporter: Attila Doroszlai
> Assignee: Attila Doroszlai
> Priority: Major
>
> Intermittent failure of key generator after datanodes are restarted in
> {{kubernetes}}:
> {code}
> Ozone Client Key Generator | FAIL |
> 255 != 0
> {code}
> {code}
> 2026-10-04 10:57:23 INFO BaseFreonGenerator:317 - Executing test with prefix
> ockgpost and number-of-tests 1
> 2026-10-04 10:57:23 INFO ProgressBar:167 - Progress: 0.00 % (0 out of 1)
> 2026-10-04 10:57:24 ERROR BaseFreonGenerator:231 - Error on executing task 0
> INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Unable to
> allocate a container to the block of size: 268435456, replicationConfig:
> RATIS/THREE. Waiting for one of pipelines to be OPEN failed. The input
> pipeline IDs 521df264-010e-43d8-a0e2-dbbe8ac0f757 cannot be found
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:805)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleSubmitRequestAndSCMSafeModeRetry(OzoneManagerProtocolClientSideTranslatorPB.java:2531)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:792)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.openOutputStream(RpcClient.java:1520)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:1435)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:1420)
> at
> org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:500)
> at
> org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:480)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.lambda$createKeyWithData$0(OzoneClientKeyGenerator.java:136)
> at com.codahale.metrics.Timer.time(Timer.java:101)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKeyWithData(OzoneClientKeyGenerator.java:135)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKey(OzoneClientKeyGenerator.java:129)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:225)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.runTaskLoop(BaseFreonGenerator.java:206)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:180)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:171)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
> at java.base/java.lang.Thread.run(Thread.java:1474)
> 2026-10-04 10:57:24 INFO ProgressBar:167 - Progress: 100.00 % (1 out of 1)
> One or more freon test is failed.
> 2026-10-04 10:57:24 INFO metrics:373 - type=TIMER, name=key-create, count=1,
> min=137.43811, max=137.43811, mean=137.43811, stddev=0.0, median=137.43811,
> p75=137.43811, p95=137.43811, p98=137.43811, p99=137.43811, p999=137.43811,
> mean_rate=4.5610858427462215, m1=0.0, m5=0.0, m15=0.0,
> rate_unit=events/second, duration_unit=milliseconds
> 2026-10-04 10:57:24 INFO BaseFreonGenerator:75 - Total execution time (sec):
> 1
> 2026-10-04 10:57:24 INFO BaseFreonGenerator:75 - Failures: 1
> {code}
> - Datanodes register after the restart one by one, triggering pipeline
> creation and closure. State becomes stable after the third DN is updated.
> - If there are no candidate containers, WritableRatisContainerProvider checks
> if any pipelines are in ALLOCATED state, and waits for them to become OPEN.
> If they are then closed, new ones will be allocated. If they are closed in
> the meantime, SCM gives up.
> - Client only retries write, but not {{openKey}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
