Attila Doroszlai created HDDS-16712:
---------------------------------------
Summary: Key creation fails intermittently in k8s after datanodes
restart
Key: HDDS-16712
URL: https://issues.apache.org/jira/browse/HDDS-16712
Project: Apache Ozone
Issue Type: Sub-task
Components: test
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai
Intermittent failure of key generator after datanodes are restarted in
{{kubernetes}}:
{code}
Ozone Client Key Generator | FAIL |
255 != 0
{code}
{code}
2026-10-04 10:57:23 INFO BaseFreonGenerator:317 - Executing test with prefix
ockgpost and number-of-tests 1
2026-10-04 10:57:23 INFO ProgressBar:167 - Progress: 0.00 % (0 out of 1)
2026-10-04 10:57:24 ERROR BaseFreonGenerator:231 - Error on executing task 0
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Unable to
allocate a container to the block of size: 268435456, replicationConfig:
RATIS/THREE. Waiting for one of pipelines to be OPEN failed. The input pipeline
IDs 521df264-010e-43d8-a0e2-dbbe8ac0f757 cannot be found
at
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:805)
at
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleSubmitRequestAndSCMSafeModeRetry(OzoneManagerProtocolClientSideTranslatorPB.java:2531)
at
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:792)
at
org.apache.hadoop.ozone.client.rpc.RpcClient.openOutputStream(RpcClient.java:1520)
at
org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:1435)
at
org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:1420)
at
org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:500)
at
org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:480)
at
org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.lambda$createKeyWithData$0(OzoneClientKeyGenerator.java:136)
at com.codahale.metrics.Timer.time(Timer.java:101)
at
org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKeyWithData(OzoneClientKeyGenerator.java:135)
at
org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKey(OzoneClientKeyGenerator.java:129)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:225)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.runTaskLoop(BaseFreonGenerator.java:206)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:180)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:171)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
at java.base/java.lang.Thread.run(Thread.java:1474)
2026-10-04 10:57:24 INFO ProgressBar:167 - Progress: 100.00 % (1 out of 1)
One or more freon test is failed.
2026-10-04 10:57:24 INFO metrics:373 - type=TIMER, name=key-create, count=1,
min=137.43811, max=137.43811, mean=137.43811, stddev=0.0, median=137.43811,
p75=137.43811, p95=137.43811, p98=137.43811, p99=137.43811, p999=137.43811,
mean_rate=4.5610858427462215, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second,
duration_unit=milliseconds
2026-10-04 10:57:24 INFO BaseFreonGenerator:75 - Total execution time (sec): 1
2026-10-04 10:57:24 INFO BaseFreonGenerator:75 - Failures: 1
{code}
- Datanodes register after the restart one by one, triggering pipeline creation
and closure. State becomes stable after the third DN is updated.
- If there are no candidate containers, WritableRatisContainerProvider checks
if any pipelines are in ALLOCATED state, and waits for them to become OPEN. If
they are then closed, new ones will be allocated. If they are closed in the
meantime, SCM gives up.
- Client only retries write, but not {{openKey}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
