[
https://issues.apache.org/jira/browse/HDDS-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757674#comment-17757674
]
Ethan Rose commented on HDDS-3009:
----------------------------------
There have been many improvements in datanode disk usage checks in the last
year or two, this issue may not still be relevant. DU results could be stale
since we cache the exact value, which would lead the datanode to using a full
volume. However, the datanode should retry on a different volume if available.
It looks like that did not happen here. The issue may still be relevant but we
would have to test this scenario.
> Create Container failed using a disk which is full
> --------------------------------------------------
>
> Key: HDDS-3009
> URL: https://issues.apache.org/jira/browse/HDDS-3009
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Bharat Viswanadham
> Assignee: Hemant Kumar
> Priority: Major
> Labels: teragentest
>
> /dev/nvme2n1 985G 935G 0 100% /ozone-data
> /dev/nvme4n1 985G 453G 482G 49% /ozone-data1
>
> The container path used is full, but in creating container we have used the
> disk which is filled up 100% and failing during creating container path
> creation.
>
> *hdds.datanode.dir is configured with /ozone-data, /ozone*
>
>
> {code:java}
> 2020-02-13 10:58:01,097 ERROR
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil:
> Unable to create directory for metadata storage. Path:
> /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
> 2020-02-13 10:58:01,097 INFO
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler: Operation:
> CreateContainer , Trace ID: , Message: Container creation failed. Unable to
> create directory for metadata storage. Path:
> /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
> , Result: CONTAINER_INTERNAL_ERROR , StorageContainerException Occurred.
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Container creation failed. Unable to create directory for metadata storage.
> Path:
> /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:177)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:244)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:164)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:412)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:248)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:162)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:396)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:406)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$2(ContainerStateMachine.java:441)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Unable to create directory for metadata
> storage. Path:
> /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.createContainerMetaData(KeyValueContainerUtil.java:73)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:142)
> ... 12 more
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
