[
https://issues.apache.org/jira/browse/HDDS-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757635#comment-17757635
]
Siyao Meng commented on HDDS-3009:
----------------------------------
The question to ask is, what is the expected behavior?
The easiest to do right now might be to just throw "disk is full" error
gracefully back to the client. and indicate which DN mount point is full.
Later we could figure out whether the client should ask for a new pipeline. Or,
add DN mount point usage metrics to rack awareness so it can hopefully avoid
those (almost) full DNs in the first place.
> Create Container failed using a disk which is full
> --------------------------------------------------
>
> Key: HDDS-3009
> URL: https://issues.apache.org/jira/browse/HDDS-3009
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Bharat Viswanadham
> Assignee: Hemant Kumar
> Priority: Major
> Labels: teragentest
>
> /dev/nvme2n1 985G 935G 0 100% /ozone-data
> /dev/nvme4n1 985G 453G 482G 49% /ozone-data1
>
> The container path used is full, but in creating container we have used the
> disk which is filled up 100% and failing during creating container path
> creation.
>
> *hdds.datanode.dir is configured with /ozone-data, /ozone*
>
>
> {code:java}
> 2020-02-13 10:58:01,097 ERROR
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil:
> Unable to create directory for metadata storage. Path:
> /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
> 2020-02-13 10:58:01,097 INFO
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler: Operation:
> CreateContainer , Trace ID: , Message: Container creation failed. Unable to
> create directory for metadata storage. Path:
> /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
> , Result: CONTAINER_INTERNAL_ERROR , StorageContainerException Occurred.
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Container creation failed. Unable to create directory for metadata storage.
> Path:
> /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:177)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:244)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:164)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:412)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:248)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:162)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:396)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:406)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$2(ContainerStateMachine.java:441)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Unable to create directory for metadata
> storage. Path:
> /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.createContainerMetaData(KeyValueContainerUtil.java:73)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:142)
> ... 12 more
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
