[
https://issues.apache.org/jira/browse/HDDS-12422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17930974#comment-17930974
]
Sammi Chen commented on HDDS-12422:
-----------------------------------
I'm not sure we need this. When container export fails due to disk full on one
datanode, SCM can schedule the task to another datanode which has the same
replica. Introducing a new configurable path makes thing complicate. If it's on
the same physical disk, then it will suffer the same disk full. If it's on a
different physical disk, how to guarantee that this different physical disk
will not become full ahead of this data volume.
> Make metadata dump path configurable for container export
> ---------------------------------------------------------
>
> Key: HDDS-12422
> URL: https://issues.apache.org/jira/browse/HDDS-12422
> Project: Apache Ozone
> Issue Type: Task
> Components: Ozone Datanode
> Reporter: Saketa Chalamchala
> Assignee: Saketa Chalamchala
> Priority: Major
>
> With SCHEMA_V3, When container is exported the container metadata is dumped
> from container.db into <containerPath>/metadata/db before getting tar'ed and
> sent to destination.
> In some edge cases when a volume becomes full (100% utilized), we might want
> to run balancer but, container replications might fail because there is no
> more space in the volume to dump metadata files.
> The metadata dump directory needs to be configurable so that metadata can be
> dumped to a different volume if necessary so that replications from the full
> volume can succeed.
> {code:java}
> 2025-02-04 18:04:58 INFO KeyValueContainer.packContainerToDestination:1011 -
> ATTENTION! using v3 path
> 2025-02-04T18:04:58.358333464Z 2025-02-04 18:04:58 INFO
> KeyValueContainer.packContainerToDestination:1013 - ATTENTION! dumping
> metadata
> 2025-02-04T18:04:58.365236964Z 2025-02-04 18:04:58 INFO
> KeyValueContainer.dumpKVContainerDataToFiles:314 - ATTENTION! Creating dir
> /data/hdds/hdds/CID-232dac26-1ebf-4ca0-bda1-27b75af01e9d/current/containerDir0/1/metadata/db
> for container 1 to dump metadata to files
> 2025-02-04T18:04:58.365330089Z 2025-02-04 18:04:58 INFO
> AbstractDatanodeStore.getTableDumpFile:151 - ATTENTION! dumping table
> 'metadata' under
> '/data/hdds/hdds/CID-232dac26-1ebf-4ca0-bda1-27b75af01e9d/current/containerDir0/1/metadata/db'
> 2025-02-04T18:04:58.378313339Z 2025-02-04 18:04:58 WARN
> PushReplicator.replicate:73 - Container 1 replication was unsuccessful.
> 2025-02-04T18:04:58.378334672Z
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Failed to dump metadata for container 1
> 2025-02-04T18:04:58.378337714Z at
> org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils.dumpKVContainerDataToFiles(BlockUtils.java:320)
> 2025-02-04T18:04:58.378342547Z at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.packContainerToDestination(KeyValueContainer.java:1014)
> 2025-02-04T18:04:58.378343714Z at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.exportContainerData(KeyValueContainer.java:719)
> 2025-02-04T18:04:58.378344839Z at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.exportContainer(KeyValueHandler.java:1194)
> 2025-02-04T18:04:58.378346006Z at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.exportContainer(ContainerController.java:195)
> 2025-02-04T18:04:58.378347089Z at
> org.apache.hadoop.ozone.container.replication.OnDemandContainerReplicationSource.copyData(OnDemandContainerReplicationSource.java:62)
> 2025-02-04T18:04:58.378348381Z at
> org.apache.hadoop.ozone.container.replication.PushReplicator.replicate(PushReplicator.java:67)
> 2025-02-04T18:04:58.378349547Z at
> org.apache.hadoop.ozone.container.replication.MeasuredReplicator.replicate(MeasuredReplicator.java:82)
> ...
> 2025-02-04T18:04:58.378354922Z at
> java.base/java.lang.Thread.run(Thread.java:1583)
> 2025-02-04T18:04:58.378356006Z Caused by: java.io.IOException: Failed to
> finish dumping into file
> /data/hdds/hdds/CID-232dac26-1ebf-4ca0-bda1-27b75af01e9d/current/containerDir0/1/metadata/db/metadata.data;
> status : IOError(NoSpace); message : While appending to file:
> /data/hdds/hdds/CID-232dac26-1ebf-4ca0-bda1-27b75af01e9d/current/containerDir0/1/metadata/db/metadata.data:
> No space left on device
> ... {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
