Saketa Chalamchala created HDDS-12422:
-----------------------------------------
Summary: Make metadata dump path configurable for container export
Key: HDDS-12422
URL: https://issues.apache.org/jira/browse/HDDS-12422
Project: Apache Ozone
Issue Type: Task
Components: Ozone Datanode
Reporter: Saketa Chalamchala
Assignee: Saketa Chalamchala
With SCHEMA_V3, When container is exported the container metadata is dumped
from container.db into <containerPath>/metadata/db before getting tar'ed and
sent to destination.
In some edge cases when a volume becomes full (100% utilized), we might want to
run balancer but, container replications might fail because there is no more
space in the volume to dump metadata files.
The metadata dump directory needs to be configurable so that metadata can be
dumped to a different volume if necessary so that replications from the full
volume can succeed.
{code:java}
2025-02-04 18:04:58 INFO KeyValueContainer.packContainerToDestination:1011 -
ATTENTION! using v3 path
2025-02-04T18:04:58.358333464Z 2025-02-04 18:04:58 INFO
KeyValueContainer.packContainerToDestination:1013 - ATTENTION! dumping metadata
2025-02-04T18:04:58.365236964Z 2025-02-04 18:04:58 INFO
KeyValueContainer.dumpKVContainerDataToFiles:314 - ATTENTION! Creating dir
/data/hdds/hdds/CID-232dac26-1ebf-4ca0-bda1-27b75af01e9d/current/containerDir0/1/metadata/db
for container 1 to dump metadata to files
2025-02-04T18:04:58.365330089Z 2025-02-04 18:04:58 INFO
AbstractDatanodeStore.getTableDumpFile:151 - ATTENTION! dumping table
'metadata' under
'/data/hdds/hdds/CID-232dac26-1ebf-4ca0-bda1-27b75af01e9d/current/containerDir0/1/metadata/db'
2025-02-04T18:04:58.378313339Z 2025-02-04 18:04:58 WARN
PushReplicator.replicate:73 - Container 1 replication was unsuccessful.
2025-02-04T18:04:58.378334672Z
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
Failed to dump metadata for container 1
2025-02-04T18:04:58.378337714Z at
org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils.dumpKVContainerDataToFiles(BlockUtils.java:320)
2025-02-04T18:04:58.378342547Z at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.packContainerToDestination(KeyValueContainer.java:1014)
2025-02-04T18:04:58.378343714Z at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.exportContainerData(KeyValueContainer.java:719)
2025-02-04T18:04:58.378344839Z at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.exportContainer(KeyValueHandler.java:1194)
2025-02-04T18:04:58.378346006Z at
org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.exportContainer(ContainerController.java:195)
2025-02-04T18:04:58.378347089Z at
org.apache.hadoop.ozone.container.replication.OnDemandContainerReplicationSource.copyData(OnDemandContainerReplicationSource.java:62)
2025-02-04T18:04:58.378348381Z at
org.apache.hadoop.ozone.container.replication.PushReplicator.replicate(PushReplicator.java:67)
2025-02-04T18:04:58.378349547Z at
org.apache.hadoop.ozone.container.replication.MeasuredReplicator.replicate(MeasuredReplicator.java:82)
...
2025-02-04T18:04:58.378354922Z at
java.base/java.lang.Thread.run(Thread.java:1583)
2025-02-04T18:04:58.378356006Z Caused by: java.io.IOException: Failed to finish
dumping into file
/data/hdds/hdds/CID-232dac26-1ebf-4ca0-bda1-27b75af01e9d/current/containerDir0/1/metadata/db/metadata.data;
status : IOError(NoSpace); message : While appending to file:
/data/hdds/hdds/CID-232dac26-1ebf-4ca0-bda1-27b75af01e9d/current/containerDir0/1/metadata/db/metadata.data:
No space left on device
... {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
