Soumitra Sulav created HDDS-14577:
-------------------------------------
Summary: MISSING_METADATA_DIR trying to mark the container as
UNHEALTHY, but creating .container file in metadata dir itself
Key: HDDS-14577
URL: https://issues.apache.org/jira/browse/HDDS-14577
Project: Apache Ozone
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Soumitra Sulav
Error was seen on datanodes as MISSING_METADATA_DIR, but while trying to mark
the container as UNHEALTHY, it was creating {{.container}} file in metadata dir
itself (which is missing as per the error recorded) and fails with exception
({{{}Failed to create tmp file for
/data/2/hadoop-ozone/datanode/data/hdds/CID-40c1b249-f7d7-4287-bbe6-982c3bfb49a8/current/containerDir134/69019/metadata/69019.container{}}})
{code:java}
2026-01-20 09:59:59,929 ERROR
[ContainerMetadataScanner]-org.apache.hadoop.ozone.container.ozoneimpl.BackgroundContainerMetadataScanner:
Corruption detected in container [69019]. Marking it UNHEALTHY. Container has
1 error: MISSING_METADATA_DIR for file
/data/2/hadoop-ozone/datanode/data/hdds/CID-40c1b249-f7d7-4287-bbe6-982c3bfb49a8/current/containerDir134/69019/metadata
with exception: java.io.FileNotFoundException: Metadata directory
/data/2/hadoop-ozone/datanode/data/hdds/CID-40c1b249-f7d7-4287-bbe6-982c3bfb49a8/current/containerDir134/69019/metadata
not found.
2026-01-20 09:59:59,930 WARN
[ContainerMetadataScanner]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler:
Unexpected error while marking container 69019 unhealthy
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
Failed to create tmp file for
/data/2/hadoop-ozone/datanode/data/hdds/CID-40c1b249-f7d7-4287-bbe6-982c3bfb49a8/current/containerDir134/69019/metadata/69019.container:
KeyValueContainerData #69019 (UNHEALTHY, non-empty, ri=0,
origin=[dn_73ce993f-edab-4819-a73f-b938b0241e48,
pipeline_1bb5a281-249c-4262-b299-601d34b4f0bb])
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.writeToContainerFile(KeyValueContainer.java:318)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerFile(KeyValueContainer.java:337)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerData(KeyValueContainer.java:484)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerState(KeyValueContainer.java:467)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.markContainerUnhealthy(KeyValueContainer.java:392)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.markContainerUnhealthy(KeyValueHandler.java:1513)
at
org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.markContainerUnhealthy(ContainerController.java:124)
at
org.apache.hadoop.ozone.container.ozoneimpl.ContainerScanHelper.handleUnhealthyScanResult(ContainerScanHelper.java:131)
at
org.apache.hadoop.ozone.container.ozoneimpl.ContainerScanHelper.scanMetadata(ContainerScanHelper.java:107)
at
org.apache.hadoop.ozone.container.ozoneimpl.BackgroundContainerMetadataScanner.scanContainer(BackgroundContainerMetadataScanner.java:57)
at
org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.scanContainers(AbstractBackgroundContainerScanner.java:115)
at
org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.runIteration(AbstractBackgroundContainerScanner.java:78)
at
org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.run(AbstractBackgroundContainerScanner.java:59)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: No such file or directory
at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.base/java.io.File.createTempFile(File.java:2170)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.createTempFile(KeyValueContainer.java:907)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.writeToContainerFile(KeyValueContainer.java:300)
... 13 more
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
