[
https://issues.apache.org/jira/browse/HDDS-14577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Soumitra Sulav updated HDDS-14577:
----------------------------------
Component/s: Ozone Filesystem
> MISSING_METADATA_DIR trying to mark the container as UNHEALTHY, but creating
> .container file in metadata dir itself
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-14577
> URL: https://issues.apache.org/jira/browse/HDDS-14577
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Filesystem
> Affects Versions: 2.0.0
> Reporter: Soumitra Sulav
> Priority: Major
>
> Error was seen on datanodes as MISSING_METADATA_DIR, but while trying to mark
> the container as UNHEALTHY, it was creating {{.container}} file in metadata
> dir itself (which is missing as per the error recorded) and fails with
> exception ({{{}Failed to create tmp file for
> /data/2/hadoop-ozone/datanode/data/hdds/CID-40c1b249-f7d7-4287-bbe6-982c3bfb49a8/current/containerDir134/69019/metadata/69019.container{}}})
> {code:java}
> 2026-01-20 09:59:59,929 ERROR
> [ContainerMetadataScanner]-org.apache.hadoop.ozone.container.ozoneimpl.BackgroundContainerMetadataScanner:
> Corruption detected in container [69019]. Marking it UNHEALTHY. Container
> has 1 error: MISSING_METADATA_DIR for file
> /data/2/hadoop-ozone/datanode/data/hdds/CID-40c1b249-f7d7-4287-bbe6-982c3bfb49a8/current/containerDir134/69019/metadata
> with exception: java.io.FileNotFoundException: Metadata directory
> /data/2/hadoop-ozone/datanode/data/hdds/CID-40c1b249-f7d7-4287-bbe6-982c3bfb49a8/current/containerDir134/69019/metadata
> not found.
> 2026-01-20 09:59:59,930 WARN
> [ContainerMetadataScanner]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler:
> Unexpected error while marking container 69019 unhealthy
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Failed to create tmp file for
> /data/2/hadoop-ozone/datanode/data/hdds/CID-40c1b249-f7d7-4287-bbe6-982c3bfb49a8/current/containerDir134/69019/metadata/69019.container:
> KeyValueContainerData #69019 (UNHEALTHY, non-empty, ri=0,
> origin=[dn_73ce993f-edab-4819-a73f-b938b0241e48,
> pipeline_1bb5a281-249c-4262-b299-601d34b4f0bb])
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.writeToContainerFile(KeyValueContainer.java:318)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerFile(KeyValueContainer.java:337)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerData(KeyValueContainer.java:484)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerState(KeyValueContainer.java:467)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.markContainerUnhealthy(KeyValueContainer.java:392)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.markContainerUnhealthy(KeyValueHandler.java:1513)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.markContainerUnhealthy(ContainerController.java:124)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerScanHelper.handleUnhealthyScanResult(ContainerScanHelper.java:131)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerScanHelper.scanMetadata(ContainerScanHelper.java:107)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.BackgroundContainerMetadataScanner.scanContainer(BackgroundContainerMetadataScanner.java:57)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.scanContainers(AbstractBackgroundContainerScanner.java:115)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.runIteration(AbstractBackgroundContainerScanner.java:78)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.run(AbstractBackgroundContainerScanner.java:59)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: java.io.IOException: No such file or directory
> at java.base/java.io.UnixFileSystem.createFileExclusively(Native
> Method)
> at java.base/java.io.File.createTempFile(File.java:2170)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.createTempFile(KeyValueContainer.java:907)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.writeToContainerFile(KeyValueContainer.java:300)
> ... 13 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
