[
https://issues.apache.org/jira/browse/HDDS-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang deleted HDDS-4960:
----------------------------------
> Disk space leak after deleting Ozone files
> ------------------------------------------
>
> Key: HDDS-4960
> URL: https://issues.apache.org/jira/browse/HDDS-4960
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> I'm finding cases where deleting Ozone files does not reclaim disk space,
> even if the corresponding containers are empty and in CLOSED state.
> In some cases, blocks are removed from container, resulting in empty and
> closed container. However, the container metadata is not removed, and SCM
> still records the container. No way to remove them. In this case, each
> dangling container occupies ~140mb of space. Overtime, tens or even hundreds
> of GB is wasted on each DN.
> Because of HDDS-2534, there is no way to manually delete the containers.
> Steps to reproduce:
> {code}
> ozone sh volume create o3://ozone1/vol1
> ozone sh bucket create o3://ozone1/vol1/bucket1
> # Generated data.
> export EXAMPLES_PATH=
> yarn jar
> /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce//hadoop-mapreduce-examples.jar
> teragen -Dmapreduce.job.maps=10 -DmDmapreduce.map.memory.mb=4096
> -Dmapreduce.reduce.memory.mb=4096 1000000000
> o3fs://bucket1.vol1.ozone1/teragen11
> Delete skip trash:
> hdfs dfs -rm -r -skipTrash "o3fs://bucket1.vol1.ozone1/*”
> (run "ozone admin container close ${containerid} to force closing the
> containers)
> # check disk usage:
> du -h
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/
> # the container directory is not removed
> ...
> 142M
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata/34-dn-container.db
> 4.0K
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata/db.checkpoints
> 142M
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata
> 4.0K
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/chunks
> 142M
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34
> 13G
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
