[
https://issues.apache.org/jira/browse/HDFS-10256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249340#comment-15249340
]
Vinayakumar B edited comment on HDFS-10256 at 4/20/16 6:09 AM:
---------------------------------------------------------------
bq. MiniDFSCluster#shutdown() registers base_dir to be deleted on shutdown. If
this gets slow, the next test JVM will start to run before the shutdown hook
completes. But forcing every test to call shutdown(true) can slowdown things.
Instead, each instance should get a random base_dir, so that the deletion
through shutdown hook and the subsequent new test setup can overlap.
I have investigated a bit on this. Registering a dir to be deleted during
shutdown will not actually delete the directory unless its empty. I have tried
to register all files/directories recursively to be deleted on shutdown. But
this will sometimes keeps the empty directories.
Instead, I have used unique base directory (testclassname.methodname) for each
MiniDFSCluster instance and {{FileUtils.fullyDelete()}} during shutdown itself,
instead of registering for deleteOnExit() in {{MiniDFSCluster#shutdown()}}.
All tests which actually uses MiniDfsCluster will delete its basedir. Apart
from this, there are some more tests, which creates directories on its own (
using some Random String), and these may not get deleted after test run.
At the end of test run, there was ~300MB of data created in
"hadoop-hdfs/target/test/data" directory, which is way less than ~6GB reported
by Chris.
So I am not sure why {{base_dir.deleteOnExit()}} was used during
{{MiniDfsCluster#shutdown()}}, which will not actually delete anything because
base_dir will not be empty.
was (Author: vinayrpet):
bq. MiniDFSCluster#shutdown() registers base_dir to be deleted on shutdown. If
this gets slow, the next test JVM will start to run before the shutdown hook
completes. But forcing every test to call shutdown(true) can slowdown things.
Instead, each instance should get a random base_dir, so that the deletion
through shutdown hook and the subsequent new test setup can overlap.
I have investigated a bit on this. Registering a dir to be deleted during
shutdown will not actually delete the directory unless its empty. I have tried
to register all files/directories recursively to be deleted on shutdown. But
this will sometimes keeps the empty directories.
Instead, I have used unique base directory (testclassname.methodname) for each
MiniDFSCluster instance and FileUtils.fullyDelete()}} during shutdown itself,
instead of registering for deleteOnExit() in MiniDFSCluster#shutdown().
All tests which actually uses MiniDfsCluster will delete its basedir. Apart
from this, there are some more tests, which creates directories on its own (
using some Random String), and these may not get deleted after test run.
At the end of test run, there was ~300MB of data created in
"hadoop-hdfs/target/test/data" directory, which is way less than ~6GB reported
by Chris.
So I am not sure why {{base_dir.deleteOnExit()}} was used during
{{MiniDfsCluster#shutdown()}}, which will not actually delete anything because
base_dir will not be empty.
> Use GenericTestUtils.getTestDir method in tests for temporary directories
> -------------------------------------------------------------------------
>
> Key: HDFS-10256
> URL: https://issues.apache.org/jira/browse/HDFS-10256
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: build, test
> Reporter: Vinayakumar B
> Assignee: Vinayakumar B
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
