[
https://issues.apache.org/jira/browse/MAPREDUCE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095249#comment-16095249
]
lawrence andrews commented on MAPREDUCE-6917:
---------------------------------------------
Following is the fix
{noformat}
git diff
diff --git
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
index 06e08e428c1..f69de6285c6 100644
---
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
+++
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
@@ -487,12 +487,17 @@ private static Configuration getDefaultConf() {
return config;
}
- private synchronized void cleanup() {
+ /**
+ * Clean the staging folder created by distcp
+ */
+ public void cleanup() {
try {
if (metaFolder == null) return;
- jobFS.delete(metaFolder, true);
- metaFolder = null;
+ synchronized (this) {
+ jobFS.delete(metaFolder, true);
+ metaFolder = null;
+ }
} catch (IOException e) {
LOG.error("Unable to cleanup meta folder: " + metaFolder, e);
}
{noformat}
> DistCp does not clean staging folder if class extends DistCp
> ------------------------------------------------------------
>
> Key: MAPREDUCE-6917
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6917
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Reporter: lawrence andrews
> Priority: Minor
>
> My code extends Distcp class and for some reason if distcp fails staging
> folder is not delete and this staging folder piles up occupying space on hdfs.
> After checking the code i found that cleanup() function is private. Making
> the cleanup() method as public, user should be able to invoke the cleanup if
> job fails.
> This works fine with command line argument. But fails only if we extend
> Distcp class.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
