[jira] [Created] (MAPREDUCE-6607) .staging dir is not cleanup if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set

[Maysam Yabandeh (JIRA)]

Maysam Yabandeh created MAPREDUCE-6607:
------------------------------------------

             Summary: .staging dir is not cleanup if 
mapreduce.task.files.preserve.failedtask or 
mapreduce.task.files.preserve.filepattern are set
                 Key: MAPREDUCE-6607
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: applicationmaster
    Affects Versions: 2.7.1
            Reporter: Maysam Yabandeh
            Priority: Minor


if either of the following configs are set, then .staging dir is not cleaned up:
* mapreduce.task.files.preserve.failedtask 
* mapreduce.task.files.preserve.filepattern

The former was supposed to keep only .staging of failed tasks and the latter 
was supposed to be used only if that task name matches against the specified 
regular expression.

{code}
  protected boolean keepJobFiles(JobConf conf) {
    return (conf.getKeepTaskFilesPattern() != null || conf
        .getKeepFailedTaskFiles());
  }
{code}

{code}
  public void cleanupStagingDir() throws IOException {
    /* make sure we clean the staging files */
    String jobTempDir = null;
    FileSystem fs = getFileSystem(getConfig());
    try {
      if (!keepJobFiles(new JobConf(getConfig()))) {
        jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR);
        if (jobTempDir == null) {
          LOG.warn("Job Staging directory is null");
          return;
        }
        Path jobTempDirPath = new Path(jobTempDir);
        LOG.info("Deleting staging directory " + 
FileSystem.getDefaultUri(getConfig()) +
            " " + jobTempDir);
        fs.delete(jobTempDirPath, true);
      }
    } catch(IOException io) {
      LOG.error("Failed to cleanup staging dir " + jobTempDir, io);
    }
  }
{code}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)