Morty Zhong created MAPREDUCE-7229: -------------------------------------- Summary: Job History Server should ignore continuing move done_intermediate files when rename failed Key: MAPREDUCE-7229 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7229 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.1.2 Reporter: Morty Zhong
{code:java} private void moveToDoneNow(final Path src, final Path target) throws IOException { LOG.info("Moving " + src.toString() + " to " + target.toString()); try { intermediateDoneDirFc.rename(src, target, Options.Rename.NONE); } catch (FileNotFoundException e) { if (doneDirFc.util().exists(target)) { LOG.info("Source file " + src.toString() + " not found, but target " + "file " + target.toString() + " already exists. Move already " + "happened."); } else { throw e; } } } {code} if files in done_intermediate dir were already deleted(deleted manually), not move to done dir. Job history will infinitely rename these files later, and throw exp always.This will cause newly created files in done_intermediate dir can not be moved. Later, when scan done_intermediate dir, FileContext.listStatus(done_intermediate) will so slow, and gc problem -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org