[
https://issues.apache.org/jira/browse/CRUNCH-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645137#comment-13645137
]
Micah Whitacre commented on CRUNCH-200:
---
Changes look good. One little tweak might be..
{code}
public void handleOutputs(Configuration conf, Path workingPath, int index,
boolean mapOnlyJob) throws IOException {
FileSystem srcFs = workingPath.getFileSystem(conf);
Path srcPattern = getSourcePattern();
Path[] srcs = FileUtil.stat2Paths(srcFs.globStatus(src), src);
FileSystem dstFs = FileSystem.get(conf);
if (!dstFs.exists(path)) {
dstFs.mkdirs(path);
}
boolean sameFs = isCompatible(srcFs, path);
for (Path s : srcs) {
Path d = getDestFile(conf, s, path, mapOnlyJob);
if (sameFs) {
srcFs.rename(s, d);
} else {
FileUtil.copy(srcFs, s, dstFs, d, true, true, conf);
}
}
}
protected Path getSourcePattern(){
return new Path(workingPath, PlanningParameters.MULTI_OUTPUT_PREFIX + index +
-*);
}
{code}
Move the logic for handling job outputs from CrunchJobHooks into the
PathTarget objects
---
Key: CRUNCH-200
URL: https://issues.apache.org/jira/browse/CRUNCH-200
Project: Crunch
Issue Type: Improvement
Reporter: Josh Wills
Attachments: CRUNCH-200.patch
Certain output formats, like Trevni, have a non-standard output file format
that requires custom handling logic. Currently, the logic for moving the
output of a Crunch job from a temporary place on the filesystem to its final
output directory is handled by logic in CrunchJobHooks, but the actual work
of moving the files should be delegated to the Target object, which will be
aware of how the files are laid out in the filesystem and can move them
appropriately.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
