[
https://issues.apache.org/jira/browse/CRUNCH-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645137#comment-13645137
]
Micah Whitacre edited comment on CRUNCH-200 at 4/30/13 1:48 AM:
----------------------------------------------------------------
Changes look good. One little tweak might be..
{code}
public void handleOutputs(Configuration conf, Path workingPath, int index,
boolean mapOnlyJob) throws IOException {
FileSystem srcFs = workingPath.getFileSystem(conf);
Path srcPattern = getSourcePattern(workingPath);
Path[] srcs = FileUtil.stat2Paths(srcFs.globStatus(src), src);
FileSystem dstFs = FileSystem.get(conf);
if (!dstFs.exists(path)) {
dstFs.mkdirs(path);
}
boolean sameFs = isCompatible(srcFs, path);
for (Path s : srcs) {
Path d = getDestFile(conf, s, path, mapOnlyJob);
if (sameFs) {
srcFs.rename(s, d);
} else {
FileUtil.copy(srcFs, s, dstFs, d, true, true, conf);
}
}
}
protected Path getSourcePattern(Path workingPath){
return new Path(workingPath, PlanningParameters.MULTI_OUTPUT_PREFIX + index +
"-*");
}
{code}
was (Author: mkwhitacre):
Changes look good. One little tweak might be..
{code}
public void handleOutputs(Configuration conf, Path workingPath, int index,
boolean mapOnlyJob) throws IOException {
FileSystem srcFs = workingPath.getFileSystem(conf);
Path srcPattern = getSourcePattern();
Path[] srcs = FileUtil.stat2Paths(srcFs.globStatus(src), src);
FileSystem dstFs = FileSystem.get(conf);
if (!dstFs.exists(path)) {
dstFs.mkdirs(path);
}
boolean sameFs = isCompatible(srcFs, path);
for (Path s : srcs) {
Path d = getDestFile(conf, s, path, mapOnlyJob);
if (sameFs) {
srcFs.rename(s, d);
} else {
FileUtil.copy(srcFs, s, dstFs, d, true, true, conf);
}
}
}
protected Path getSourcePattern(){
return new Path(workingPath, PlanningParameters.MULTI_OUTPUT_PREFIX + index +
"-*");
}
{code}
> Move the logic for handling job outputs from CrunchJobHooks into the
> PathTarget objects
> ---------------------------------------------------------------------------------------
>
> Key: CRUNCH-200
> URL: https://issues.apache.org/jira/browse/CRUNCH-200
> Project: Crunch
> Issue Type: Improvement
> Reporter: Josh Wills
> Attachments: CRUNCH-200.patch
>
>
> Certain output formats, like Trevni, have a non-standard output file format
> that requires custom handling logic. Currently, the logic for moving the
> output of a Crunch job from a temporary place on the filesystem to its final
> output directory is handled by logic in CrunchJobHooks, but the actual work
> of moving the files should be delegated to the Target object, which will be
> aware of how the files are laid out in the filesystem and can move them
> appropriately.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
