[
https://issues.apache.org/jira/browse/MAPREDUCE-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068573#comment-13068573
]
Harsh J commented on MAPREDUCE-2715:
------------------------------------
A few questions here:
* What is {{MultipleOutputFile}}? Pardon my ignorance, but I could not find a
class or type like that in my workspace, so would be good to know if its not
{{MultipleOutputs}} or {{MultiFileOutputFormat}} we're talking about here.
* The output directory existence check is done by the {{OutputFormat}} class in
use, and not the {{submitAndMonitorJob()}} call. The specific method to
override in this case would usually be {{OutputFormat#checkOutputSpecs}}, which
gets invoked pre-submit. Is this what you are talking about too?
* Does {{MultipleOutputFile}} (or if this is one of the two classes I mentioned
earlier) create arbitrary directories in the HDFS? How does it ensure task
commits in this case?
> submitAndMonitorJob() doesn't play nice with MultipleOutputFile
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Geoffrey Young
>
> part of submitAndMonitorJob() balks if the output directory currently exists
> but is non-empty:
> "Error launching job , Output path already exists : "
> this logic actually conflicts with the ideas behind MultipleOutputFile, where
> the output file path is calculated later on.
> it would be really nice to remove the restriction for non-empty output
> directories in submitAndMonitorJob() so that MultipleOutputFile becomes more
> useful - as it stands now, I can't, for example, specify a base output path
> then use MutlipleOutputFile to partition by date on a daily basis.
> thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
