[
https://issues.apache.org/jira/browse/MAPREDUCE-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069698#comment-13069698
]
Harsh J commented on MAPREDUCE-2715:
------------------------------------
Geoffrey,
Thanks for the answers to those!
So your major pain point is with the OutputFormat checking if the directory
already exists during submit time. This is a client-side behavior, over-ridable
by your MultipleOutputFormat derivative {{(@Override checkOutputSpecs)}}. This
isn't a hadoop issue per se, cause the default check to FileOutputFormat was
added with safety-first principles in mind, but as I mentioned already, its
over-ridable by developers. You have that flexibility already available.
Also, maybe you'd know about it already, but just for warning's sake, take a
read of the default OutputCommitter behavior in Hadoop:
http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2BAC8-write-to_hdfs_files_directly_from_map.2BAC8-reduce_tasks.3F
> submitAndMonitorJob() doesn't play nice with MultipleOutputFile
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Geoffrey Young
>
> part of submitAndMonitorJob() balks if the output directory currently exists
> but is non-empty:
> "Error launching job , Output path already exists : "
> this logic actually conflicts with the ideas behind MultipleOutputFile, where
> the output file path is calculated later on.
> it would be really nice to remove the restriction for non-empty output
> directories in submitAndMonitorJob() so that MultipleOutputFile becomes more
> useful - as it stands now, I can't, for example, specify a base output path
> then use MutlipleOutputFile to partition by date on a daily basis.
> thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
