[
https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350537#comment-14350537
]
Jason Lowe commented on MAPREDUCE-4815:
---------------------------------------
Thanks for the clarification, Siqi. That should only happen if the client
submitting the job did not have a default value in their mapred-site.xml. If
it did then the setting would make it into the job's conf (job.xml) and
override what mapred-site.xml on the node says.
>From personal experience, you do not want jobs picking up confs from nodes as
>the nodes are upgrading. Besides concerns like the one here, there are all
>sorts of other things that can break. If the job doesn't try to pull from the
>node's configs then it's not an issue, and we run our clusters that way so we
>can safely upgrade the nodes without affecting the confs of the jobs on those
>nodes.
To be clear, I'm not against the code trying to protect itself from this
(somewhat dubious) use case, but I don't think we need to spell it out in the
docs. It's not a use case we want to encourage.
> FileOutputCommitter.commitJob can be very slow for jobs with many output files
> ------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1
> Reporter: Jason Lowe
> Assignee: Siqi Li
> Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch,
> MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch,
> MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v3.patch,
> MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch,
> MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch
>
>
> If a job generates many files to commit then the commitJob method call at the
> end of the job can take minutes. This is a performance regression from 1.x,
> as 1.x had the tasks commit directly to the final output directory as they
> were completing and commitJob had very little to do. The commit work was
> processed in parallel and overlapped the processing of outstanding tasks. In
> 0.23/2.x, the commit is single-threaded and waits until all tasks have
> completed before commencing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
