[
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162283#comment-16162283
]
Robert Kanter commented on MAPREDUCE-6954:
------------------------------------------
Overall looks good. A couple minor things:
- Update mapred-default like you suggested
- The staging dir config is {{yarn.app.mapreduce.am.staging-dir}} and defined
in MRJobConfig as {{MR_AM_STAGING_DIR = MR_AM_PREFIX+"staging-dir"}}. I think
we should be consistent with this and make
{{MR_JOB_STAGING_ERASURECODING_ENABLED =
"mapreduce.job.staging-dir.erasurecoding.enabled"}} be
{{MR_AM_STAGING_DIR_ERASURECODING_ENABLED = MR_AM_STAGING_DIR +
"erasurecoding.enabled"}} which would resolve to
{{yarn.app.mapreduce.am.staging-dir.erasurecoding.enabled}}.
- It would be good to add a unit test.
> Disable erasure coding for files that are uploaded to the MR staging area
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
> Affects Versions: 3.0.0-alpha4
> Reporter: Peter Bacsko
> Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6954-001.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might
> negatively affect the performance of an MR job if too many files are
> localized.
> In such a scenario, users might want to disable EC in the staging area to
> speed up the execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
