[
https://issues.apache.org/jira/browse/MAPREDUCE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930412#action_12930412
]
Todd Lipcon commented on MAPREDUCE-2181:
----------------------------------------
bq. I can see major problems with sites that set quotas.
It seems to me that quotas should include temporary space used during job
submission. If your job jar or dcache resources are very large, by all means it
should count against your quota, don't you think? On any _reasonable_ workload,
the space used in the staging directory will be many orders of magnitude lower
than any user quotas, anyway.
bq. Why not just make it an explicit /tmp?
Because /tmp is not necessarily created on a fresh HDFS either. Note that this
is an *HDFS* directory, not local fs.
> mapreduce.jobtracker.staging.root.dir default is unreasonable
> -------------------------------------------------------------
>
> Key: MAPREDUCE-2181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2181
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, jobtracker
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
>
> The default for mapreduce.jobtracker.staging.root.dir is set to
> ${hadoop.tmp.dir}/mapred/staging, which doesn't really work on a normal
> cluster. hadoop.tmp.dir is overloaded in different places where sometimes it
> is a local path and sometimes it is a path on HDFS, which makes things even
> more confusing.
> We should change the default for the staging directory to /user (as is
> suggested by the description of that configuration) and then fix
> LocalJobRunner to use a different configuration -- perhaps
> mapreduce.localjobrunner.staging.root.dir -- to make it clear that it's a
> *local* path. That one could legitimately default to something inside
> hadoop.tmp.dir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
