[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14030349#comment-14030349
]
Gera Shegalov commented on MAPREDUCE-5875:
------------------------------------------
Hi Robert,
bq. We were trying out v3 of the patch some more and ran into a problem if you
configure your history files (i.e. the "done" directory) to go under /user
instead of /tmp
This should be unrelated to the root location /tmp vs /user. It's related to
the permissions/umask used to create the history done dir:
{code}
public static final FsPermission HISTORY_DONE_DIR_PERMISSION =
FsPermission.createImmutable((short) 0770);
/**
* Umask for the done dir and derivatives.
*/
public static final FsPermission HISTORY_DONE_DIR_UMASK = FsPermission
.createImmutable((short) (0770 ^ 0777));
{code}
Presumably the goal is to prevent users from listing other users job files.
Looks like we can't support job-specified limits when dealing with history if
the user does not share the group with JHS user.
> Make Counter limits consistent conf across JobClient, MRAppMaster, and
> YarnChild
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
> Affects Versions: 2.4.0
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
> MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
