[
https://issues.apache.org/jira/browse/MAPREDUCE-7033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe reassigned MAPREDUCE-7033:
-------------------------------------
Assignee: Jason Lowe
Priority: Critical (was: Major)
Target Version/s: 3.1.0, 3.0.1
Raising the priority of this since I recently discovered this long-lingering
bug can actually prevent proper shuffling on secure clusters in Hadoop 3.x.
HADOOP-11347 fixed a bug in the local filesystem so it started honoring the
configured fs.permissions.umask-mode for local files. That fix now exposes the
lack of explicit permissions in the MapReduce framework for any case where the
configured umask is at least 0044.
> Map outputs implicitly rely on permissive umask for shuffle
> -----------------------------------------------------------
>
> Key: MAPREDUCE-7033
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7033
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
>
> Map tasks do not explicitly set the permissions of their output files for
> shuffle. In a secure cluster the shuffle service is running as a different
> user than the map task, so the output files require group readability in
> order to serve up the data during the shuffle phase. If the user's UNIX
> umask is too restrictive (e.g.: 077) then the map task's file.out and
> file.out.index permissions can be too restrictive to allow the shuffle
> handler to access them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
