[
https://issues.apache.org/jira/browse/HADOOP-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530232#comment-16530232
]
Xiaoyu Yao commented on HADOOP-15571:
-------------------------------------
{quote}That was my first approach too, but there are too many uses of
FileContext (in YARN and potentially as well as in downstream projects) and
forcing all of them to create a new Configuration object is not right. This
creation of such a new Config object per URI was not needed in 2.x.
{quote}
[~vinodkv], that makes sense to me. +1 for adding a regression test for the
fix.
> After HADOOP-13440, multiple filesystems/file-contexts created with the same
> Configuration object are forced to have the same umask
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-15571
> URL: https://issues.apache.org/jira/browse/HADOOP-15571
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Attachments: HADOOP-15571.txt
>
>
> Ran into a super hard-to-debug issue due to this. [Edit: Turns out the same
> issue as YARN-5749 that [~Tao Yang] ran into]
> h4. Issue
> Configuration conf = new Configuration();
> fc1 = FileContext.getFileContext(uri1, conf);
> fc2 = FileContext.getFileContext(uri2, conf);
> fc.setUMask(umask_for_fc1); // Screws up umask for fc2 also!
> This was not the case before HADOOP-13440.
> h4. Symptoms:
> h5. Scenario I ran into
> When trying to localize a HDFS directory (hdfs:///my/dir/1.txt), NodeManager
> tries to replicate the directory structure on the local file-system
> ($yarn-local-dirs/filecache/my/dir/1.txt).
> Now depending on whether NM has ever done a log-aggregation (completely
> unrelated code that sets umask to be 137 for its own files on HDFS), the
> directories /my and /my/dir on local-fs may have different permissions. In
> the specific case where NM did log-aggregation, /my/dir was created with 137
> umask and so localization of 1.txt completely failed due to absent directory
> executable permissions!
> h5. Previous scenarios:
> We ran into this before in test-cases and instead of fixing the root-cause,
> we just fixed the test-cases: YARN-5679 / YARN-5749
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
