[jira] [Commented] (HADOOP-10191) Missing executable permission on viewfs internal dirs
[
https://issues.apache.org/jira/browse/HADOOP-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944109#comment-13944109
]
Hudson commented on HADOOP-10191:
-
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1734 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1734/])
HADOOP-10191. Missing executable permission on viewfs internal dirs.
Contributed by Gera Shegalov. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1580170)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
> Missing executable permission on viewfs internal dirs
> -
>
> Key: HADOOP-10191
> URL: https://issues.apache.org/jira/browse/HADOOP-10191
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Blocker
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HADOOP-10191.v01.patch
>
>
> ViewFileSystem allows 1) unconditional listing of internal directories (mount
> points) and 2) and changing work directories.
> 1) requires read permission
> 2) requires executable permission
> However, the hardcoded PERMISSION_RRR == 444 for FileStatus representing an
> internal dir does not have executable bit set.
> This confuses YARN localizer for public resources on viewfs because it
> requires executable permission for "other" on all of the ancestor directories
> of the resource.
> {code}
> java.io.IOException: Resource viewfs:/pubcache/cache.txt is not publicly
> accessable and as such cannot be part of the public cache.
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:182)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:51)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:279)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:277)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10191) Missing executable permission on viewfs internal dirs
[
https://issues.apache.org/jira/browse/HADOOP-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944072#comment-13944072
]
Hudson commented on HADOOP-10191:
-
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1709 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1709/])
HADOOP-10191. Missing executable permission on viewfs internal dirs.
Contributed by Gera Shegalov. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1580170)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
> Missing executable permission on viewfs internal dirs
> -
>
> Key: HADOOP-10191
> URL: https://issues.apache.org/jira/browse/HADOOP-10191
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Blocker
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HADOOP-10191.v01.patch
>
>
> ViewFileSystem allows 1) unconditional listing of internal directories (mount
> points) and 2) and changing work directories.
> 1) requires read permission
> 2) requires executable permission
> However, the hardcoded PERMISSION_RRR == 444 for FileStatus representing an
> internal dir does not have executable bit set.
> This confuses YARN localizer for public resources on viewfs because it
> requires executable permission for "other" on all of the ancestor directories
> of the resource.
> {code}
> java.io.IOException: Resource viewfs:/pubcache/cache.txt is not publicly
> accessable and as such cannot be part of the public cache.
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:182)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:51)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:279)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:277)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10191) Missing executable permission on viewfs internal dirs
[
https://issues.apache.org/jira/browse/HADOOP-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944019#comment-13944019
]
Hudson commented on HADOOP-10191:
-
SUCCESS: Integrated in Hadoop-Yarn-trunk #517 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/517/])
HADOOP-10191. Missing executable permission on viewfs internal dirs.
Contributed by Gera Shegalov. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1580170)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
> Missing executable permission on viewfs internal dirs
> -
>
> Key: HADOOP-10191
> URL: https://issues.apache.org/jira/browse/HADOOP-10191
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Blocker
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HADOOP-10191.v01.patch
>
>
> ViewFileSystem allows 1) unconditional listing of internal directories (mount
> points) and 2) and changing work directories.
> 1) requires read permission
> 2) requires executable permission
> However, the hardcoded PERMISSION_RRR == 444 for FileStatus representing an
> internal dir does not have executable bit set.
> This confuses YARN localizer for public resources on viewfs because it
> requires executable permission for "other" on all of the ancestor directories
> of the resource.
> {code}
> java.io.IOException: Resource viewfs:/pubcache/cache.txt is not publicly
> accessable and as such cannot be part of the public cache.
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:182)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:51)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:279)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:277)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10191) Missing executable permission on viewfs internal dirs
[
https://issues.apache.org/jira/browse/HADOOP-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13943916#comment-13943916
]
Hudson commented on HADOOP-10191:
-
SUCCESS: Integrated in Hadoop-trunk-Commit #5382 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5382/])
HADOOP-10191. Missing executable permission on viewfs internal dirs.
Contributed by Gera Shegalov. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1580170)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
> Missing executable permission on viewfs internal dirs
> -
>
> Key: HADOOP-10191
> URL: https://issues.apache.org/jira/browse/HADOOP-10191
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Gera Shegalov
>Priority: Blocker
> Attachments: HADOOP-10191.v01.patch
>
>
> ViewFileSystem allows 1) unconditional listing of internal directories (mount
> points) and 2) and changing work directories.
> 1) requires read permission
> 2) requires executable permission
> However, the hardcoded PERMISSION_RRR == 444 for FileStatus representing an
> internal dir does not have executable bit set.
> This confuses YARN localizer for public resources on viewfs because it
> requires executable permission for "other" on all of the ancestor directories
> of the resource.
> {code}
> java.io.IOException: Resource viewfs:/pubcache/cache.txt is not publicly
> accessable and as such cannot be part of the public cache.
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:182)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:51)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:279)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:277)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10191) Missing executable permission on viewfs internal dirs
[
https://issues.apache.org/jira/browse/HADOOP-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13943707#comment-13943707
]
Chris Nauroth commented on HADOOP-10191:
bq. are you ever able to see the file status of an unresolved mount link.
Theoretically, yes, this is possible via things like the
{{FileSystem#getFileLinkStatus}} API. I don't see this overridden in
{{ViewFileSystem}} though, which is possibly a distinct bug. {{ViewFs}}
appears to do it correctly.
Anyway, I'm still +1 for this patch. I'll plan to commit later, and we can
revisit discussion on some of these less critical aspects later if needed.
Thanks [~jira.shegalov]!
> Missing executable permission on viewfs internal dirs
> -
>
> Key: HADOOP-10191
> URL: https://issues.apache.org/jira/browse/HADOOP-10191
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Gera Shegalov
>Priority: Blocker
> Attachments: HADOOP-10191.v01.patch
>
>
> ViewFileSystem allows 1) unconditional listing of internal directories (mount
> points) and 2) and changing work directories.
> 1) requires read permission
> 2) requires executable permission
> However, the hardcoded PERMISSION_RRR == 444 for FileStatus representing an
> internal dir does not have executable bit set.
> This confuses YARN localizer for public resources on viewfs because it
> requires executable permission for "other" on all of the ancestor directories
> of the resource.
> {code}
> java.io.IOException: Resource viewfs:/pubcache/cache.txt is not publicly
> accessable and as such cannot be part of the public cache.
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:182)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:51)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:279)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:277)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10191) Missing executable permission on viewfs internal dirs
[
https://issues.apache.org/jira/browse/HADOOP-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13943698#comment-13943698
]
Gera Shegalov commented on HADOOP-10191:
Thanks for your comments, Chris, Sanjay!
[~sanjay.radia], I think [~cnauroth] points out that mount links per se are
like symlinks, are not real files nor directories and therefore always have
777. Unfortunately, I don't have much time right now to verify the usefulness
of this modification, i.e., are you ever able to see the file status of an
unresolved mount link.
> Missing executable permission on viewfs internal dirs
> -
>
> Key: HADOOP-10191
> URL: https://issues.apache.org/jira/browse/HADOOP-10191
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Gera Shegalov
>Priority: Blocker
> Attachments: HADOOP-10191.v01.patch
>
>
> ViewFileSystem allows 1) unconditional listing of internal directories (mount
> points) and 2) and changing work directories.
> 1) requires read permission
> 2) requires executable permission
> However, the hardcoded PERMISSION_RRR == 444 for FileStatus representing an
> internal dir does not have executable bit set.
> This confuses YARN localizer for public resources on viewfs because it
> requires executable permission for "other" on all of the ancestor directories
> of the resource.
> {code}
> java.io.IOException: Resource viewfs:/pubcache/cache.txt is not publicly
> accessable and as such cannot be part of the public cache.
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:182)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:51)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:279)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:277)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10191) Missing executable permission on viewfs internal dirs
[
https://issues.apache.org/jira/browse/HADOOP-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13943680#comment-13943680
]
Sanjay Radia commented on HADOOP-10191:
---
Chris, 0555 is more correct since the mount links are not writable (like other
internal dirs).
> Missing executable permission on viewfs internal dirs
> -
>
> Key: HADOOP-10191
> URL: https://issues.apache.org/jira/browse/HADOOP-10191
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Gera Shegalov
>Priority: Blocker
> Attachments: HADOOP-10191.v01.patch
>
>
> ViewFileSystem allows 1) unconditional listing of internal directories (mount
> points) and 2) and changing work directories.
> 1) requires read permission
> 2) requires executable permission
> However, the hardcoded PERMISSION_RRR == 444 for FileStatus representing an
> internal dir does not have executable bit set.
> This confuses YARN localizer for public resources on viewfs because it
> requires executable permission for "other" on all of the ancestor directories
> of the resource.
> {code}
> java.io.IOException: Resource viewfs:/pubcache/cache.txt is not publicly
> accessable and as such cannot be part of the public cache.
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:182)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:51)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:279)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:277)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10191) Missing executable permission on viewfs internal dirs
[
https://issues.apache.org/jira/browse/HADOOP-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941913#comment-13941913
]
Chris Nauroth commented on HADOOP-10191:
[~jira.shegalov], the patch looks great. I just have one question.
All HDFS symlinks have their permissions set to 777. Conceptually, symlinks
don't really have permissions of their own. Instead, permission enforcement
checks are done using the permissions of the symlink target.
For consistency, I'm wondering if we should change this patch so that viewfs
internal dirs are set to 555 (as you have already done) and the mount link is
set to 777. From what I understand, this would still meet your needs for the
YARN localizer. What do you think?
> Missing executable permission on viewfs internal dirs
> -
>
> Key: HADOOP-10191
> URL: https://issues.apache.org/jira/browse/HADOOP-10191
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Gera Shegalov
>Priority: Blocker
> Attachments: HADOOP-10191.v01.patch
>
>
> ViewFileSystem allows 1) unconditional listing of internal directories (mount
> points) and 2) and changing work directories.
> 1) requires read permission
> 2) requires executable permission
> However, the hardcoded PERMISSION_RRR == 444 for FileStatus representing an
> internal dir does not have executable bit set.
> This confuses YARN localizer for public resources on viewfs because it
> requires executable permission for "other" on all of the ancestor directories
> of the resource.
> {code}
> java.io.IOException: Resource viewfs:/pubcache/cache.txt is not publicly
> accessable and as such cannot be part of the public cache.
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:182)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:51)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:279)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:277)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10191) Missing executable permission on viewfs internal dirs
[
https://issues.apache.org/jira/browse/HADOOP-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941407#comment-13941407
]
Chris Nauroth commented on HADOOP-10191:
Hi, [~jira.shegalov]. Thanks for the patch. This change makes sense to me.
Tomorrow morning, I'll put it through some additional testing and give it the
final review.
> Missing executable permission on viewfs internal dirs
> -
>
> Key: HADOOP-10191
> URL: https://issues.apache.org/jira/browse/HADOOP-10191
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Gera Shegalov
>Priority: Blocker
> Attachments: HADOOP-10191.v01.patch
>
>
> ViewFileSystem allows 1) unconditional listing of internal directories (mount
> points) and 2) and changing work directories.
> 1) requires read permission
> 2) requires executable permission
> However, the hardcoded PERMISSION_RRR == 444 for FileStatus representing an
> internal dir does not have executable bit set.
> This confuses YARN localizer for public resources on viewfs because it
> requires executable permission for "other" on all of the ancestor directories
> of the resource.
> {code}
> java.io.IOException: Resource viewfs:/pubcache/cache.txt is not publicly
> accessable and as such cannot be part of the public cache.
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:182)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:51)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:279)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:277)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10191) Missing executable permission on viewfs internal dirs
[
https://issues.apache.org/jira/browse/HADOOP-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13857250#comment-13857250
]
Hadoop QA commented on HADOOP-10191:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620576/HADOOP-10191.v01.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3382//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3382//console
This message is automatically generated.
> Missing executable permission on viewfs internal dirs
> -
>
> Key: HADOOP-10191
> URL: https://issues.apache.org/jira/browse/HADOOP-10191
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Gera Shegalov
> Attachments: HADOOP-10191.v01.patch
>
>
> ViewFileSystem allows 1) unconditional listing of internal directories (mount
> points) and 2) and changing work directories.
> 1) requires read permission
> 2) requires executable permission
> However, the hardcoded PERMISSION_RRR == 444 for FileStatus representing an
> internal dir does not have executable bit set.
> This confuses YARN localizer for public resources on viewfs because it
> requires executable permission for "other" on all of the ancestor directories
> of the resource.
> {code}
> java.io.IOException: Resource viewfs:/pubcache/cache.txt is not publicly
> accessable and as such cannot be part of the public cache.
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:182)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:51)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:279)
> at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:277)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
