[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798828#comment-13798828
]
Arun C Murthy commented on HADOOP-9652:
---
[~andrew.wang]? [~cmccabe]?
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Fix For: 2.3.0
Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
hadoop-9652-6.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Resolved] (HADOOP-10034) optimize same-filesystem symlinks by doing resolution server-side
[ https://issues.apache.org/jira/browse/HADOOP-10034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved HADOOP-10034. -- Resolution: Duplicate Dupe of HDFS-932 optimize same-filesystem symlinks by doing resolution server-side - Key: HADOOP-10034 URL: https://issues.apache.org/jira/browse/HADOOP-10034 Project: Hadoop Common Issue Type: Sub-task Components: fs Reporter: Colin Patrick McCabe We should optimize same-filesystem symlinks by doing resolution server-side rather than client side, as discussed on HADOOP-9780. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-9989) Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file.
[ https://issues.apache.org/jira/browse/HADOOP-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-9989: --- Fix Version/s: (was: 2.2.0) 2.2.1 Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file. Key: HADOOP-9989 URL: https://issues.apache.org/jira/browse/HADOOP-9989 Project: Hadoop Common Issue Type: Bug Components: security, util Affects Versions: 2.1.0-beta Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 Reporter: Jinghui Wang Fix For: 3.0.0, 2.2.1 Attachments: HADOOP-9989.patch Original Estimate: 0h Remaining Estimate: 0h The code in JIRA HADOOP-9374's patch introduced a bug, where the value of the tokenCacheFile parameter is being parsed as a binary file and set it to the mapreduce.job.credentials.json parameter in GenericOptionsParser, which cannot be parsed by JobSubmitter when it gets the value. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-9849) License information is missing
[ https://issues.apache.org/jira/browse/HADOOP-9849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-9849: --- Fix Version/s: (was: 2.2.0) 2.2.1 License information is missing -- Key: HADOOP-9849 URL: https://issues.apache.org/jira/browse/HADOOP-9849 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Timothy St. Clair Labels: newbie Fix For: 2.2.1 The following files are licensed under the BSD license but the BSD license is not part if the distribution: hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c I believe this file is BSD as well: hadoop-hdfs-project/hadoop-hdfs/src/main/native/util/tree.h -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10055) FileSystemShell.apt.vm doc has typo numRepicas
[ https://issues.apache.org/jira/browse/HADOOP-10055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799008#comment-13799008 ] Hudson commented on HADOOP-10055: - SUCCESS: Integrated in Hadoop-Yarn-trunk #366 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/366/]) HADOOP-10055. FileSystemShell.apt.vm doc has typo numRepicas. Contributed by Akira Ajisaka. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1533188) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm FileSystemShell.apt.vm doc has typo numRepicas - Key: HADOOP-10055 URL: https://issues.apache.org/jira/browse/HADOOP-10055 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.2.0 Reporter: Eli Collins Assignee: Akira AJISAKA Priority: Trivial Labels: newbie Fix For: 3.0.0, 2.2.1 Attachments: HADOOP-10055.patch HDFS-5139 added numRepicas to FileSystemShell.apt.vm, should be numReplicas. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10055) FileSystemShell.apt.vm doc has typo numRepicas
[ https://issues.apache.org/jira/browse/HADOOP-10055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799035#comment-13799035 ] Hudson commented on HADOOP-10055: - FAILURE: Integrated in Hadoop-Hdfs-trunk #1556 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1556/]) HADOOP-10055. FileSystemShell.apt.vm doc has typo numRepicas. Contributed by Akira Ajisaka. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1533188) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm FileSystemShell.apt.vm doc has typo numRepicas - Key: HADOOP-10055 URL: https://issues.apache.org/jira/browse/HADOOP-10055 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.2.0 Reporter: Eli Collins Assignee: Akira AJISAKA Priority: Trivial Labels: newbie Fix For: 3.0.0, 2.2.1 Attachments: HADOOP-10055.patch HDFS-5139 added numRepicas to FileSystemShell.apt.vm, should be numReplicas. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9291) enhance unit-test coverage of package o.a.h.metrics2
[ https://issues.apache.org/jira/browse/HADOOP-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799086#comment-13799086 ] Nathan Roberts commented on HADOOP-9291: Thanks Ivan. Updated patch (N8) looks good to me. +1 enhance unit-test coverage of package o.a.h.metrics2 Key: HADOOP-9291 URL: https://issues.apache.org/jira/browse/HADOOP-9291 Project: Hadoop Common Issue Type: Test Affects Versions: 3.0.0, 2.3.0 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Attachments: HADOOP-9291-branch-0.23--N4.patch, HADOOP-9291--N7.patch, HADOOP-9291--N8.patch, HADOOP-9291-trunk--N4.patch, HADOOP-9291-trunk--N5.patch, HADOOP-9291-trunk--N6.patch, HADOOP-9291-trunk--N6.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10055) FileSystemShell.apt.vm doc has typo numRepicas
[ https://issues.apache.org/jira/browse/HADOOP-10055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799094#comment-13799094 ] Hudson commented on HADOOP-10055: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1582 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1582/]) HADOOP-10055. FileSystemShell.apt.vm doc has typo numRepicas. Contributed by Akira Ajisaka. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1533188) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm FileSystemShell.apt.vm doc has typo numRepicas - Key: HADOOP-10055 URL: https://issues.apache.org/jira/browse/HADOOP-10055 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.2.0 Reporter: Eli Collins Assignee: Akira AJISAKA Priority: Trivial Labels: newbie Fix For: 3.0.0, 2.2.1 Attachments: HADOOP-10055.patch HDFS-5139 added numRepicas to FileSystemShell.apt.vm, should be numReplicas. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HADOOP-10059) RPC authentication and authorization metrics overflow to negative values on busy clusters
Jason Lowe created HADOOP-10059: --- Summary: RPC authentication and authorization metrics overflow to negative values on busy clusters Key: HADOOP-10059 URL: https://issues.apache.org/jira/browse/HADOOP-10059 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 2.2.0, 0.23.9 Reporter: Jason Lowe Priority: Minor The RPC metrics for authorization and authentication successes can easily overflow to negative values on a busy cluster that has been up for a long time. We should consider providing 64-bit values for these counters. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799167#comment-13799167
]
Suresh Srinivas commented on HADOOP-9652:
-
+1 for reverting this. Lets revisit this later with other symlink issues.
[~sanjay.radia], please consider this in symlink discussions.
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Fix For: 2.3.0
Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
hadoop-9652-6.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799361#comment-13799361
]
Colin Patrick McCabe commented on HADOOP-9652:
--
I think the easiest way to handle this is to enable the useLegacy thing. We
don't need a full revert. Will post a patch shortly.
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Fix For: 2.3.0
Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
hadoop-9652-6.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-9652:
-
Attachment: 0001-temporarily-disable-HADOOP-9652.patch
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Fix For: 2.3.0
Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-9652:
-
Status: Patch Available (was: Reopened)
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.3.0
Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Reopened] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe reopened HADOOP-9652:
--
Assignee: Colin Patrick McCabe (was: Andrew Wang)
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.3.0
Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HADOOP-10059) RPC authentication and authorization metrics overflow to negative values on busy clusters
[ https://issues.apache.org/jira/browse/HADOOP-10059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA reassigned HADOOP-10059: --- Assignee: Tsuyoshi OZAWA RPC authentication and authorization metrics overflow to negative values on busy clusters - Key: HADOOP-10059 URL: https://issues.apache.org/jira/browse/HADOOP-10059 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 0.23.9, 2.2.0 Reporter: Jason Lowe Assignee: Tsuyoshi OZAWA Priority: Minor The RPC metrics for authorization and authentication successes can easily overflow to negative values on a busy cluster that has been up for a long time. We should consider providing 64-bit values for these counters. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-10059) RPC authentication and authorization metrics overflow to negative values on busy clusters
[ https://issues.apache.org/jira/browse/HADOOP-10059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated HADOOP-10059: Attachment: HADOOP-10059.1.patch Updated to use MutableCounterLong for providing 64-bit values for rpcAuthenticationFailures/rpcAuthenticationSuccesses/rpcAuthorizationFailures/rpcAuthorizationSuccesses. RPC authentication and authorization metrics overflow to negative values on busy clusters - Key: HADOOP-10059 URL: https://issues.apache.org/jira/browse/HADOOP-10059 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 0.23.9, 2.2.0 Reporter: Jason Lowe Assignee: Tsuyoshi OZAWA Priority: Minor Attachments: HADOOP-10059.1.patch The RPC metrics for authorization and authentication successes can easily overflow to negative values on a busy cluster that has been up for a long time. We should consider providing 64-bit values for these counters. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-10059) RPC authentication and authorization metrics overflow to negative values on busy clusters
[ https://issues.apache.org/jira/browse/HADOOP-10059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated HADOOP-10059: Status: Patch Available (was: Open) RPC authentication and authorization metrics overflow to negative values on busy clusters - Key: HADOOP-10059 URL: https://issues.apache.org/jira/browse/HADOOP-10059 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 2.2.0, 0.23.9 Reporter: Jason Lowe Assignee: Tsuyoshi OZAWA Priority: Minor Attachments: HADOOP-10059.1.patch The RPC metrics for authorization and authentication successes can easily overflow to negative values on a busy cluster that has been up for a long time. We should consider providing 64-bit values for these counters. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799407#comment-13799407
]
Hadoop QA commented on HADOOP-9652:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609182/0001-temporarily-disable-HADOOP-9652.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.fs.TestSymlinkLocalFSFileSystem
org.apache.hadoop.fs.TestSymlinkLocalFSFileContext
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3235//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3235//console
This message is automatically generated.
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.3.0
Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799408#comment-13799408
]
Andrew Wang commented on HADOOP-9652:
-
Fixup patch LGTM, +1.
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.3.0
Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799409#comment-13799409
]
Colin Patrick McCabe commented on HADOOP-9652:
--
I'm going to update it to keep using the new {{getFileLinkStatus}} in
{{TestSymlinkLocalFSFileSystem}}. We don't want those tests to bitrot.
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.3.0
Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-10059) RPC authentication and authorization metrics overflow to negative values on busy clusters
[
https://issues.apache.org/jira/browse/HADOOP-10059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799427#comment-13799427
]
Hadoop QA commented on HADOOP-10059:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609184/HADOOP-10059.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ipc.TestRPC
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3236//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3236//console
This message is automatically generated.
RPC authentication and authorization metrics overflow to negative values on
busy clusters
-
Key: HADOOP-10059
URL: https://issues.apache.org/jira/browse/HADOOP-10059
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Affects Versions: 0.23.9, 2.2.0
Reporter: Jason Lowe
Assignee: Tsuyoshi OZAWA
Priority: Minor
Attachments: HADOOP-10059.1.patch
The RPC metrics for authorization and authentication successes can easily
overflow to negative values on a busy cluster that has been up for a long
time. We should consider providing 64-bit values for these counters.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799499#comment-13799499 ] Sanjay Radia commented on HADOOP-9984: -- *Background:* Applications often call listStatus and then call fileStatus.isDir() on each of the retuned children to decide if a node is a dir or a file. Such code would potentially break if any of the children are symlinks. This jira proposed that listStatus should follow any child symlinks and return a resolved list of children. Note symlinks that occur in the pathname passed to listStatus are always transparently followed and are not an issue. Also note that when symlinks was introduced, isDir() was deprecated and isDirectory(), isFile(), iSymlink() were added. *Compare with Posix:* Posix has separate readDir and stat/lstat. While readDir does not return the full status of each child, it does return the file type in the struct-dirent (i.e. regular file, dir, symlink etc). *Issue with following child symlinks* This lira's proposed solution (follow the child symlinks) has an issue. Comment [by daryn|https://issues.apache.org/jira/browse/HADOOP-9984?focusedCommentId=13786431page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13786431] and [Oct9th|https://issues.apache.org/jira/browse/HADOOP-9984?focusedCommentId=13790972page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13790972] in this jira shows potential problems with following child symlinks - the most egregious being the duplicate entry. *New Proposed Solution* listStatus should NOT follow child symlinks. Fix all internal utilities, hive, pig, map reduce, yarn, etc to not use isDir() and understand that a directory may contain symlinks. We have two choices for isDir() (which, btw, has already been deprecated) a) isDir() returns the file type of child without following the symlink (this is the code in trunk) b) isDir() returns the file type of child after following the symlink. ( unless the link is dangling). My own preference is (a). The argument in favor of (b) is that it would provide greater compatibility. I think regardless of which choice we pick we will break some apps; in that case I rather pick the cleaner solution, (a). FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch, HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799537#comment-13799537
]
Andrew Wang commented on HADOOP-9652:
-
I figured we were applying this fix only to branch-2 as a stopgap for a proper
fix using JNI later. Regardless, fixing the unit tests sounds like a good idea.
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.3.0
Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-10057) Add ability in Hadoop servers (Namenode, JobTracker, Datanode ) to support multiple QOP (Authentication , Privacy) simultaneously
[ https://issues.apache.org/jira/browse/HADOOP-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799555#comment-13799555 ] Benoy Antony commented on HADOOP-10057: --- Attaching the design document. Add ability in Hadoop servers (Namenode, JobTracker, Datanode ) to support multiple QOP (Authentication , Privacy) simultaneously -- Key: HADOOP-10057 URL: https://issues.apache.org/jira/browse/HADOOP-10057 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.2.1 Reporter: Benoy Antony Assignee: Benoy Antony Attachments: hadoop-10057-branch-1.2.patch, HADOOP-10057.pdf Add ability in Hadoop servers (Namenode, JobTracker Datanode ) to support multiple QOP (Authentication , Privacy) simlutaneously Hadoop Servers currently support only one QOP(quality of protection)for the whole cluster. We want Hadoop servers to support multiple QOP at the same time. The logic used to determine the QOP should be pluggable. This will enable hadoop servers to communicate with different types of clients with different QOP. A sample usecase: Let each Hadoop server support two QOP . 1. Authentication 2. Privacy (Privacy includes Authentication) . The Hadoop servers and internal clients require to do Authentication only without incurring cost of encryption. External clients use Privacy. An ip-whitelist logic to determine the QOP is provided and used as the default QOP resolution logic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-10057) Add ability in Hadoop servers (Namenode, JobTracker, Datanode ) to support multiple QOP (Authentication , Privacy) simultaneously
[ https://issues.apache.org/jira/browse/HADOOP-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10057: -- Attachment: HADOOP-10057.pdf Add ability in Hadoop servers (Namenode, JobTracker, Datanode ) to support multiple QOP (Authentication , Privacy) simultaneously -- Key: HADOOP-10057 URL: https://issues.apache.org/jira/browse/HADOOP-10057 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.2.1 Reporter: Benoy Antony Assignee: Benoy Antony Attachments: hadoop-10057-branch-1.2.patch, HADOOP-10057.pdf Add ability in Hadoop servers (Namenode, JobTracker Datanode ) to support multiple QOP (Authentication , Privacy) simlutaneously Hadoop Servers currently support only one QOP(quality of protection)for the whole cluster. We want Hadoop servers to support multiple QOP at the same time. The logic used to determine the QOP should be pluggable. This will enable hadoop servers to communicate with different types of clients with different QOP. A sample usecase: Let each Hadoop server support two QOP . 1. Authentication 2. Privacy (Privacy includes Authentication) . The Hadoop servers and internal clients require to do Authentication only without incurring cost of encryption. External clients use Privacy. An ip-whitelist logic to determine the QOP is provided and used as the default QOP resolution logic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-10057) Add ability in Hadoop servers (Namenode, JobTracker, Datanode ) to support multiple QOP (Authentication , Privacy) simultaneously
[ https://issues.apache.org/jira/browse/HADOOP-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10057: -- Attachment: (was: HADOOP-10057.pdf) Add ability in Hadoop servers (Namenode, JobTracker, Datanode ) to support multiple QOP (Authentication , Privacy) simultaneously -- Key: HADOOP-10057 URL: https://issues.apache.org/jira/browse/HADOOP-10057 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.2.1 Reporter: Benoy Antony Assignee: Benoy Antony Attachments: hadoop-10057-branch-1.2.patch, HADOOP-10057.pdf Add ability in Hadoop servers (Namenode, JobTracker Datanode ) to support multiple QOP (Authentication , Privacy) simlutaneously Hadoop Servers currently support only one QOP(quality of protection)for the whole cluster. We want Hadoop servers to support multiple QOP at the same time. The logic used to determine the QOP should be pluggable. This will enable hadoop servers to communicate with different types of clients with different QOP. A sample usecase: Let each Hadoop server support two QOP . 1. Authentication 2. Privacy (Privacy includes Authentication) . The Hadoop servers and internal clients require to do Authentication only without incurring cost of encryption. External clients use Privacy. An ip-whitelist logic to determine the QOP is provided and used as the default QOP resolution logic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-10057) Add ability in Hadoop servers (Namenode, JobTracker, Datanode ) to support multiple QOP (Authentication , Privacy) simultaneously
[ https://issues.apache.org/jira/browse/HADOOP-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10057: -- Attachment: HADOOP-10057.pdf Add ability in Hadoop servers (Namenode, JobTracker, Datanode ) to support multiple QOP (Authentication , Privacy) simultaneously -- Key: HADOOP-10057 URL: https://issues.apache.org/jira/browse/HADOOP-10057 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.2.1 Reporter: Benoy Antony Assignee: Benoy Antony Attachments: hadoop-10057-branch-1.2.patch, HADOOP-10057.pdf Add ability in Hadoop servers (Namenode, JobTracker Datanode ) to support multiple QOP (Authentication , Privacy) simlutaneously Hadoop Servers currently support only one QOP(quality of protection)for the whole cluster. We want Hadoop servers to support multiple QOP at the same time. The logic used to determine the QOP should be pluggable. This will enable hadoop servers to communicate with different types of clients with different QOP. A sample usecase: Let each Hadoop server support two QOP . 1. Authentication 2. Privacy (Privacy includes Authentication) . The Hadoop servers and internal clients require to do Authentication only without incurring cost of encryption. External clients use Privacy. An ip-whitelist logic to determine the QOP is provided and used as the default QOP resolution logic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799562#comment-13799562 ] Daryn Sharp commented on HADOOP-9984: - bq. listStatus should NOT follow child symlinks. Fix all internal utilities, hive, pig, map reduce, yarn, etc to not use isDir() and understand that a directory may contain symlinks. I do not agree. This means symlinks are not transparent and not compatible with pre-2.x. I also do not agree that any solution will/has to break existing apps. Furthermore, the user will rarely if ever care that something is a symlink. So requiring every user that gets a file status through any of the existing API methods should _not_ be burdened to check if it's a symlink, then resolve it before checking various criteria - this is about more than just isDir(). What if I'm checking file size? Or owner/group/permissions? I expect the results to be of the target, not the link. I think the only sensible solution to ensure compatibility: # A new filtered fs wrapper whose sole responsibility is resolving symlinks. FileSystem.get can automatically add the wrapper. If the user really wants to see symlinks, they can call getRawFs. # No other filesystem does symlink resolution of any kind. I've outlined in other jiras how having individual filesystems resolve symlinks is fundamentally broken, ex. viewfs. # The new symlink aware fs wrapper will return file statuses for symlinks that lazy resolve the file status ala RLFS. The lazy resolve handles the problem of unresolvable symlinks, that the user wasn't going to select based on name, from causing exceptions. Let's make hadoop work like every other filesystem by making symlinks be transparent unless the user explicitly wants to know if something is a symlink. FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch, HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799708#comment-13799708
]
Colin Patrick McCabe commented on HADOOP-9652:
--
I'm having some problems running the unit tests on trunk as well. Have the
symlink unit tests bitrotted already? Will take a look later.
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.3.0
Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799716#comment-13799716 ] Sanjay Radia commented on HADOOP-9984: -- bq. Let's make hadoop work like every other filesystem by making symlinks be transparent Unix's readir does return the file type - see my comment. So your statement is not true. It is mostly transparent. So your prefer the second option (b) for readDir. Is you layer file system proposal for fixing symlinks, an implementation choice for option (b) or something with fundamentally different semantics? FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch, HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
