[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[
https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969272#comment-13969272
]
Akira AJISAKA commented on HADOOP-8989:
---
Thanks [~jonallen] for updating the patch.
bq. How about splitting this to small patches?
+1 (non-binding) for splitting. If you don't have a time to do this, I'm
willing to help.
{code}
/**
* Construct a Print {@Expression} with an operational ASCII NULL
* suffix.
*/
private Print(boolean appendNull) {
{code}
{{@Expression}} should be {{@link Expression}}.
{code}
public int apply(int current, int shift, int value) {
return current | (value shift);
};
{code}
In {{Perm.java}} some semicolons can be removed.
hadoop dfs -find feature
Key: HADOOP-8989
URL: https://issues.apache.org/jira/browse/HADOOP-8989
Project: Hadoop Common
Issue Type: New Feature
Reporter: Marco Nicosia
Assignee: Jonathan Allen
Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch
Both sysadmins and users make frequent use of the unix 'find' command, but
Hadoop has no correlate. Without this, users are writing scripts which make
heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs
-lsr is somewhat taxing on the NameNode, and a really slow experience on the
client side. Possibly an in-NameNode find operation would be only a bit more
taxing on the NameNode, but significantly faster from the client's point of
view?
The minimum set of options I can think of which would make a Hadoop find
command generally useful is (in priority order):
* -type (file or directory, for now)
* -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments)
* -print0 (for piping to xargs -0)
* -depth
* -owner/-group (and -nouser/-nogroup)
* -name (allowing for shell pattern, or even regex?)
* -perm
* -size
One possible special case, but could possibly be really cool if it ran from
within the NameNode:
* -delete
The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow.
Lower priority, some people do use operators, mostly to execute -or searches
such as:
* find / \(-nouser -or -nogroup\)
Finally, I thought I'd include a link to the [Posix spec for
find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[ https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969277#comment-13969277 ] wenwupeng commented on HADOOP-10389: Thanks a lot [~cmccabe] clarification about libuv version and patch HADOOP-10389.004.patch ,now, compile passed in CentOS6.4. Native RPCv9 client --- Key: HADOOP-10389 URL: https://issues.apache.org/jira/browse/HADOOP-10389 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HADOOP-10388 Reporter: Binglin Chang Assignee: Colin Patrick McCabe Attachments: HADOOP-10388.001.patch, HADOOP-10389.002.patch, HADOOP-10389.004.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10251) Both NameNodes could be in STANDBY State if SNN network is unstable
[ https://issues.apache.org/jira/browse/HADOOP-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969340#comment-13969340 ] Vinayakumar B commented on HADOOP-10251: Hi.. Can please someone review this patch? Thanks in advance Both NameNodes could be in STANDBY State if SNN network is unstable --- Key: HADOOP-10251 URL: https://issues.apache.org/jira/browse/HADOOP-10251 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.2.0 Reporter: Vinayakumar B Assignee: Vinayakumar B Priority: Critical Attachments: HADOOP-10251.patch, HADOOP-10251.patch, HADOOP-10251.patch, HADOOP-10251.patch Following corner scenario happened in one of our cluster. 1. NN1 was Active and NN2 was Standby 2. NN2 machine's network was slow 3. NN1 got shutdown. 4. NN2 ZKFC got the notification and trying to check for old active for fencing. (This took little more time, again due to slow network) 5. In between, NN1 got restarted by our automatic monitoring, and ZKFC made it Active. 6. Now NN2 ZKFC got Old Active as NN2 and it did graceful fencing of NN1 to STANBY. 7. Before writing ActiveBreadCrumb to ZK, NN2 ZKFC got session timeout and got shutdown before making NN2 Active. *Now cluster having both NameNodes as STANDBY.* NN1 ZKFC still thinks that its nameNode is in Active state. NN2 ZKFC waiting for election. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10500) TestDoAsEffectiveUser fails on JDK7 due to failure to reset proxy user configuration.
[
https://issues.apache.org/jira/browse/HADOOP-10500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969443#comment-13969443
]
Hudson commented on HADOOP-10500:
-
SUCCESS: Integrated in Hadoop-Yarn-trunk #541 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/541/])
HADOOP-10500. TestDoAsEffectiveUser fails on JDK7 due to failure to reset proxy
user configuration. Contributed by Chris Nauroth. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1587361)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
TestDoAsEffectiveUser fails on JDK7 due to failure to reset proxy user
configuration.
-
Key: HADOOP-10500
URL: https://issues.apache.org/jira/browse/HADOOP-10500
Project: Hadoop Common
Issue Type: Bug
Components: security, test
Affects Versions: 3.0.0, 2.4.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Trivial
Fix For: 3.0.0, 2.5.0
Attachments: HADOOP-10500.1.patch, HADOOP-10500.2.patch
The proxy user configuration is held in a static variable. Various tests in
{{TestDoAsEffectiveUser}} mutate this data. JDK7 executes tests in a
different order than JDK6, so this can cause a failure if the state hasn't
been set explicitly to what the test needs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10500) TestDoAsEffectiveUser fails on JDK7 due to failure to reset proxy user configuration.
[
https://issues.apache.org/jira/browse/HADOOP-10500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969552#comment-13969552
]
Hudson commented on HADOOP-10500:
-
FAILURE: Integrated in Hadoop-Hdfs-trunk #1733 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1733/])
HADOOP-10500. TestDoAsEffectiveUser fails on JDK7 due to failure to reset proxy
user configuration. Contributed by Chris Nauroth. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1587361)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
TestDoAsEffectiveUser fails on JDK7 due to failure to reset proxy user
configuration.
-
Key: HADOOP-10500
URL: https://issues.apache.org/jira/browse/HADOOP-10500
Project: Hadoop Common
Issue Type: Bug
Components: security, test
Affects Versions: 3.0.0, 2.4.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Trivial
Fix For: 3.0.0, 2.5.0
Attachments: HADOOP-10500.1.patch, HADOOP-10500.2.patch
The proxy user configuration is held in a static variable. Various tests in
{{TestDoAsEffectiveUser}} mutate this data. JDK7 executes tests in a
different order than JDK6, so this can cause a failure if the state hasn't
been set explicitly to what the test needs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10503) Move junit up to v 4.12
Steve Loughran created HADOOP-10503: --- Summary: Move junit up to v 4.12 Key: HADOOP-10503 URL: https://issues.apache.org/jira/browse/HADOOP-10503 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 2.4.0 Reporter: Steve Loughran Priority: Minor JUnit 4.11 has been out for a while; other projects are happy with it, so update it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10503) Move junit up to v 4.11
[ https://issues.apache.org/jira/browse/HADOOP-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10503: Summary: Move junit up to v 4.11 (was: Move junit up to v 4.12) Move junit up to v 4.11 --- Key: HADOOP-10503 URL: https://issues.apache.org/jira/browse/HADOOP-10503 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 2.4.0 Reporter: Steve Loughran Priority: Minor JUnit 4.11 has been out for a while; other projects are happy with it, so update it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10498) Add support for proxy server
[ https://issues.apache.org/jira/browse/HADOOP-10498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969569#comment-13969569 ] Kihwal Lee commented on HADOOP-10498: - According to the IETF draft, http://tools.ietf.org/html/draft-ietf-appsawg-http-forwarded-10 , the header format may change to include information about multiple proxies involved in forwarding. Even if this becomes the standard, proxy servers will most likely still put the defacto X-Forwarded-For header for compatibility reasons. For this reason, I think it is reasonable to assume X-Forwarded-For for now. If the proposal becomes a standard, it will be more than just header name change, so making it configurable doesn't seem to add much value. +1 for the patch. Add support for proxy server Key: HADOOP-10498 URL: https://issues.apache.org/jira/browse/HADOOP-10498 Project: Hadoop Common Issue Type: New Feature Components: util Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HADOOP-10498.patch HDFS-6218 HDFS-6219 require support for configurable proxy servers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10500) TestDoAsEffectiveUser fails on JDK7 due to failure to reset proxy user configuration.
[
https://issues.apache.org/jira/browse/HADOOP-10500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969607#comment-13969607
]
Hudson commented on HADOOP-10500:
-
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1758 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1758/])
HADOOP-10500. TestDoAsEffectiveUser fails on JDK7 due to failure to reset proxy
user configuration. Contributed by Chris Nauroth. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1587361)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
TestDoAsEffectiveUser fails on JDK7 due to failure to reset proxy user
configuration.
-
Key: HADOOP-10500
URL: https://issues.apache.org/jira/browse/HADOOP-10500
Project: Hadoop Common
Issue Type: Bug
Components: security, test
Affects Versions: 3.0.0, 2.4.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Trivial
Fix For: 3.0.0, 2.5.0
Attachments: HADOOP-10500.1.patch, HADOOP-10500.2.patch
The proxy user configuration is held in a static variable. Various tests in
{{TestDoAsEffectiveUser}} mutate this data. JDK7 executes tests in a
different order than JDK6, so this can cause a failure if the state hasn't
been set explicitly to what the test needs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10498) Add support for proxy server
[ https://issues.apache.org/jira/browse/HADOOP-10498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-10498: - Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks all. I've committed to branch-2 and trunk. Add support for proxy server Key: HADOOP-10498 URL: https://issues.apache.org/jira/browse/HADOOP-10498 Project: Hadoop Common Issue Type: New Feature Components: util Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 3.0.0, 2.5.0 Attachments: HADOOP-10498.patch HDFS-6218 HDFS-6219 require support for configurable proxy servers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10501) Server#getHandlers() accesses handlers without synchronization
[
https://issues.apache.org/jira/browse/HADOOP-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969624#comment-13969624
]
Kihwal Lee commented on HADOOP-10501:
-
It's currently only meant for unit tests. Do you have any other use case in
mind?
Server#getHandlers() accesses handlers without synchronization
--
Key: HADOOP-10501
URL: https://issues.apache.org/jira/browse/HADOOP-10501
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
Iterable? extends Thread getHandlers() {
return Arrays.asList(handlers);
}
{code}
All the other methods accessing handlers are synchronized methods.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10498) Add support for proxy server
[ https://issues.apache.org/jira/browse/HADOOP-10498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969631#comment-13969631 ] Hudson commented on HADOOP-10498: - SUCCESS: Integrated in Hadoop-trunk-Commit #5519 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5519/]) HADOOP-10498. Add support for proxy server. (daryn) (daryn: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1587608) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ProxyUsers.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestProxyUsers.java Add support for proxy server Key: HADOOP-10498 URL: https://issues.apache.org/jira/browse/HADOOP-10498 Project: Hadoop Common Issue Type: New Feature Components: util Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 3.0.0, 2.5.0 Attachments: HADOOP-10498.patch HDFS-6218 HDFS-6219 require support for configurable proxy servers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10501) Server#getHandlers() accesses handlers without synchronization
[
https://issues.apache.org/jira/browse/HADOOP-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969659#comment-13969659
]
Ted Yu commented on HADOOP-10501:
-
I did see VisibleForTesting annotation.
Is it expected that tests may see inconsistent Handler array ?
Server#getHandlers() accesses handlers without synchronization
--
Key: HADOOP-10501
URL: https://issues.apache.org/jira/browse/HADOOP-10501
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
Iterable? extends Thread getHandlers() {
return Arrays.asList(handlers);
}
{code}
All the other methods accessing handlers are synchronized methods.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10504) Document proxy server support
Daryn Sharp created HADOOP-10504: Summary: Document proxy server support Key: HADOOP-10504 URL: https://issues.apache.org/jira/browse/HADOOP-10504 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 3.0.0, 2.5.0 Reporter: Daryn Sharp Document http proxy support introduced by HADOOP-10498. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10467) Enable proxyuser specification to support list of users in addition to list of groups.
[
https://issues.apache.org/jira/browse/HADOOP-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969696#comment-13969696
]
Daryn Sharp commented on HADOOP-10467:
--
Would you please elaborate on the use case and how the current config is
insufficient? Please include a conf snippet that demonstrates how this change
supports the new use case.
From a cursory glance at the code, {{cacheGroupsAdd}} appears to be a no-op
for most group providers so I don't think you can rely on that for
initialization.
Enable proxyuser specification to support list of users in addition to list
of groups.
--
Key: HADOOP-10467
URL: https://issues.apache.org/jira/browse/HADOOP-10467
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HADOOP-10467.patch, HADOOP-10467.patch
Today , the proxy user specification supports only list of groups. In some
cases, it is useful to specify the list of users in addition to list of
groups.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10501) Server#getHandlers() accesses handlers without synchronization
[
https://issues.apache.org/jira/browse/HADOOP-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969712#comment-13969712
]
Kihwal Lee commented on HADOOP-10501:
-
It should be okay as long as it is called after Server#start(). Adding
{{synchronized}} won't do much good, but won't hurt either.
Server#getHandlers() accesses handlers without synchronization
--
Key: HADOOP-10501
URL: https://issues.apache.org/jira/browse/HADOOP-10501
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
Iterable? extends Thread getHandlers() {
return Arrays.asList(handlers);
}
{code}
All the other methods accessing handlers are synchronized methods.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10505) LinuxContainerExecutor is incompatible with Simple Security mode.
jay vyas created HADOOP-10505:
-
Summary: LinuxContainerExecutor is incompatible with Simple
Security mode.
Key: HADOOP-10505
URL: https://issues.apache.org/jira/browse/HADOOP-10505
Project: Hadoop Common
Issue Type: Bug
Reporter: jay vyas
As of hadoop 2.3.0, commit cc74a18c makes it so that nonsecureLocalUser
replaces the user who submits a job if security is disabled:
{noformat}
return UserGroupInformation.isSecurityEnabled() ? user : nonsecureLocalUser;
{noformat}
However, the only way to enable security, is to NOT use SIMPLE authentication
mode:
{noformat}
public static boolean isSecurityEnabled() {
return !isAuthenticationMethodEnabled(AuthenticationMethod.SIMPLE);
}
{noformat}
Thus, the framework ENFORCES that SIMPLE login security -- nonSecureuser for
submission of LinuxExecutorContainer.
This results in a confusing issue, wherein we submit a job as sally and then
get an exception that user nobody is not whitelisted and has UID MAX_ID.
My proposed solution is that we should be able to leverage
LinuxContainerExector regardless of hadoop's view of the security settings on
the cluster, i.e. decouple LinuxContainerExecutor logic from the
isSecurityEnabled return value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10505) Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
[
https://issues.apache.org/jira/browse/HADOOP-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated HADOOP-10505:
--
Summary: Multitenant LinuxContainerExecutor is incompatible with Simple
Security mode. (was: LinuxContainerExecutor is incompatible with Simple
Security mode.)
Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
-
Key: HADOOP-10505
URL: https://issues.apache.org/jira/browse/HADOOP-10505
Project: Hadoop Common
Issue Type: Bug
Reporter: jay vyas
As of hadoop 2.3.0, commit cc74a18c makes it so that nonsecureLocalUser
replaces the user who submits a job if security is disabled:
{noformat}
return UserGroupInformation.isSecurityEnabled() ? user : nonsecureLocalUser;
{noformat}
However, the only way to enable security, is to NOT use SIMPLE authentication
mode:
{noformat}
public static boolean isSecurityEnabled() {
return !isAuthenticationMethodEnabled(AuthenticationMethod.SIMPLE);
}
{noformat}
Thus, the framework ENFORCES that SIMPLE login security -- nonSecureuser
for submission of LinuxExecutorContainer.
This results in a confusing issue, wherein we submit a job as sally and
then get an exception that user nobody is not whitelisted and has UID
MAX_ID.
My proposed solution is that we should be able to leverage
LinuxContainerExector regardless of hadoop's view of the security settings on
the cluster, i.e. decouple LinuxContainerExecutor logic from the
isSecurityEnabled return value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10251) Both NameNodes could be in STANDBY State if SNN network is unstable
[ https://issues.apache.org/jira/browse/HADOOP-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-10251: --- Description: Following corner scenario happened in one of our cluster. 1. NN1 was Active and NN2 was Standby 2. NN2 machine's network was slow 3. NN1 got shutdown. 4. NN2 ZKFC got the notification and trying to check for old active for fencing. (This took little more time, again due to slow network) 5. In between, NN1 got restarted by our automatic monitoring, and ZKFC made it Active. 6. Now NN2 ZKFC got Old Active as NN1 and it did graceful fencing of NN1 to STANBY. 7. Before writing ActiveBreadCrumb to ZK, NN2 ZKFC got session timeout and got shutdown before making NN2 Active. *Now cluster having both NameNodes as STANDBY.* NN1 ZKFC still thinks that its nameNode is in Active state. NN2 ZKFC waiting for election. was: Following corner scenario happened in one of our cluster. 1. NN1 was Active and NN2 was Standby 2. NN2 machine's network was slow 3. NN1 got shutdown. 4. NN2 ZKFC got the notification and trying to check for old active for fencing. (This took little more time, again due to slow network) 5. In between, NN1 got restarted by our automatic monitoring, and ZKFC made it Active. 6. Now NN2 ZKFC got Old Active as NN2 and it did graceful fencing of NN1 to STANBY. 7. Before writing ActiveBreadCrumb to ZK, NN2 ZKFC got session timeout and got shutdown before making NN2 Active. *Now cluster having both NameNodes as STANDBY.* NN1 ZKFC still thinks that its nameNode is in Active state. NN2 ZKFC waiting for election. Both NameNodes could be in STANDBY State if SNN network is unstable --- Key: HADOOP-10251 URL: https://issues.apache.org/jira/browse/HADOOP-10251 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.2.0 Reporter: Vinayakumar B Assignee: Vinayakumar B Priority: Critical Attachments: HADOOP-10251.patch, HADOOP-10251.patch, HADOOP-10251.patch, HADOOP-10251.patch Following corner scenario happened in one of our cluster. 1. NN1 was Active and NN2 was Standby 2. NN2 machine's network was slow 3. NN1 got shutdown. 4. NN2 ZKFC got the notification and trying to check for old active for fencing. (This took little more time, again due to slow network) 5. In between, NN1 got restarted by our automatic monitoring, and ZKFC made it Active. 6. Now NN2 ZKFC got Old Active as NN1 and it did graceful fencing of NN1 to STANBY. 7. Before writing ActiveBreadCrumb to ZK, NN2 ZKFC got session timeout and got shutdown before making NN2 Active. *Now cluster having both NameNodes as STANDBY.* NN1 ZKFC still thinks that its nameNode is in Active state. NN2 ZKFC waiting for election. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10499) Remove unused parameter from ProxyUsers.authorize()
[ https://issues.apache.org/jira/browse/HADOOP-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969721#comment-13969721 ] Daryn Sharp commented on HADOOP-10499: -- Although it makes sense to remove an unused parameter, I wonder if the parameter should be honored? It would allow different rpc services to have different proxy superuser configurations as opposed to the current static configuration... It probably made sense when daemons generally had only one rpc service. I don't have a strong opinion because I suppose it could be re-added/implemented by another jira. Remove unused parameter from ProxyUsers.authorize() --- Key: HADOOP-10499 URL: https://issues.apache.org/jira/browse/HADOOP-10499 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 2.4.0 Reporter: Benoy Antony Assignee: Benoy Antony Priority: Minor Attachments: HADOOP-10499.patch The Configuration parameter is not used in the authorize() function. It can be removed and callers can be updated. Attaching the simple patch which removes the unused _conf_ parameter and updates the callers. The ProxyUsers is defined as a private audience and so there shouldn't be any external callers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10505) Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
[
https://issues.apache.org/jira/browse/HADOOP-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969732#comment-13969732
]
jay vyas commented on HADOOP-10505:
---
Upon further investigation, you cannot hack the yarn-site.xml file.
1) If you thinly a non-simple field for the value of
hadoop.securiy.authentication (i.e. proxy , kerberos, kerberos_ssl,...)
then you will get a security method is not enabled exception.
2) If you try the other hack of a dummy security method (i.e.
hadoop.security.authentication = foo), you get the (expected) exception
Invalid attribute value of hadoop.security.authentication error.
So I dont see a good workaround, unless maybe there is a simple way to
implement a dummy implementation of security .
Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
-
Key: HADOOP-10505
URL: https://issues.apache.org/jira/browse/HADOOP-10505
Project: Hadoop Common
Issue Type: Bug
Reporter: jay vyas
As of hadoop 2.3.0, commit cc74a18c makes it so that nonsecureLocalUser
replaces the user who submits a job if security is disabled:
{noformat}
return UserGroupInformation.isSecurityEnabled() ? user : nonsecureLocalUser;
{noformat}
However, the only way to enable security, is to NOT use SIMPLE authentication
mode:
{noformat}
public static boolean isSecurityEnabled() {
return !isAuthenticationMethodEnabled(AuthenticationMethod.SIMPLE);
}
{noformat}
Thus, the framework ENFORCES that SIMPLE login security -- nonSecureuser
for submission of LinuxExecutorContainer.
This results in a confusing issue, wherein we submit a job as sally and
then get an exception that user nobody is not whitelisted and has UID
MAX_ID.
My proposed solution is that we should be able to leverage
LinuxContainerExector regardless of hadoop's view of the security settings on
the cluster, i.e. decouple LinuxContainerExecutor logic from the
isSecurityEnabled return value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10505) Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
[
https://issues.apache.org/jira/browse/HADOOP-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated HADOOP-10505:
--
Fix Version/s: 2.3.0
Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
-
Key: HADOOP-10505
URL: https://issues.apache.org/jira/browse/HADOOP-10505
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.3.0
Reporter: jay vyas
Labels: linux
Fix For: 2.3.0
As of hadoop 2.3.0, commit cc74a18c makes it so that nonsecureLocalUser
replaces the user who submits a job if security is disabled:
{noformat}
return UserGroupInformation.isSecurityEnabled() ? user : nonsecureLocalUser;
{noformat}
However, the only way to enable security, is to NOT use SIMPLE authentication
mode:
{noformat}
public static boolean isSecurityEnabled() {
return !isAuthenticationMethodEnabled(AuthenticationMethod.SIMPLE);
}
{noformat}
Thus, the framework ENFORCES that SIMPLE login security -- nonSecureuser
for submission of LinuxExecutorContainer.
This results in a confusing issue, wherein we submit a job as sally and
then get an exception that user nobody is not whitelisted and has UID
MAX_ID.
My proposed solution is that we should be able to leverage
LinuxContainerExector regardless of hadoop's view of the security settings on
the cluster, i.e. decouple LinuxContainerExecutor logic from the
isSecurityEnabled return value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10505) Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
[
https://issues.apache.org/jira/browse/HADOOP-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated HADOOP-10505:
--
Component/s: security
Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
-
Key: HADOOP-10505
URL: https://issues.apache.org/jira/browse/HADOOP-10505
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.3.0
Reporter: jay vyas
Labels: linux
Fix For: 2.3.0
As of hadoop 2.3.0, commit cc74a18c makes it so that nonsecureLocalUser
replaces the user who submits a job if security is disabled:
{noformat}
return UserGroupInformation.isSecurityEnabled() ? user : nonsecureLocalUser;
{noformat}
However, the only way to enable security, is to NOT use SIMPLE authentication
mode:
{noformat}
public static boolean isSecurityEnabled() {
return !isAuthenticationMethodEnabled(AuthenticationMethod.SIMPLE);
}
{noformat}
Thus, the framework ENFORCES that SIMPLE login security -- nonSecureuser
for submission of LinuxExecutorContainer.
This results in a confusing issue, wherein we submit a job as sally and
then get an exception that user nobody is not whitelisted and has UID
MAX_ID.
My proposed solution is that we should be able to leverage
LinuxContainerExector regardless of hadoop's view of the security settings on
the cluster, i.e. decouple LinuxContainerExecutor logic from the
isSecurityEnabled return value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10505) Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
[
https://issues.apache.org/jira/browse/HADOOP-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated HADOOP-10505:
--
Labels: linux (was: )
Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
-
Key: HADOOP-10505
URL: https://issues.apache.org/jira/browse/HADOOP-10505
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.3.0
Reporter: jay vyas
Labels: linux
Fix For: 2.3.0
As of hadoop 2.3.0, commit cc74a18c makes it so that nonsecureLocalUser
replaces the user who submits a job if security is disabled:
{noformat}
return UserGroupInformation.isSecurityEnabled() ? user : nonsecureLocalUser;
{noformat}
However, the only way to enable security, is to NOT use SIMPLE authentication
mode:
{noformat}
public static boolean isSecurityEnabled() {
return !isAuthenticationMethodEnabled(AuthenticationMethod.SIMPLE);
}
{noformat}
Thus, the framework ENFORCES that SIMPLE login security -- nonSecureuser
for submission of LinuxExecutorContainer.
This results in a confusing issue, wherein we submit a job as sally and
then get an exception that user nobody is not whitelisted and has UID
MAX_ID.
My proposed solution is that we should be able to leverage
LinuxContainerExector regardless of hadoop's view of the security settings on
the cluster, i.e. decouple LinuxContainerExecutor logic from the
isSecurityEnabled return value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10505) Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
[
https://issues.apache.org/jira/browse/HADOOP-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated HADOOP-10505:
--
Affects Version/s: 2.3.0
Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
-
Key: HADOOP-10505
URL: https://issues.apache.org/jira/browse/HADOOP-10505
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.3.0
Reporter: jay vyas
Labels: linux
Fix For: 2.3.0
As of hadoop 2.3.0, commit cc74a18c makes it so that nonsecureLocalUser
replaces the user who submits a job if security is disabled:
{noformat}
return UserGroupInformation.isSecurityEnabled() ? user : nonsecureLocalUser;
{noformat}
However, the only way to enable security, is to NOT use SIMPLE authentication
mode:
{noformat}
public static boolean isSecurityEnabled() {
return !isAuthenticationMethodEnabled(AuthenticationMethod.SIMPLE);
}
{noformat}
Thus, the framework ENFORCES that SIMPLE login security -- nonSecureuser
for submission of LinuxExecutorContainer.
This results in a confusing issue, wherein we submit a job as sally and
then get an exception that user nobody is not whitelisted and has UID
MAX_ID.
My proposed solution is that we should be able to leverage
LinuxContainerExector regardless of hadoop's view of the security settings on
the cluster, i.e. decouple LinuxContainerExecutor logic from the
isSecurityEnabled return value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10505) Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
[
https://issues.apache.org/jira/browse/HADOOP-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated HADOOP-10505:
--
Priority: Critical (was: Major)
Multitenant LinuxContainerExecutor is incompatible with Simple Security mode.
-
Key: HADOOP-10505
URL: https://issues.apache.org/jira/browse/HADOOP-10505
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.3.0
Reporter: jay vyas
Priority: Critical
Labels: linux
Fix For: 2.3.0
As of hadoop 2.3.0, commit cc74a18c makes it so that nonsecureLocalUser
replaces the user who submits a job if security is disabled:
{noformat}
return UserGroupInformation.isSecurityEnabled() ? user : nonsecureLocalUser;
{noformat}
However, the only way to enable security, is to NOT use SIMPLE authentication
mode:
{noformat}
public static boolean isSecurityEnabled() {
return !isAuthenticationMethodEnabled(AuthenticationMethod.SIMPLE);
}
{noformat}
Thus, the framework ENFORCES that SIMPLE login security -- nonSecureuser
for submission of LinuxExecutorContainer.
This results in a confusing issue, wherein we submit a job as sally and
then get an exception that user nobody is not whitelisted and has UID
MAX_ID.
My proposed solution is that we should be able to leverage
LinuxContainerExector regardless of hadoop's view of the security settings on
the cluster, i.e. decouple LinuxContainerExecutor logic from the
isSecurityEnabled return value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10499) Remove unused parameter from ProxyUsers.authorize()
[
https://issues.apache.org/jira/browse/HADOOP-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969756#comment-13969756
]
Chris Nauroth commented on HADOOP-10499:
[~benoyantony], would you please investigate the compilation failure in HDFS
{{JspHelper}} and post a new patch?
bq. It would allow different rpc services to have different proxy superuser
configurations as opposed to the current static configuration.
I wonder if part of HADOOP-10448 is intended to support this requirement in a
better way. Benoy, can you respond to Daryn's question too? Thanks!
Remove unused parameter from ProxyUsers.authorize()
---
Key: HADOOP-10499
URL: https://issues.apache.org/jira/browse/HADOOP-10499
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Affects Versions: 2.4.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Priority: Minor
Attachments: HADOOP-10499.patch
The Configuration parameter is not used in the authorize() function. It can
be removed and callers can be updated.
Attaching the simple patch which removes the unused _conf_ parameter and
updates the callers.
The ProxyUsers is defined as a private audience and so there shouldn't be any
external callers.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-7688) When a servlet filter throws an exception in init(..), the Jetty server failed silently.
[ https://issues.apache.org/jira/browse/HADOOP-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-7688: --- Fix Version/s: 0.23.11 When a servlet filter throws an exception in init(..), the Jetty server failed silently. - Key: HADOOP-7688 URL: https://issues.apache.org/jira/browse/HADOOP-7688 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.23.0, 0.24.0 Reporter: Tsz Wo Nicholas Sze Assignee: Uma Maheswara Rao G Fix For: 1.2.0, 3.0.0, 2.0.3-alpha, 0.23.11 Attachments: HADOOP-7688-branch-1.patch, HADOOP-7688-branch-2.patch, HADOOP-7688.patch, filter-init-exception-test.patch, org.apache.hadoop.http.TestServletFilter-output.txt When a servlet filter throws a ServletException in init(..), the exception is logged by Jetty but not re-throws to the caller. As a result, the Jetty server failed silently. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10504) Document proxy server support
[
https://issues.apache.org/jira/browse/HADOOP-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969789#comment-13969789
]
Kihwal Lee commented on HADOOP-10504:
-
{{hadoop-common-project/hadoop-common/src/site/apt/SecureMode.apt.vm}} needs to
be updated. Since the change is already in, I am making it a blocker for 2.5.0.
Document proxy server support
-
Key: HADOOP-10504
URL: https://issues.apache.org/jira/browse/HADOOP-10504
Project: Hadoop Common
Issue Type: Improvement
Components: documentation
Affects Versions: 3.0.0, 2.5.0
Reporter: Daryn Sharp
Document http proxy support introduced by HADOOP-10498.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10504) Document proxy server support
[ https://issues.apache.org/jira/browse/HADOOP-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-10504: Component/s: (was: util) documentation Priority: Blocker (was: Major) Target Version/s: 2.5.0 (was: 3.0.0, 2.5.0) Document proxy server support - Key: HADOOP-10504 URL: https://issues.apache.org/jira/browse/HADOOP-10504 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 3.0.0, 2.5.0 Reporter: Daryn Sharp Priority: Blocker Document http proxy support introduced by HADOOP-10498. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10503) Move junit up to v 4.11
[ https://issues.apache.org/jira/browse/HADOOP-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10503: --- Attachment: HADOOP-10503.1.patch +1 for the proposal. Here is a patch. Let's see what Jenkins thinks of it. Before we commit anything, I'd like to get a full test run on Windows too. I'm planning on queuing that up for this evening. Move junit up to v 4.11 --- Key: HADOOP-10503 URL: https://issues.apache.org/jira/browse/HADOOP-10503 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 2.4.0 Reporter: Steve Loughran Priority: Minor Attachments: HADOOP-10503.1.patch JUnit 4.11 has been out for a while; other projects are happy with it, so update it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10503) Move junit up to v 4.11
[ https://issues.apache.org/jira/browse/HADOOP-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10503: --- Status: Patch Available (was: Open) Move junit up to v 4.11 --- Key: HADOOP-10503 URL: https://issues.apache.org/jira/browse/HADOOP-10503 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 2.4.0 Reporter: Steve Loughran Priority: Minor Attachments: HADOOP-10503.1.patch JUnit 4.11 has been out for a while; other projects are happy with it, so update it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10503) Move junit up to v 4.11
[ https://issues.apache.org/jira/browse/HADOOP-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10503: --- Target Version/s: 3.0.0, 2.5.0 Move junit up to v 4.11 --- Key: HADOOP-10503 URL: https://issues.apache.org/jira/browse/HADOOP-10503 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 2.4.0 Reporter: Steve Loughran Assignee: Chris Nauroth Priority: Minor Attachments: HADOOP-10503.1.patch JUnit 4.11 has been out for a while; other projects are happy with it, so update it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10503) Move junit up to v 4.11
[ https://issues.apache.org/jira/browse/HADOOP-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth reassigned HADOOP-10503: -- Assignee: Chris Nauroth Move junit up to v 4.11 --- Key: HADOOP-10503 URL: https://issues.apache.org/jira/browse/HADOOP-10503 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 2.4.0 Reporter: Steve Loughran Assignee: Chris Nauroth Priority: Minor Attachments: HADOOP-10503.1.patch JUnit 4.11 has been out for a while; other projects are happy with it, so update it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10503) Move junit up to v 4.11
[
https://issues.apache.org/jira/browse/HADOOP-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969857#comment-13969857
]
Hadoop QA commented on HADOOP-10503:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12640303/HADOOP-10503.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 2331 javac
compiler warnings (more than the trunk's current 1288 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-maven-plugins.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3796//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3796//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3796//console
This message is automatically generated.
Move junit up to v 4.11
---
Key: HADOOP-10503
URL: https://issues.apache.org/jira/browse/HADOOP-10503
Project: Hadoop Common
Issue Type: Sub-task
Components: build
Affects Versions: 2.4.0
Reporter: Steve Loughran
Assignee: Chris Nauroth
Priority: Minor
Attachments: HADOOP-10503.1.patch
JUnit 4.11 has been out for a while; other projects are happy with it, so
update it.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10506) LimitedPrivate annotation not useful
Thomas Graves created HADOOP-10506: -- Summary: LimitedPrivate annotation not useful Key: HADOOP-10506 URL: https://issues.apache.org/jira/browse/HADOOP-10506 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0, 3.0.0 Reporter: Thomas Graves -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10503) Move junit up to v 4.11
[ https://issues.apache.org/jira/browse/HADOOP-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969885#comment-13969885 ] Chris Nauroth commented on HADOOP-10503: bq. -1 javac. The applied patch generated 2331 javac compiler warnings (more than the trunk's current 1288 warnings). Wow, that's fun. :-) JUnit 4.11 has deprecated some things that we're calling. This will be a pretty large (but mundane) cleanup effort. Move junit up to v 4.11 --- Key: HADOOP-10503 URL: https://issues.apache.org/jira/browse/HADOOP-10503 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 2.4.0 Reporter: Steve Loughran Assignee: Chris Nauroth Priority: Minor Attachments: HADOOP-10503.1.patch JUnit 4.11 has been out for a while; other projects are happy with it, so update it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10499) Remove unused parameter from ProxyUsers.authorize()
[ https://issues.apache.org/jira/browse/HADOOP-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10499: -- Attachment: HADOOP-10499.patch Attaching the patch which corrects all the calls to _authorize_ . I did not run a mvn clean and hence the classes which were not changed did not get recompiled. Sorry for that. Remove unused parameter from ProxyUsers.authorize() --- Key: HADOOP-10499 URL: https://issues.apache.org/jira/browse/HADOOP-10499 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 2.4.0 Reporter: Benoy Antony Assignee: Benoy Antony Priority: Minor Attachments: HADOOP-10499.patch, HADOOP-10499.patch The Configuration parameter is not used in the authorize() function. It can be removed and callers can be updated. Attaching the simple patch which removes the unused _conf_ parameter and updates the callers. The ProxyUsers is defined as a private audience and so there shouldn't be any external callers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10499) Remove unused parameter from ProxyUsers.authorize()
[ https://issues.apache.org/jira/browse/HADOOP-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969949#comment-13969949 ] Benoy Antony commented on HADOOP-10499: --- I believe , the static nature of the ProxyUser capability should be addressed separately, if there is a requirement for different services to have different ProxyUser capabilities. HADOOP-10448 would make it easier since it will move the state out of ProxyUsers. I am trying to clean up the interface exposed by _ProxyUsers_ in order to work on HADOOP-10448 and that's why this jira. Remove unused parameter from ProxyUsers.authorize() --- Key: HADOOP-10499 URL: https://issues.apache.org/jira/browse/HADOOP-10499 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 2.4.0 Reporter: Benoy Antony Assignee: Benoy Antony Priority: Minor Attachments: HADOOP-10499.patch, HADOOP-10499.patch The Configuration parameter is not used in the authorize() function. It can be removed and callers can be updated. Attaching the simple patch which removes the unused _conf_ parameter and updates the callers. The ProxyUsers is defined as a private audience and so there shouldn't be any external callers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10506) LimitedPrivate annotation not useful
[ https://issues.apache.org/jira/browse/HADOOP-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-10506: --- Release Note: (was: The LimitedPrivate annotation isn't useful. The intention seems to have been those interfaces were only intended to be used by these components. But in many cases those components are separate from core hadoop. This means any changes to them will break backwards compatibility with those, which breaks the new compatibility rules in Hadoop. Note that many of the annotation are also not marked properly, or have fallen out of date. I see Public Interfaces that use LimitedPrivate classes in the api. (TokenCache using Credentials is an example). ) LimitedPrivate annotation not useful Key: HADOOP-10506 URL: https://issues.apache.org/jira/browse/HADOOP-10506 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Thomas Graves -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10506) LimitedPrivate annotation not useful
[ https://issues.apache.org/jira/browse/HADOOP-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-10506: --- Description: The LimitedPrivate annotation isn't useful. The intention seems to have been those interfaces were only intended to be used by these components. But in many cases those components are separate from core hadoop. This means any changes to them will break backwards compatibility with those, which breaks the new compatibility rules in Hadoop. Note that many of the annotation are also not marked properly, or have fallen out of date. I see Public Interfaces that use LimitedPrivate classes in the api. (TokenCache using Credentials is an example). LimitedPrivate annotation not useful Key: HADOOP-10506 URL: https://issues.apache.org/jira/browse/HADOOP-10506 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Thomas Graves The LimitedPrivate annotation isn't useful. The intention seems to have been those interfaces were only intended to be used by these components. But in many cases those components are separate from core hadoop. This means any changes to them will break backwards compatibility with those, which breaks the new compatibility rules in Hadoop. Note that many of the annotation are also not marked properly, or have fallen out of date. I see Public Interfaces that use LimitedPrivate classes in the api. (TokenCache using Credentials is an example). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10507) FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is specified
[
https://issues.apache.org/jira/browse/HADOOP-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-10507:
---
Target Version/s: 3.0.0, 2.5.0
FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is
specified
--
Key: HADOOP-10507
URL: https://issues.apache.org/jira/browse/HADOOP-10507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 2.4.0
Reporter: Stephen Chu
Assignee: sathish
Priority: Minor
Attachments: HDFS-6205.patch
If users don't specify the perm of an acl when using the FsShell's setfacl
command, a fatal internal error ArrayIndexOutOfBoundsException will be thrown.
{code}
[root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob: /user/hdfs/td1
-setfacl: Fatal internal error
java.lang.ArrayIndexOutOfBoundsException: 2
at
org.apache.hadoop.fs.permission.AclEntry.parseAclEntry(AclEntry.java:285)
at
org.apache.hadoop.fs.permission.AclEntry.parseAclSpec(AclEntry.java:221)
at
org.apache.hadoop.fs.shell.AclCommands$SetfaclCommand.processOptions(AclCommands.java:260)
at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
[root@hdfs-nfs ~]#
{code}
An improvement would be if it returned something like this:
{code}
[root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob:rww /user/hdfs/td1
-setfacl: Invalid permission in aclSpec : user:bob:rww
Usage: hadoop fs [generic options] -setfacl [-R] [{-b|-k} {-m|-x acl_spec}
path]|[--set acl_spec path]
[root@hdfs-nfs ~]#
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Moved] (HADOOP-10507) FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is specified
[
https://issues.apache.org/jira/browse/HADOOP-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth moved HDFS-6205 to HADOOP-10507:
--
Component/s: (was: hdfs-client)
fs
Affects Version/s: (was: 2.4.0)
(was: 3.0.0)
3.0.0
2.4.0
Key: HADOOP-10507 (was: HDFS-6205)
Project: Hadoop Common (was: Hadoop HDFS)
FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is
specified
--
Key: HADOOP-10507
URL: https://issues.apache.org/jira/browse/HADOOP-10507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.4.0, 3.0.0
Reporter: Stephen Chu
Assignee: sathish
Priority: Minor
Attachments: HDFS-6205.patch
If users don't specify the perm of an acl when using the FsShell's setfacl
command, a fatal internal error ArrayIndexOutOfBoundsException will be thrown.
{code}
[root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob: /user/hdfs/td1
-setfacl: Fatal internal error
java.lang.ArrayIndexOutOfBoundsException: 2
at
org.apache.hadoop.fs.permission.AclEntry.parseAclEntry(AclEntry.java:285)
at
org.apache.hadoop.fs.permission.AclEntry.parseAclSpec(AclEntry.java:221)
at
org.apache.hadoop.fs.shell.AclCommands$SetfaclCommand.processOptions(AclCommands.java:260)
at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
[root@hdfs-nfs ~]#
{code}
An improvement would be if it returned something like this:
{code}
[root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob:rww /user/hdfs/td1
-setfacl: Invalid permission in aclSpec : user:bob:rww
Usage: hadoop fs [generic options] -setfacl [-R] [{-b|-k} {-m|-x acl_spec}
path]|[--set acl_spec path]
[root@hdfs-nfs ~]#
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10507) FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is specified
[
https://issues.apache.org/jira/browse/HADOOP-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969987#comment-13969987
]
Chris Nauroth commented on HADOOP-10507:
I moved this issue from HDFS to HADOOP, because the bug fix is in hadoop-common
code.
FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is
specified
--
Key: HADOOP-10507
URL: https://issues.apache.org/jira/browse/HADOOP-10507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 2.4.0
Reporter: Stephen Chu
Assignee: sathish
Priority: Minor
Attachments: HDFS-6205.patch
If users don't specify the perm of an acl when using the FsShell's setfacl
command, a fatal internal error ArrayIndexOutOfBoundsException will be thrown.
{code}
[root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob: /user/hdfs/td1
-setfacl: Fatal internal error
java.lang.ArrayIndexOutOfBoundsException: 2
at
org.apache.hadoop.fs.permission.AclEntry.parseAclEntry(AclEntry.java:285)
at
org.apache.hadoop.fs.permission.AclEntry.parseAclSpec(AclEntry.java:221)
at
org.apache.hadoop.fs.shell.AclCommands$SetfaclCommand.processOptions(AclCommands.java:260)
at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
[root@hdfs-nfs ~]#
{code}
An improvement would be if it returned something like this:
{code}
[root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob:rww /user/hdfs/td1
-setfacl: Invalid permission in aclSpec : user:bob:rww
Usage: hadoop fs [generic options] -setfacl [-R] [{-b|-k} {-m|-x acl_spec}
path]|[--set acl_spec path]
[root@hdfs-nfs ~]#
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10499) Remove unused parameter from ProxyUsers.authorize()
[ https://issues.apache.org/jira/browse/HADOOP-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969992#comment-13969992 ] Chris Nauroth commented on HADOOP-10499: +1 for the new patch, pending Jenkins run. After this passes Jenkins, I'll split this into separate issues/separate patches for HADOOP, HDFS and MAPREDUCE. (We're supposed to track the commits to the different sub-projects separately.) Remove unused parameter from ProxyUsers.authorize() --- Key: HADOOP-10499 URL: https://issues.apache.org/jira/browse/HADOOP-10499 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 2.4.0 Reporter: Benoy Antony Assignee: Benoy Antony Priority: Minor Attachments: HADOOP-10499.patch, HADOOP-10499.patch The Configuration parameter is not used in the authorize() function. It can be removed and callers can be updated. Attaching the simple patch which removes the unused _conf_ parameter and updates the callers. The ProxyUsers is defined as a private audience and so there shouldn't be any external callers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10506) LimitedPrivate annotation not useful
[ https://issues.apache.org/jira/browse/HADOOP-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969998#comment-13969998 ] Tsz Wo Nicholas Sze commented on HADOOP-10506: -- ... But in many cases those components are separate from core hadoop. This means any changes to them will break backwards compatibility with those, ... This part I don't agree. When changing a LimitedPrivate API, we should aware that the depended components, which may sit outside core, should still work with the change. In some cases, we may also make change to the depended components. LimitedPrivate does not imply backwards compatibility since the LimitedPrivate APIs could be Unstable. It just means that the depended components are using the API. Note that many of the annotation are also not marked properly, or have fallen out of date. I see Public Interfaces that use LimitedPrivate classes in the api. (TokenCache using Credentials is an example). Okay, there is an ambiguity -- When a Public API method using LimitedPrivate classes, does it mean that the Public API method automatically become a LimitedPrivate method (this is similar to the Java private methods in a Public classes; they are not Public APIs)? Or Public APIs are not allowed using LimitedPrivate. (Then, it is a bug as you mentioned but it does not mean LimitedPrivate annotation not useful.) Anyway, I do agree that we do not define LimitedPrivate very carefully so that there are confusions in these special cases. LimitedPrivate annotation not useful Key: HADOOP-10506 URL: https://issues.apache.org/jira/browse/HADOOP-10506 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Thomas Graves The LimitedPrivate annotation isn't useful. The intention seems to have been those interfaces were only intended to be used by these components. But in many cases those components are separate from core hadoop. This means any changes to them will break backwards compatibility with those, which breaks the new compatibility rules in Hadoop. Note that many of the annotation are also not marked properly, or have fallen out of date. I see Public Interfaces that use LimitedPrivate classes in the api. (TokenCache using Credentials is an example). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10507) FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is specified
[
https://issues.apache.org/jira/browse/HADOOP-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970008#comment-13970008
]
Chris Nauroth commented on HADOOP-10507:
One more minor nitpick: please camel-case method names. Thanks!
FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is
specified
--
Key: HADOOP-10507
URL: https://issues.apache.org/jira/browse/HADOOP-10507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 2.4.0
Reporter: Stephen Chu
Assignee: sathish
Priority: Minor
Attachments: HDFS-6205.patch
If users don't specify the perm of an acl when using the FsShell's setfacl
command, a fatal internal error ArrayIndexOutOfBoundsException will be thrown.
{code}
[root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob: /user/hdfs/td1
-setfacl: Fatal internal error
java.lang.ArrayIndexOutOfBoundsException: 2
at
org.apache.hadoop.fs.permission.AclEntry.parseAclEntry(AclEntry.java:285)
at
org.apache.hadoop.fs.permission.AclEntry.parseAclSpec(AclEntry.java:221)
at
org.apache.hadoop.fs.shell.AclCommands$SetfaclCommand.processOptions(AclCommands.java:260)
at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
[root@hdfs-nfs ~]#
{code}
An improvement would be if it returned something like this:
{code}
[root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob:rww /user/hdfs/td1
-setfacl: Invalid permission in aclSpec : user:bob:rww
Usage: hadoop fs [generic options] -setfacl [-R] [{-b|-k} {-m|-x acl_spec}
path]|[--set acl_spec path]
[root@hdfs-nfs ~]#
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970026#comment-13970026 ] Jonathan Allen commented on HADOOP-8989: I'm happy to split into small patches but could do with advice on how to do this. Are you suggesting creating extra issues or multiple files attached to this one or something else? hadoop dfs -find feature Key: HADOOP-8989 URL: https://issues.apache.org/jira/browse/HADOOP-8989 Project: Hadoop Common Issue Type: New Feature Reporter: Marco Nicosia Assignee: Jonathan Allen Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch Both sysadmins and users make frequent use of the unix 'find' command, but Hadoop has no correlate. Without this, users are writing scripts which make heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs -lsr is somewhat taxing on the NameNode, and a really slow experience on the client side. Possibly an in-NameNode find operation would be only a bit more taxing on the NameNode, but significantly faster from the client's point of view? The minimum set of options I can think of which would make a Hadoop find command generally useful is (in priority order): * -type (file or directory, for now) * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) * -print0 (for piping to xargs -0) * -depth * -owner/-group (and -nouser/-nogroup) * -name (allowing for shell pattern, or even regex?) * -perm * -size One possible special case, but could possibly be really cool if it ran from within the NameNode: * -delete The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow. Lower priority, some people do use operators, mostly to execute -or searches such as: * find / \(-nouser -or -nogroup\) Finally, I thought I'd include a link to the [Posix spec for find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10506) LimitedPrivate annotation not useful
[ https://issues.apache.org/jira/browse/HADOOP-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970028#comment-13970028 ] Jason Lowe commented on HADOOP-10506: - bq. LimitedPrivate does not imply backwards compatibility since the LimitedPrivate APIs could be Unstable. It just means that the depended components are using the API. The problem is if we ignore backwards compatibility when changing classes marked LimitedPrivate used by downstream components then we are creating a knife-switch type of upgrade, where upgrading the Hadoop core requires a simultaneous corresponding update of hive/pig/hbase/pick-your-favorite-LimitedPrivate-user. That becomes particularly problematic when supporting a rolling upgrade of the cluster, as tasks launched by the downstream component on new nodes may suddenly fail to find a LimitedPrivate method/class they relied upon. IMHO that means we need to preserve backwards compatibility for LimitedPrivate APIs otherwise we can't do rolling upgrades when those APIs break. LimitedPrivate annotation not useful Key: HADOOP-10506 URL: https://issues.apache.org/jira/browse/HADOOP-10506 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Thomas Graves The LimitedPrivate annotation isn't useful. The intention seems to have been those interfaces were only intended to be used by these components. But in many cases those components are separate from core hadoop. This means any changes to them will break backwards compatibility with those, which breaks the new compatibility rules in Hadoop. Note that many of the annotation are also not marked properly, or have fallen out of date. I see Public Interfaces that use LimitedPrivate classes in the api. (TokenCache using Credentials is an example). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10508) RefreshCallQueue fails when authorization is enabled
Chris Li created HADOOP-10508: - Summary: RefreshCallQueue fails when authorization is enabled Key: HADOOP-10508 URL: https://issues.apache.org/jira/browse/HADOOP-10508 Project: Hadoop Common Issue Type: Bug Components: ipc Affects Versions: 2.4.0 Reporter: Chris Li Assignee: Chris Li When hadoop.security.authorization=true, the callqueue cannot be refreshed with hadoop dfsadmin -refreshCallQueue (protocol not found). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10506) LimitedPrivate annotation not useful
[ https://issues.apache.org/jira/browse/HADOOP-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970033#comment-13970033 ] Tsz Wo Nicholas Sze commented on HADOOP-10506: -- I did not say that LimitedPrivate could not have backwards compatibility. I just say that LimitedPrivate APIs could also be @Stable, @Evolving or @Unstable. LimitedPrivate annotation not useful Key: HADOOP-10506 URL: https://issues.apache.org/jira/browse/HADOOP-10506 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Thomas Graves The LimitedPrivate annotation isn't useful. The intention seems to have been those interfaces were only intended to be used by these components. But in many cases those components are separate from core hadoop. This means any changes to them will break backwards compatibility with those, which breaks the new compatibility rules in Hadoop. Note that many of the annotation are also not marked properly, or have fallen out of date. I see Public Interfaces that use LimitedPrivate classes in the api. (TokenCache using Credentials is an example). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10507) FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is specified
[
https://issues.apache.org/jira/browse/HADOOP-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970032#comment-13970032
]
Hadoop QA commented on HADOOP-10507:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639364/HDFS-6205.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3798//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3798//console
This message is automatically generated.
FsShell setfacl can throw ArrayIndexOutOfBoundsException when no perm is
specified
--
Key: HADOOP-10507
URL: https://issues.apache.org/jira/browse/HADOOP-10507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 2.4.0
Reporter: Stephen Chu
Assignee: sathish
Priority: Minor
Attachments: HDFS-6205.patch
If users don't specify the perm of an acl when using the FsShell's setfacl
command, a fatal internal error ArrayIndexOutOfBoundsException will be thrown.
{code}
[root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob: /user/hdfs/td1
-setfacl: Fatal internal error
java.lang.ArrayIndexOutOfBoundsException: 2
at
org.apache.hadoop.fs.permission.AclEntry.parseAclEntry(AclEntry.java:285)
at
org.apache.hadoop.fs.permission.AclEntry.parseAclSpec(AclEntry.java:221)
at
org.apache.hadoop.fs.shell.AclCommands$SetfaclCommand.processOptions(AclCommands.java:260)
at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
[root@hdfs-nfs ~]#
{code}
An improvement would be if it returned something like this:
{code}
[root@hdfs-nfs ~]# hdfs dfs -setfacl -m user:bob:rww /user/hdfs/td1
-setfacl: Invalid permission in aclSpec : user:bob:rww
Usage: hadoop fs [generic options] -setfacl [-R] [{-b|-k} {-m|-x acl_spec}
path]|[--set acl_spec path]
[root@hdfs-nfs ~]#
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10508) RefreshCallQueue fails when authorization is enabled
[ https://issues.apache.org/jira/browse/HADOOP-10508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10508: -- Attachment: HADOOP-10508.patch RefreshCallQueue fails when authorization is enabled Key: HADOOP-10508 URL: https://issues.apache.org/jira/browse/HADOOP-10508 Project: Hadoop Common Issue Type: Bug Components: ipc Affects Versions: 2.4.0 Reporter: Chris Li Assignee: Chris Li Attachments: HADOOP-10508.patch When hadoop.security.authorization=true, the callqueue cannot be refreshed with hadoop dfsadmin -refreshCallQueue (protocol not found). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10508) RefreshCallQueue fails when authorization is enabled
[ https://issues.apache.org/jira/browse/HADOOP-10508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Li updated HADOOP-10508: -- Status: Patch Available (was: Open) Patch registers RefreshCallQueueProtocol.class in the HDFSPolicyProvider so that it can be called when authorization is enabled. Access control is done by setting security.refresh.callqueue.protocol.acl in the policy.xml file RefreshCallQueue fails when authorization is enabled Key: HADOOP-10508 URL: https://issues.apache.org/jira/browse/HADOOP-10508 Project: Hadoop Common Issue Type: Bug Components: ipc Affects Versions: 2.4.0 Reporter: Chris Li Assignee: Chris Li Attachments: HADOOP-10508.patch When hadoop.security.authorization=true, the callqueue cannot be refreshed with hadoop dfsadmin -refreshCallQueue (protocol not found). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10467) Enable proxyuser specification to support list of users in addition to list of groups.
[ https://issues.apache.org/jira/browse/HADOOP-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970074#comment-13970074 ] Benoy Antony commented on HADOOP-10467: --- In some cases, it takes a longer time to add users to groups. In those situations, the ability to specify usernames in the _proxyuserconfiguration_ will enable administrators to quickly unblock users. That's the use case. Similar ability is available in yarn queue acls. An implementation choice would have been to add a separate property for users alone. But to maintain parity with queue acls , I added usergroups which can accept users and groups. If maintaining parity with Queue ACL specification is not important, I can create a new property - users instead of usergroups. The syntax is _user1,user2SPACEgroup1,group2_ (Same as in queue acls). Since these properties are per user, they are not in core-default.xml. I can specify the syntax and new configuration in the release notes. Would that be appropriate ? Enable proxyuser specification to support list of users in addition to list of groups. -- Key: HADOOP-10467 URL: https://issues.apache.org/jira/browse/HADOOP-10467 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Benoy Antony Assignee: Benoy Antony Attachments: HADOOP-10467.patch, HADOOP-10467.patch Today , the proxy user specification supports only list of groups. In some cases, it is useful to specify the list of users in addition to list of groups. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10467) Enable proxyuser specification to support list of users in addition to list of groups.
[ https://issues.apache.org/jira/browse/HADOOP-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10467: -- Attachment: HADOOP-10467.patch This patch does not rely on _cacheGroupsAdd_ . Invocation of _cacheGroupsAdd_ was already existing to initialize NetGroups. I have modified the patch so that code to initialize groups s sharec for usergoups/groups. Enable proxyuser specification to support list of users in addition to list of groups. -- Key: HADOOP-10467 URL: https://issues.apache.org/jira/browse/HADOOP-10467 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Benoy Antony Assignee: Benoy Antony Attachments: HADOOP-10467.patch, HADOOP-10467.patch, HADOOP-10467.patch Today , the proxy user specification supports only list of groups. In some cases, it is useful to specify the list of users in addition to list of groups. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10506) LimitedPrivate annotation not useful
[
https://issues.apache.org/jira/browse/HADOOP-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970085#comment-13970085
]
Thomas Graves commented on HADOOP-10506:
So you are sayings it purely informational and if I'm a closely related
product I can use it and should update the annotation? Backwards compatibility
guarantees are by the InterfaceStability tag but in the LimitedPrivate we will
contact/negotiate with the components listed before making any changes.
So I guess the question is what is a closely related product. Do all apache
products fall into that category?
One example of this that seems a bit ridiculous is UserGroupInformation:
@InterfaceAudience.LimitedPrivate({HDFS, MapReduce, HBase, Hive,
Oozie})
LimitedPrivate annotation not useful
Key: HADOOP-10506
URL: https://issues.apache.org/jira/browse/HADOOP-10506
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.4.0
Reporter: Thomas Graves
The LimitedPrivate annotation isn't useful. The intention seems to have been
those interfaces were only intended to be used by these components. But in
many cases those components are separate from core hadoop. This means any
changes to them will break backwards compatibility with those, which breaks
the new compatibility rules in Hadoop.
Note that many of the annotation are also not marked properly, or have fallen
out of date. I see Public Interfaces that use LimitedPrivate classes in the
api. (TokenCache using Credentials is an example).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10499) Remove unused parameter from ProxyUsers.authorize()
[
https://issues.apache.org/jira/browse/HADOOP-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970123#comment-13970123
]
Hadoop QA commented on HADOOP-10499:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12640316/HADOOP-10499.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:
org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3797//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3797//console
This message is automatically generated.
Remove unused parameter from ProxyUsers.authorize()
---
Key: HADOOP-10499
URL: https://issues.apache.org/jira/browse/HADOOP-10499
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Affects Versions: 2.4.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Priority: Minor
Attachments: HADOOP-10499.patch, HADOOP-10499.patch
The Configuration parameter is not used in the authorize() function. It can
be removed and callers can be updated.
Attaching the simple patch which removes the unused _conf_ parameter and
updates the callers.
The ProxyUsers is defined as a private audience and so there shouldn't be any
external callers.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10506) LimitedPrivate annotation not useful
[
https://issues.apache.org/jira/browse/HADOOP-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970127#comment-13970127
]
Haohui Mai commented on HADOOP-10506:
-
bq. The problem is if we ignore backwards compatibility when changing classes
marked LimitedPrivate used by downstream components then we are creating a
knife-switch type of upgrade, where upgrading the Hadoop core requires a
simultaneous corresponding update of
hive/pig/hbase/pick-your-favorite-LimitedPrivate-user.
Note that {{LimitedPrivate}} is a workaround of the lack of a strong concept of
symbol visibility in Java. It is a contract that is enforced voluntarily, but
not by the compiler. What it means is that classes with this annotation are not
supposed to be visible to the dependent projects. This is a contract between
hadoop-core and dependent projects.
Please consider the following scenarios:
# My code uses Java reflection to get an instance of {{BlockReaderLocal}},
which is a class only visible in the same package. The Java compiler will check
for that.
# My code uses {{RemoteBlockReader2}}, which is annotated with
{{InterfaceAudience.Private}}. The compiler won't check whether the code
violates the contract of visibility.
Note that in both cases they break the exact same contract of visibility. The
only difference is that whether the Java compiler can check for it. If you
agree that I probably should fix for (1), because I'm doing it with my own
risks, how come I can dodge the responsibility of fixing for (2), which is
almost identical to the scenario described in this jira?
I understand that hadoop-core might not have the cleanest separation of public
/ private APIs, and it is important to be compatible. Having the options of
being incompatible, however, is critical to move forward in projects like
hadoop-core. I think a more reasonable approach is to drive some of the APIs to
be public and to clarify the contracts of compatibility between hadoop-core and
dependent projects, but not removing the contract and losing the option of
being incompatible in minor releases.
LimitedPrivate annotation not useful
Key: HADOOP-10506
URL: https://issues.apache.org/jira/browse/HADOOP-10506
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.4.0
Reporter: Thomas Graves
The LimitedPrivate annotation isn't useful. The intention seems to have been
those interfaces were only intended to be used by these components. But in
many cases those components are separate from core hadoop. This means any
changes to them will break backwards compatibility with those, which breaks
the new compatibility rules in Hadoop.
Note that many of the annotation are also not marked properly, or have fallen
out of date. I see Public Interfaces that use LimitedPrivate classes in the
api. (TokenCache using Credentials is an example).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10467) Enable proxyuser specification to support list of users in addition to list of groups.
[ https://issues.apache.org/jira/browse/HADOOP-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10467: -- Attachment: HADOOP-10467.patch Modified hadoop-common-project/hadoop-common/src/site/apt/SecureMode.apt.vm to add documentation and the syntax of the value. Enable proxyuser specification to support list of users in addition to list of groups. -- Key: HADOOP-10467 URL: https://issues.apache.org/jira/browse/HADOOP-10467 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Benoy Antony Assignee: Benoy Antony Attachments: HADOOP-10467.patch, HADOOP-10467.patch, HADOOP-10467.patch, HADOOP-10467.patch Today , the proxy user specification supports only list of groups. In some cases, it is useful to specify the list of users in addition to list of groups. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10499) Remove unused parameter from ProxyUsers.authorize()
[ https://issues.apache.org/jira/browse/HADOOP-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970139#comment-13970139 ] Benoy Antony commented on HADOOP-10499: --- I don't think that the test case failure is related to the changes. Remove unused parameter from ProxyUsers.authorize() --- Key: HADOOP-10499 URL: https://issues.apache.org/jira/browse/HADOOP-10499 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 2.4.0 Reporter: Benoy Antony Assignee: Benoy Antony Priority: Minor Attachments: HADOOP-10499.patch, HADOOP-10499.patch The Configuration parameter is not used in the authorize() function. It can be removed and callers can be updated. Attaching the simple patch which removes the unused _conf_ parameter and updates the callers. The ProxyUsers is defined as a private audience and so there shouldn't be any external callers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10506) LimitedPrivate annotation not useful
[ https://issues.apache.org/jira/browse/HADOOP-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970243#comment-13970243 ] Thomas Graves commented on HADOOP-10506: Also to clarify, I do understand that the limitedprivate was probably added for historical reasons (the api's weren't properly categorized before) so components used them. We have made great improvements to get them categorized for the official 2.2 release and we couldn't get everything fixed up due to timing. So perhaps the title of the jira isn't correct and should be updated. Personally I think the following should be done: 1) clarify documentation what limitedPrivate is. This means us internally agreeing on what it really means. 2) No new classes/interfaces should use this tag. They should be properly classified as either public or private. If there was a bug and class not properly tagged its fine to use there. 3) All existing classes with this tag should be eventually be deprecated and we should work towards that. I realize this isn't going to happen immediately as everyone has to balance this with features and other bug fixes. If one component is using the interface more then likely its useful for other components and more then likely other components are already using it. Thus in my opinion the documentation of listing the component its private to isn't very useful. The longer we wait with this the more applications (in the case of yarn) and components will use these api's and it will be harder to change them. Yes we can say it was marked a certain way and the other components shouldn't have used it but when its ambiguous like this and we don't provide an equivalent Public api I don't see how we can defend that. Also as more and more components are created it gets harder and harder to upgrade/deploy them all at once. Hence why all the push for HA and rolling upgrades. Customers don't want downtime. I do agree we need to have the option for moving forward. But I think this can be done with a proper set of Public apis. The reason I ran into this was because I am writing an application to run on yarn and also another one to just read HDFS files. Its almost impossible (if not impossible) to do that properly (with security) without using classes marked LimitedPrivate right now. If it is possible it would require copying lots of code. We should be making writing applications as easy as possible. LimitedPrivate annotation not useful Key: HADOOP-10506 URL: https://issues.apache.org/jira/browse/HADOOP-10506 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Thomas Graves The LimitedPrivate annotation isn't useful. The intention seems to have been those interfaces were only intended to be used by these components. But in many cases those components are separate from core hadoop. This means any changes to them will break backwards compatibility with those, which breaks the new compatibility rules in Hadoop. Note that many of the annotation are also not marked properly, or have fallen out of date. I see Public Interfaces that use LimitedPrivate classes in the api. (TokenCache using Credentials is an example). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10467) Enable proxyuser specification to support list of users in addition to list of groups.
[
https://issues.apache.org/jira/browse/HADOOP-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970253#comment-13970253
]
Hadoop QA commented on HADOOP-10467:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12640347/HADOOP-10467.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3800//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3800//console
This message is automatically generated.
Enable proxyuser specification to support list of users in addition to list
of groups.
--
Key: HADOOP-10467
URL: https://issues.apache.org/jira/browse/HADOOP-10467
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HADOOP-10467.patch, HADOOP-10467.patch,
HADOOP-10467.patch, HADOOP-10467.patch
Today , the proxy user specification supports only list of groups. In some
cases, it is useful to specify the list of users in addition to list of
groups.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970295#comment-13970295 ] Akira AJISAKA commented on HADOOP-8989: --- I suggest to create extra issues. You can create sub-tasks and attach a small patch on each issue. hadoop dfs -find feature Key: HADOOP-8989 URL: https://issues.apache.org/jira/browse/HADOOP-8989 Project: Hadoop Common Issue Type: New Feature Reporter: Marco Nicosia Assignee: Jonathan Allen Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch Both sysadmins and users make frequent use of the unix 'find' command, but Hadoop has no correlate. Without this, users are writing scripts which make heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs -lsr is somewhat taxing on the NameNode, and a really slow experience on the client side. Possibly an in-NameNode find operation would be only a bit more taxing on the NameNode, but significantly faster from the client's point of view? The minimum set of options I can think of which would make a Hadoop find command generally useful is (in priority order): * -type (file or directory, for now) * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) * -print0 (for piping to xargs -0) * -depth * -owner/-group (and -nouser/-nogroup) * -name (allowing for shell pattern, or even regex?) * -perm * -size One possible special case, but could possibly be really cool if it ran from within the NameNode: * -delete The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow. Lower priority, some people do use operators, mostly to execute -or searches such as: * find / \(-nouser -or -nogroup\) Finally, I thought I'd include a link to the [Posix spec for find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10509) cancelToken doesn't work in some instances
Mohammad Kamrul Islam created HADOOP-10509:
--
Summary: cancelToken doesn't work in some instances
Key: HADOOP-10509
URL: https://issues.apache.org/jira/browse/HADOOP-10509
Project: Hadoop Common
Issue Type: Bug
Components: security
Reporter: Mohammad Kamrul Islam
Assignee: Mohammad Kamrul Islam
When the owner of a token tries to explicitly cancel the token, it gets the
following error/exception
{noformat}
2014-04-14 20:07:35,744 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException
as:someuser/machine_name.linkedin.com@realm.LINKEDIN.COM (auth:KERBEROS)
cause:org.apache.hadoop.security.AccessControlException: someuser is not
authorized to cancel the token
2014-04-14 20:07:35,744 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2
on 10020, call
org.apache.hadoop.mapreduce.v2.api.HSClientProtocolPB.cancelDelegationToken
from 172.20.158.61:49042 Call#4 Retry#0: error:
org.apache.hadoop.security.AccessControlException: someuser is not authorized
to cancel the token
org.apache.hadoop.security.AccessControlException: someuser is not authorized
to cancel the token
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.cancelToken(AbstractDelegationTokenSecretManager.java:429)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.cancelDelegationToken(HistoryClientService.java:400)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.cancelDelegationToken(MRClientProtocolPBServiceImpl.java:286)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:301)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956)
{noformat}
Details:
AbstractDelegationTokenSecretManager.cacelToken() gets the owner as full
principal name where as the canceller is the short name.
The potential code snippets:
{code}
String owner = id.getUser().getUserName();
Text renewer = id.getRenewer();
HadoopKerberosName cancelerKrbName = new HadoopKerberosName(canceller);
String cancelerShortName = cancelerKrbName.getShortName();
if (!canceller.equals(owner)
(renewer == null || renewer.toString().isEmpty() ||
!cancelerShortName
.equals(renewer.toString( {
throw new AccessControlException(canceller
+ is not authorized to cancel the token);
}
{code}
The code shows 'owner' gets the full principal name. Where as the value of
'canceller' depends on who is calling it.
In some cases, it is the short name. REF: HistoryClientService.java
{code}
String user = UserGroupInformation.getCurrentUser().getShortUserName();
jhsDTSecretManager.cancelToken(token, user);
{code}
In other cases, the value could be full principal name. REF: FSNamesystem.java.
{code}
String canceller = getRemoteUser().getUserName();
DelegationTokenIdentifier id = dtSecretManager
.cancelToken(token, canceller);
{code}
Possible resolution:
--
Option 1: in cancelToken() method, compare with both : short name and full
principal name.
Pros: Easy. Have to change in one place.
Cons: Someone can argue that it is hacky!
Option 2:
All the caller sends the consistent value as 'canceller' : either short name or
full principal name.
Pros: Cleaner.
Cons: A lot of code changes and potential bug injections.
I'm open for both options.
Please give your opinion.
Btw, how it is working now in most cases? The short name and the full
principal name are usually the same for end-users.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10467) Enable proxyuser specification to support list of users in addition to list of groups.
[
https://issues.apache.org/jira/browse/HADOOP-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970301#comment-13970301
]
Hadoop QA commented on HADOOP-10467:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12640347/HADOOP-10467.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3801//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3801//console
This message is automatically generated.
Enable proxyuser specification to support list of users in addition to list
of groups.
--
Key: HADOOP-10467
URL: https://issues.apache.org/jira/browse/HADOOP-10467
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HADOOP-10467.patch, HADOOP-10467.patch,
HADOOP-10467.patch, HADOOP-10467.patch
Today , the proxy user specification supports only list of groups. In some
cases, it is useful to specify the list of users in addition to list of
groups.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10508) RefreshCallQueue fails when authorization is enabled
[
https://issues.apache.org/jira/browse/HADOOP-10508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970312#comment-13970312
]
Hadoop QA commented on HADOOP-10508:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12640332/HADOOP-10508.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3799//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3799//console
This message is automatically generated.
RefreshCallQueue fails when authorization is enabled
Key: HADOOP-10508
URL: https://issues.apache.org/jira/browse/HADOOP-10508
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 2.4.0
Reporter: Chris Li
Assignee: Chris Li
Attachments: HADOOP-10508.patch
When hadoop.security.authorization=true, the callqueue cannot be refreshed
with hadoop dfsadmin -refreshCallQueue (protocol not found).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-8808) Update FsShell documentation to mention deprecation of some of the commands, and mention alternatives
[ https://issues.apache.org/jira/browse/HADOOP-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-8808: -- Target Version/s: 2.5.0 (was: 2.4.0) Update FsShell documentation to mention deprecation of some of the commands, and mention alternatives - Key: HADOOP-8808 URL: https://issues.apache.org/jira/browse/HADOOP-8808 Project: Hadoop Common Issue Type: Bug Components: documentation, fs Affects Versions: 2.2.0 Reporter: Hemanth Yamijala Assignee: Akira AJISAKA Attachments: HADOOP-8808.2.patch, HADOOP-8808.patch In HADOOP-7286, we deprecated the following 3 commands dus, lsr and rmr, in favour of du -s, ls -r and rm -r respectively. The FsShell documentation should be updated to mention these, so that users can start switching. Also, there are places where we refer to the deprecated commands as alternatives. This can be changed as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9907) Webapp http://hostname:port/metrics link is not working
[ https://issues.apache.org/jira/browse/HADOOP-9907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-9907: -- Target Version/s: 2.5.0 (was: 2.4.0) Webapp http://hostname:port/metrics link is not working - Key: HADOOP-9907 URL: https://issues.apache.org/jira/browse/HADOOP-9907 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.3.0 Reporter: Jian He Assignee: Akira AJISAKA Priority: Critical Attachments: HADOOP-9907.patch This link is not working which just shows a blank page. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10150) Hadoop cryptographic file system
[ https://issues.apache.org/jira/browse/HADOOP-10150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10150: Attachment: HDFSDataatRestEncryptionAttackVectors.pdf HDFSDataatRestEncryptionProposal.pdf HDFSDataAtRestEncryptionAlternatives.pdf [Cross-posting with HDFS-6134, closed HDFS-6134 as duplicate, discussion to continue here] Larry, Steve, Nicholas, thanks for your comments. Todd Lipcon and I had an offline discussion with Andrew Purtell, Yi Liu and Avik Dey to see if we could combine what HADOOP-10150 and HDFS-6134 into one proposal while supporting both, encryption for multiple filesystems and transparent encryption for HDFS. Also, following Steve’s suggestion, I’ve put together a Attack Vectors Matrix for all approaches. I think both documents, the proposal and the attack vectors, address most if not all the questions/concerns raised in the JIRA. Attaching 3 documents: * Alternatives: the original doc posted in HDFS-6134 * Proposal: the combined proposal * Attack Vectors: a matrix with the different attacks for the alternatives and the proposal Hadoop cryptographic file system Key: HADOOP-10150 URL: https://issues.apache.org/jira/browse/HADOOP-10150 Project: Hadoop Common Issue Type: New Feature Components: security Affects Versions: 3.0.0 Reporter: Yi Liu Assignee: Yi Liu Labels: rhino Fix For: 3.0.0 Attachments: CryptographicFileSystem.patch, HADOOP cryptographic file system-V2.docx, HADOOP cryptographic file system.pdf, HDFSDataAtRestEncryptionAlternatives.pdf, HDFSDataatRestEncryptionAttackVectors.pdf, HDFSDataatRestEncryptionProposal.pdf, cfs.patch, extended information based on INode feature.patch There is an increasing need for securing data when Hadoop customers use various upper layer applications, such as Map-Reduce, Hive, Pig, HBase and so on. HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based on HADOOP “FilterFileSystem” decorating DFS or other file systems, and transparent to upper layer applications. It’s configurable, scalable and fast. High level requirements: 1.Transparent to and no modification required for upper layer applications. 2.“Seek”, “PositionedReadable” are supported for input stream of CFS if the wrapped file system supports them. 3.Very high performance for encryption and decryption, they will not become bottleneck. 4.Can decorate HDFS and all other file systems in Hadoop, and will not modify existing structure of file system, such as namenode and datanode structure if the wrapped file system is HDFS. 5.Admin can configure encryption policies, such as which directory will be encrypted. 6.A robust key management framework. 7.Support Pread and append operations if the wrapped file system supports them. -- This message was sent by Atlassian JIRA (v6.2#6252)
