[jira] [Moved] (HADOOP-11736) KMSClientProvider addDelegationToken does not notify callers when Auth failure is due to Proxy User configuration a
[
https://issues.apache.org/jira/browse/HADOOP-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh moved HDFS-7970 to HADOOP-11736:
Key: HADOOP-11736 (was: HDFS-7970)
Project: Hadoop Common (was: Hadoop HDFS)
KMSClientProvider addDelegationToken does not notify callers when Auth
failure is due to Proxy User configuration a
Key: HADOOP-11736
URL: https://issues.apache.org/jira/browse/HADOOP-11736
Project: Hadoop Common
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Attachments: HDFS-7970.1.patch
When a process such as YARN RM tries to create/renew a KMS DelegationToken on
behalf of proxy user such as Llama/Impala and if the Proxy user rules are not
correctly configured to allow yarn to proxy the required user, then the
following is found in the RM logs :
{noformat}
Unable to add the application to the delegation token renewer.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:887)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:132)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:129)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.addDelegationTokens(LoadBalancingKMSClientProvider.java:129)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
..
..
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:874)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:869)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
... 21 more
{noformat}
This gives no information to the user as to why the call has failed, and
there is generally no way for an admin to know the the ProxyUser setting is
the issue without going thru the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11736) KMSClientProvider addDelegationToken does not notify callers when Auth failure is due to Proxy User configuration a
[
https://issues.apache.org/jira/browse/HADOOP-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14374972#comment-14374972
]
Arun Suresh commented on HADOOP-11736:
--
Moving this to Hadoop common, since most of the fixes are in the hadoop-common
/ auth packages
KMSClientProvider addDelegationToken does not notify callers when Auth
failure is due to Proxy User configuration a
Key: HADOOP-11736
URL: https://issues.apache.org/jira/browse/HADOOP-11736
Project: Hadoop Common
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Attachments: HDFS-7970.1.patch
When a process such as YARN RM tries to create/renew a KMS DelegationToken on
behalf of proxy user such as Llama/Impala and if the Proxy user rules are not
correctly configured to allow yarn to proxy the required user, then the
following is found in the RM logs :
{noformat}
Unable to add the application to the delegation token renewer.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:887)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:132)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:129)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.addDelegationTokens(LoadBalancingKMSClientProvider.java:129)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
..
..
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:874)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:869)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
... 21 more
{noformat}
This gives no information to the user as to why the call has failed, and
there is generally no way for an admin to know the the ProxyUser setting is
the issue without going thru the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
GUI Cluster Manager
guys, I know there is cluster manager for Cloudera and Ambari for Hortonworks - but are there any alternatives that people are actually using for straight apache? I'm thinking folks using Apache still need some type of dashboard... right? Thanks
[jira] [Commented] (HADOOP-11736) KMSClientProvider addDelegationToken does not notify callers when Auth failure is due to Proxy User configuration a
[
https://issues.apache.org/jira/browse/HADOOP-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375026#comment-14375026
]
Hadoop QA commented on HADOOP-11736:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706390/HADOOP-11736.1.patch
against trunk revision 4cd54d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common
hadoop-common-project/hadoop-kms.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5982//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5982//console
This message is automatically generated.
KMSClientProvider addDelegationToken does not notify callers when Auth
failure is due to Proxy User configuration a
Key: HADOOP-11736
URL: https://issues.apache.org/jira/browse/HADOOP-11736
Project: Hadoop Common
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Attachments: HADOOP-11736.1.patch, HDFS-7970.1.patch
When a long running process such as YARN RM tries to create/renew a KMS
DelegationToken on behalf of proxy user and if the Proxy user rules are not
correctly configured to allow yarn to proxy the required user, then the
following is found in the RM logs :
{noformat}
Unable to add the application to the delegation token renewer.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:887)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:132)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:129)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.addDelegationTokens(LoadBalancingKMSClientProvider.java:129)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
..
..
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:874)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:869)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
... 21 more
{noformat}
This gives no information to the user as to why the call has failed, and
there is generally no way for an admin to know the the ProxyUser setting is
the issue without going thru the code.
--
This message was sent by Atlassian JIRA
[jira] [Updated] (HADOOP-11736) KMSClientProvider addDelegationToken does not notify callers when Auth failure is due to Proxy User configuration a
[
https://issues.apache.org/jira/browse/HADOOP-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-11736:
-
Description:
When a long running process such as YARN RM tries to create/renew a KMS
DelegationToken on behalf of proxy user and if the Proxy user rules are not
correctly configured to allow yarn to proxy the required user, then the
following is found in the RM logs :
{noformat}
Unable to add the application to the delegation token renewer.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:887)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:132)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:129)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.addDelegationTokens(LoadBalancingKMSClientProvider.java:129)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
..
..
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:874)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:869)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
... 21 more
{noformat}
This gives no information to the user as to why the call has failed, and there
is generally no way for an admin to know the the ProxyUser setting is the issue
without going thru the code.
was:
When a process such as YARN RM tries to create/renew a KMS DelegationToken on
behalf of proxy user such as Llama/Impala and if the Proxy user rules are not
correctly configured to allow yarn to proxy the required user, then the
following is found in the RM logs :
{noformat}
Unable to add the application to the delegation token renewer.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:887)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:132)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:129)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.addDelegationTokens(LoadBalancingKMSClientProvider.java:129)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
..
..
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284)
at
[jira] [Updated] (HADOOP-9642) Configuration to resolve environment variables via ${env.VARIABLE} references
[
https://issues.apache.org/jira/browse/HADOOP-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kengo Seki updated HADOOP-9642:
---
Status: Patch Available (was: Open)
Configuration to resolve environment variables via ${env.VARIABLE} references
-
Key: HADOOP-9642
URL: https://issues.apache.org/jira/browse/HADOOP-9642
Project: Hadoop Common
Issue Type: Improvement
Components: conf, scripts
Affects Versions: 2.1.0-beta, 3.0.0
Reporter: Steve Loughran
Priority: Minor
Attachments: HADOOP-9642.001.patch
We should be able to get env variables from Configuration files, rather than
just system properties. I propose using the traditional {{env}} prefix
{{${env.PATH}}} to make it immediately clear to people reading a conf file
that it's an env variable -and to avoid any confusion with system properties
and existing configuration properties.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9642) Configuration to resolve environment variables via ${env.VARIABLE} references
[
https://issues.apache.org/jira/browse/HADOOP-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kengo Seki updated HADOOP-9642:
---
Attachment: HADOOP-9642.001.patch
Attaching a preliminary patch. Default values are not supported yet.
Configuration to resolve environment variables via ${env.VARIABLE} references
-
Key: HADOOP-9642
URL: https://issues.apache.org/jira/browse/HADOOP-9642
Project: Hadoop Common
Issue Type: Improvement
Components: conf, scripts
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Steve Loughran
Priority: Minor
Attachments: HADOOP-9642.001.patch
We should be able to get env variables from Configuration files, rather than
just system properties. I propose using the traditional {{env}} prefix
{{${env.PATH}}} to make it immediately clear to people reading a conf file
that it's an env variable -and to avoid any confusion with system properties
and existing configuration properties.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HADOOP-9642) Configuration to resolve environment variables via ${env.VARIABLE} references
[
https://issues.apache.org/jira/browse/HADOOP-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kengo Seki reassigned HADOOP-9642:
--
Assignee: Kengo Seki
Configuration to resolve environment variables via ${env.VARIABLE} references
-
Key: HADOOP-9642
URL: https://issues.apache.org/jira/browse/HADOOP-9642
Project: Hadoop Common
Issue Type: Improvement
Components: conf, scripts
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Steve Loughran
Assignee: Kengo Seki
Priority: Minor
Attachments: HADOOP-9642.001.patch
We should be able to get env variables from Configuration files, rather than
just system properties. I propose using the traditional {{env}} prefix
{{${env.PATH}}} to make it immediately clear to people reading a conf file
that it's an env variable -and to avoid any confusion with system properties
and existing configuration properties.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11406) xargs -P is not portable
[ https://issues.apache.org/jira/browse/HADOOP-11406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375020#comment-14375020 ] Kengo Seki commented on HADOOP-11406: - More precisely, a parallelized for loop. xargs -P is not portable Key: HADOOP-11406 URL: https://issues.apache.org/jira/browse/HADOOP-11406 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 3.0.0 Environment: Solaris Illumos AIX ... likely others Reporter: Allen Wittenauer Assignee: Brahma Reddy Battula Priority: Critical Attachments: HADOOP-11406.001.patch hadoop-functions.sh uses xargs -P in the ssh handler. -P is a GNU extension and is not available on all operating systems. We should add some detection for support and perform an appropriate action. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9642) Configuration to resolve environment variables via ${env.VARIABLE} references
[
https://issues.apache.org/jira/browse/HADOOP-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375048#comment-14375048
]
Hadoop QA commented on HADOOP-9642:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706394/HADOOP-9642.001.patch
against trunk revision 4cd54d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5983//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5983//console
This message is automatically generated.
Configuration to resolve environment variables via ${env.VARIABLE} references
-
Key: HADOOP-9642
URL: https://issues.apache.org/jira/browse/HADOOP-9642
Project: Hadoop Common
Issue Type: Improvement
Components: conf, scripts
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Steve Loughran
Assignee: Kengo Seki
Priority: Minor
Attachments: HADOOP-9642.001.patch
We should be able to get env variables from Configuration files, rather than
just system properties. I propose using the traditional {{env}} prefix
{{${env.PATH}}} to make it immediately clear to people reading a conf file
that it's an env variable -and to avoid any confusion with system properties
and existing configuration properties.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11736) KMSClientProvider addDelegationToken does not notify callers when Auth failure is due to Proxy User configuration a
[
https://issues.apache.org/jira/browse/HADOOP-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-11736:
-
Attachment: HADOOP-11736.1.patch
Fixing test cases
KMSClientProvider addDelegationToken does not notify callers when Auth
failure is due to Proxy User configuration a
Key: HADOOP-11736
URL: https://issues.apache.org/jira/browse/HADOOP-11736
Project: Hadoop Common
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Attachments: HADOOP-11736.1.patch, HDFS-7970.1.patch
When a process such as YARN RM tries to create/renew a KMS DelegationToken on
behalf of proxy user such as Llama/Impala and if the Proxy user rules are not
correctly configured to allow yarn to proxy the required user, then the
following is found in the RM logs :
{noformat}
Unable to add the application to the delegation token renewer.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:887)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:132)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:129)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.addDelegationTokens(LoadBalancingKMSClientProvider.java:129)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
..
..
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:874)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:869)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
... 21 more
{noformat}
This gives no information to the user as to why the call has failed, and
there is generally no way for an admin to know the the ProxyUser setting is
the issue without going thru the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11698) remove distcpv1 from hadoop-extras
[ https://issues.apache.org/jira/browse/HADOOP-11698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375287#comment-14375287 ] Brahma Reddy Battula commented on HADOOP-11698: --- As we are waiting for [~tlipcon] response , can we discuss offline with him and conclude..? Please let me know your opinion.. remove distcpv1 from hadoop-extras -- Key: HADOOP-11698 URL: https://issues.apache.org/jira/browse/HADOOP-11698 Project: Hadoop Common Issue Type: Improvement Components: tools/distcp Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Brahma Reddy Battula Attachments: HADOOP-11698-branch2.patch, HADOOP-11698.patch distcpv1 is pretty much unsupported. we should just remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11698) remove distcpv1 from hadoop-extras
[
https://issues.apache.org/jira/browse/HADOOP-11698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375292#comment-14375292
]
Brahma Reddy Battula commented on HADOOP-11698:
---
IMHO if users are aware of this and using {{Logalyzer}},we can keep..can we
conclude something?
remove distcpv1 from hadoop-extras
--
Key: HADOOP-11698
URL: https://issues.apache.org/jira/browse/HADOOP-11698
Project: Hadoop Common
Issue Type: Improvement
Components: tools/distcp
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Brahma Reddy Battula
Attachments: HADOOP-11698-branch2.patch, HADOOP-11698.patch
distcpv1 is pretty much unsupported. we should just remove it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11735) hadoop-nfs’ pom.xml overrides mockito version but it’s the same as parent's
[ https://issues.apache.org/jira/browse/HADOOP-11735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375386#comment-14375386 ] Tsuyoshi Ozawa commented on HADOOP-11735: - +1, committing this shortly. hadoop-nfs’ pom.xml overrides mockito version but it’s the same as parent's --- Key: HADOOP-11735 URL: https://issues.apache.org/jira/browse/HADOOP-11735 Project: Hadoop Common Issue Type: Task Components: nfs Reporter: Kengo Seki Assignee: Kengo Seki Priority: Minor Attachments: HADOOP-11735.001.patch It should be removed because only hadoop-nfs will be left behind when parent upgrades mockito. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11737) mockito's version in hadoop-nfs’ pom.xml shouldn't be specified
[ https://issues.apache.org/jira/browse/HADOOP-11737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-11737: Resolution: Fixed Fix Version/s: 2.8.0 Target Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to branch-2 and trunk. Thanks [~sekikn] for your report and contribution! mockito's version in hadoop-nfs’ pom.xml shouldn't be specified --- Key: HADOOP-11737 URL: https://issues.apache.org/jira/browse/HADOOP-11737 Project: Hadoop Common Issue Type: Improvement Components: nfs Reporter: Kengo Seki Assignee: Kengo Seki Priority: Minor Fix For: 2.8.0 Attachments: HADOOP-11735.001.patch It should be removed because only hadoop-nfs will be left behind when parent upgrades mockito. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11737) mockito's version in hadoop-nfs’ pom.xml shouldn't be specified
[ https://issues.apache.org/jira/browse/HADOOP-11737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375401#comment-14375401 ] Hudson commented on HADOOP-11737: - SUCCESS: Integrated in Hadoop-trunk-Commit #7401 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7401/]) HADOOP-11737. mockito's version in hadoop-nfsâ pom.xml shouldn't be specified. Contributed by Kengo Seki. (ozawa: rev 0b9f12c847e26103bc2304cf7114e6d103264669) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-nfs/pom.xml mockito's version in hadoop-nfs’ pom.xml shouldn't be specified --- Key: HADOOP-11737 URL: https://issues.apache.org/jira/browse/HADOOP-11737 Project: Hadoop Common Issue Type: Improvement Components: nfs Reporter: Kengo Seki Assignee: Kengo Seki Priority: Minor Fix For: 2.8.0 Attachments: HADOOP-11735.001.patch It should be removed because only hadoop-nfs will be left behind when parent upgrades mockito. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11735) mockito's version in hadoop-nfs’ pom.xml shouldn't be specified
[ https://issues.apache.org/jira/browse/HADOOP-11735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-11735: Summary: mockito's version in hadoop-nfs’ pom.xml shouldn't be specified (was: hadoop-nfs’ pom.xml overrides mockito version but it’s the same as parent's) mockito's version in hadoop-nfs’ pom.xml shouldn't be specified --- Key: HADOOP-11735 URL: https://issues.apache.org/jira/browse/HADOOP-11735 Project: Hadoop Common Issue Type: Task Components: nfs Reporter: Kengo Seki Assignee: Kengo Seki Priority: Minor Attachments: HADOOP-11735.001.patch It should be removed because only hadoop-nfs will be left behind when parent upgrades mockito. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (HADOOP-11737) mockito's version in hadoop-nfs’ pom.xml shouldn't be specified
[ https://issues.apache.org/jira/browse/HADOOP-11737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa moved HDFS-7971 to HADOOP-11737: --- Component/s: (was: nfs) nfs Key: HADOOP-11737 (was: HDFS-7971) Project: Hadoop Common (was: Hadoop HDFS) mockito's version in hadoop-nfs’ pom.xml shouldn't be specified --- Key: HADOOP-11737 URL: https://issues.apache.org/jira/browse/HADOOP-11737 Project: Hadoop Common Issue Type: Improvement Components: nfs Reporter: Kengo Seki Assignee: Kengo Seki Priority: Minor Attachments: HADOOP-11735.001.patch It should be removed because only hadoop-nfs will be left behind when parent upgrades mockito. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11014) Potential resource leak in JavaKeyStoreProvider due to unclosed stream
[
https://issues.apache.org/jira/browse/HADOOP-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375410#comment-14375410
]
Tsuyoshi Ozawa commented on HADOOP-11014:
-
The test failure looks not related.
[~cmccabe], could you take a look?
Potential resource leak in JavaKeyStoreProvider due to unclosed stream
--
Key: HADOOP-11014
URL: https://issues.apache.org/jira/browse/HADOOP-11014
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Assignee: Tsuyoshi Ozawa
Priority: Minor
Attachments: HADOOP-11014.1.patch, HADOOP-11014.2.patch,
HADOOP-11014.3.patch, HADOOP-11014.4.patch
From
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java
:
{code}
private void writeToNew(Path newPath) throws IOException {
FSDataOutputStream out =
FileSystem.create(fs, newPath, permissions);
try {
keyStore.store(out, password);
} catch (KeyStoreException e) {
throw new IOException(Can't store keystore + this, e);
} catch (NoSuchAlgorithmException e) {
throw new IOException(
No such algorithm storing keystore + this, e);
} catch (CertificateException e) {
throw new IOException(
Certificate exception storing keystore + this, e);
}
out.close();
{code}
IOException is not among the catch blocks.
According to
http://docs.oracle.com/javase/7/docs/api/java/security/KeyStore.html#store(java.io.OutputStream,%20char[]),
IOException may be thrown from the store() call. In that case, out would be
left unclosed.
In loadFromPath():
{code}
keyStore.load(fs.open(p), password);
{code}
The InputStream should be closed upon return from load()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11738) Protocol Buffers 2.5 no longer available for download
Tsuyoshi Ozawa created HADOOP-11738:
---
Summary: Protocol Buffers 2.5 no longer available for download
Key: HADOOP-11738
URL: https://issues.apache.org/jira/browse/HADOOP-11738
Project: Hadoop Common
Issue Type: Bug
Reporter: Tsuyoshi Ozawa
From REEF-216:
{quote}
Google recently switched off Google Code. They transferred the
Protocol Buffers project to
[GitHub|https://github.com/google/protobuf], and binaries are
available from [Google's developer
page|https://developers.google.com/protocol-buffers/docs/downloads].
However, only the most recent version is available. We use version 2.5
to be compatible with Hadoop. That version isn't available for
download.
{quote}
Our BUILDING.txt has same issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
