[
https://issues.apache.org/jira/browse/AMBARI-8785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated AMBARI-8785:
--------------------------------
Description:
When enabling Yarn Resource Manager HA in a Kerberos secured cluster, Ambari
fails to tell the user about the required Kerberos principal + keytab for the
new Resource Manager instance.
As as result the new Resource Manager fails to start with the following logs:
{code}2014-12-18 11:39:06,379 FATAL resourcemanager.ResourceManager
(ResourceManager.java:main(1043)) - Error starting ResourceManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to login
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:910)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1041)
Caused by: java.io.IOException: Login failure for rm/<host>@REALM from keytab
/etc/security/keytabs/rm.service.keytab
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:920)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:242)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.doSecureLogin(ResourceManager.java:929)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:908)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password
from user
at
com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
at
com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
at
com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at
javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:911)
... 5 more
2014-12-18 11:39:06,383 INFO resourcemanager.ResourceManager
(StringUtils.java:run(640)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at <host>/x.x.x.x
************************************************************/
{code}
The fix is quite simple for experienced cluster administrators, create the
principal, export the keytab, mimic the original RM permissions of yarn:hadoop
620 (not really correct octal but that's a separate issue I've already raised
to fix keytab perms AMBARI-8138) and then it will start up.
This is sort of related to AMBARI-8610 where Ambari should export a CSV for new
hosts/services to allow existing automation scripts to generate the principals
and and distributed the keytabs such as the basic kerberos_setup.sh shipped
with Ambari or the more real world FreeIPA one I publish on my github
(https://github.com/harisekhon/toolbox).
Except in this instance it's not a new service, it's just HA enablement where
it doesn't notify users that RM HA will require a new Kerberos principal +
keytab to allow them to do that, never mind provide a subset CSV for passing to
a kerberos setup script. This issue will also affect NN HA if Ambari ever
allows setting up NN HA without disabling Kerberos first.
Regards,
Hari Sekhon
http://www.linkedin.com/in/harisekhon
was:
When enabling Yarn Resource Manager HA in a Kerberos secured cluster, Ambari
fails to tell the user about the required Kerberos principal + keytab for the
new Resource Manager instance.
As as result the new Resource Manager fails to start with the following logs:
{code}2014-12-18 11:39:06,379 FATAL resourcemanager.ResourceManager
(ResourceManager.java:main(1043)) - Error starting ResourceManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to login
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:910)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1041)
Caused by: java.io.IOException: Login failure for rm/<host>@REALM from keytab
/etc/security/keytabs/rm.service.keytab
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:920)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:242)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.doSecureLogin(ResourceManager.java:929)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:908)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password
from user
at
com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
at
com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
at
com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at
javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:911)
... 5 more
2014-12-18 11:39:06,383 INFO resourcemanager.ResourceManager
(StringUtils.java:run(640)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at <host>/x.x.x.x
************************************************************/
{code}
The fix is quite simple for experienced cluster administrators, create the
principal, export the keytab, mimic the original RM permissions of yarn:hadoop
620 (not really correct octal but that's a separate issue I've already raised
to fix keytab perms AMBARI-8138) and then it will start up.
This is sort of related to AMBARI-8610 where Ambari should export a CSV for new
hosts/services to allow existing automation scripts to generate the principals
and and distributed the keytabs such as the basic kerberos_setup.sh shipped
with Ambari or the more real world FreeIPA one I publish on my github
(https://github.com/harisekhon/toolbox).
Except in this instance it's not a new service, it's just HA enablement where
it doesn't notify users that RM HA will require a new Keberos principal +
keytab to allow them to do that, never mind provide a subset CSV for passing to
a kerberos setup script. This issue will also affect NN HA if Ambari ever
allows setting up NN HA without disabling Kerberos first.
Regards,
Hari Sekhon
http://www.linkedin.com/in/harisekhon
> Resource Manager HA Kerberos principal not handled and user not notified of
> requirement
> ---------------------------------------------------------------------------------------
>
> Key: AMBARI-8785
> URL: https://issues.apache.org/jira/browse/AMBARI-8785
> Project: Ambari
> Issue Type: Bug
> Affects Versions: 1.7.0
> Environment: HDP 2.1
> Reporter: Hari Sekhon
>
> When enabling Yarn Resource Manager HA in a Kerberos secured cluster, Ambari
> fails to tell the user about the required Kerberos principal + keytab for the
> new Resource Manager instance.
> As as result the new Resource Manager fails to start with the following logs:
> {code}2014-12-18 11:39:06,379 FATAL resourcemanager.ResourceManager
> (ResourceManager.java:main(1043)) - Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to login
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:910)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1041)
> Caused by: java.io.IOException: Login failure for rm/<host>@REALM from keytab
> /etc/security/keytabs/rm.service.keytab
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:920)
> at
> org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:242)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.doSecureLogin(ResourceManager.java:929)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:908)
> ... 2 more
> Caused by: javax.security.auth.login.LoginException: Unable to obtain
> password from user
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
> at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:911)
> ... 5 more
> 2014-12-18 11:39:06,383 INFO resourcemanager.ResourceManager
> (StringUtils.java:run(640)) - SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down ResourceManager at <host>/x.x.x.x
> ************************************************************/
> {code}
> The fix is quite simple for experienced cluster administrators, create the
> principal, export the keytab, mimic the original RM permissions of
> yarn:hadoop 620 (not really correct octal but that's a separate issue I've
> already raised to fix keytab perms AMBARI-8138) and then it will start up.
> This is sort of related to AMBARI-8610 where Ambari should export a CSV for
> new hosts/services to allow existing automation scripts to generate the
> principals and and distributed the keytabs such as the basic
> kerberos_setup.sh shipped with Ambari or the more real world FreeIPA one I
> publish on my github (https://github.com/harisekhon/toolbox).
> Except in this instance it's not a new service, it's just HA enablement where
> it doesn't notify users that RM HA will require a new Kerberos principal +
> keytab to allow them to do that, never mind provide a subset CSV for passing
> to a kerberos setup script. This issue will also affect NN HA if Ambari ever
> allows setting up NN HA without disabling Kerberos first.
> Regards,
> Hari Sekhon
> http://www.linkedin.com/in/harisekhon
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
