Prabhu Joseph created OOZIE-2704:
------------------------------------
Summary: Oozie fails to start if default_realm does not match
Oozie prinicpal realm
Key: OOZIE-2704
URL: https://issues.apache.org/jira/browse/OOZIE-2704
Project: Oozie
Issue Type: Bug
Components: core, security
Affects Versions: 4.2.0
Environment: CentOS-6.6
Reporter: Prabhu Joseph
Priority: Critical
Problem:
Oozie fails to start with below exception when default_realm in /etc/krb5.conf
does not match with oozie principal realm. (krb5.conf managed by DC Centrify)
{code}
2016-10-06 04:10:15,991 FATAL Services:514 - SERVER[] E0100: Could not
initialize service [org.apache.oozie.service.HadoopAccessorService], Login
failure for oozie/[email protected] from keytab
/etc/security/keytabs/oozie.service.keytab:
javax.security.auth.login.LoginException: java.lang.IllegalArgumentException:
Illegal principal name oozie/[email protected]:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to oozie/[email protected]
org.apache.oozie.service.ServiceException: E0100: Could not initialize service
[org.apache.oozie.service.HadoopAccessorService], Login failure for
oozie/[email protected] from keytab
/etc/security/keytabs/oozie.service.keytab:
javax.security.auth.login.LoginException: java.lang.IllegalArgumentException:
Illegal principal name oozie/[email protected]:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to oozie/[email protected]
at
org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:209)
at
org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:136)
at
org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:107)
at
org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.setService(Services.java:372)
at org.apache.oozie.service.Services.loadServices(Services.java:305)
at org.apache.oozie.service.Services.init(Services.java:213)
at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:177)
at
org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:943)
at
org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:951)
at org.apache.oozie.tools.OozieDBCLI.createDB(OozieDBCLI.java:190)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:128)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:79)
Caused by: java.io.IOException: Login failure for
oozie/[email protected] from keytab
/etc/security/keytabs/oozie.service.keytab:
javax.security.auth.login.LoginException: java.lang.IllegalArgumentException:
Illegal principal name oozie/[email protected]:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to oozie/[email protected]
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:976)
at
org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:201)
... 12 more
Caused by: javax.security.auth.login.LoginException:
java.lang.IllegalArgumentException: Illegal principal name
oozie/[email protected]:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to oozie/[email protected]
at
org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at
javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:588)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:967)
... 13 more
Caused by: java.lang.IllegalArgumentException: Illegal principal name
oozie/[email protected]:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to oozie/[email protected]
at org.apache.hadoop.security.User.<init>(User.java:50)
at org.apache.hadoop.security.User.<init>(User.java:43)
at
org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:200)
... 25 more
Caused by:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to oozie/[email protected]
at
org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:417)
at org.apache.hadoop.security.User.<init>(User.java:48)
... 27 more
2016-10-06 04:10:15,998 INFO Services:520 - SERVER[] Shutdown
{code}
Steps to Reproduce: Oozie will fail to start if default_realm does not match
the principal realm.
cat /etc/krb5.conf
libdefaults
default_realm = CENTRIFY.COM
Oozie Prinical Name: oozie/[email protected]
hadoop.security.auth_to_local has a matching rule
"RULE:[2:$1@$0]([email protected])s/.*/oozie/" configured in core-site.xml but
still that is not honored.
Analysis:
During Oozie HadoopAccessorService#KerberosInit(), it has to get the short user
name from the prinicipal name "oozie/[email protected]". To
get a short user name, hadoop security code does the below things:
1. Default Rule: check if /etc/krb5.conf default_realm matches the prinicipal
realm, then return the service name. This will return Null on our case as
default_realm is different (DC centrify case)
2. Checks the hadoop.security.auth_to_local rules for a matching one. The rules
set will be empty always since we are creating a new Configuration object
inside HadoopAccessorService#kerberosInit(). The new Configuration properties
will be empty as oozie process classpath does not have any configuration
directory (Checked printing classpath of Oozie process). All other places
except this oozie reads configuration from the configured location by system
property -Doozie.config.dir. So this also will return Null as the rules set is
empty.
And so the error message "No rules applied to
oozie/[email protected]" is thrown by Hadoop Security code.
Solution is either we need to add configuration directory into classpath of
oozie process or new Configuration created has to be populated with the rules
from configured location like oozie.config.dir.
Adding core-site.xml into a WEB_INF/lib jar is the only workaround as of now
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
