Thomas Larsson created OOZIE-2809:
-------------------------------------
Summary: Oozie HA with secure zookeeper access fails because
ZKUtils#setJaasConfiguration does not replace "_HOST" in principal string
Key: OOZIE-2809
URL: https://issues.apache.org/jira/browse/OOZIE-2809
Project: Oozie
Issue Type: Bug
Components: core
Affects Versions: 4.3.0, 4.2.0, trunk
Reporter: Thomas Larsson
Setting "oozie.zookeeper.secure" = true causes the oozie server not to start if
"oozie.service.HadoopAccessorService.kerberos.principal" is defined with a
"_HOST" token instead of an actual hostname.
The symptom when this happens is that the oozie log shows something like this:
{noformat}
org.apache.oozie.service.ServiceException: E1700: Issue communicating with
ZooKeeper: KeeperErrorCode = NoAuth for /oozie/services/servers
at org.apache.oozie.service.ZKLocksService.init(ZKLocksService.java:76)
at
org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.setService(Services.java:372)
at org.apache.oozie.service.Services.loadServices(Services.java:305)
at org.apache.oozie.service.Services.init(Services.java:213)
at
org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:676)
at
org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:602)
at
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:503)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060)
at
org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at
org.apache.catalina.core.StandardService.start(StandardService.java:525)
at
org.apache.catalina.core.StandardServer.start(StandardServer.java:759)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: org.apache.zookeeper.KeeperException$NoAuthException:
KeeperErrorCode = NoAuth for /oozie/services/servers
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:199)
at
org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:682)
at
org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at
org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656)
at
org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441)
at
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431)
at
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at
org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalRegisterService(ServiceDiscoveryImpl.java:176)
at
org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.registerService(ServiceDiscoveryImpl.java:150)
at org.apache.oozie.util.ZKUtils.advertiseService(ZKUtils.java:217)
at org.apache.oozie.util.ZKUtils.<init>(ZKUtils.java:141)
at org.apache.oozie.util.ZKUtils.register(ZKUtils.java:154)
at org.apache.oozie.service.ZKLocksService.init(ZKLocksService.java:70)
... 29 more
2017-02-24 14:03:23,683 INFO Services:520 -
SERVER[datavault-dev-app1.internal.machines] Shutdown
{noformat}
This seems to come from the class
[org.apache.oozie.util.ZKUtils|https://git-wip-us.apache.org/repos/asf?p=oozie.git;a=blob;f=core/src/main/java/org/apache/oozie/util/ZKUtils.java;h=5835fb2bd7460809743d21ac16f2992c353ada71;hb=HEAD#l357]
{code}
private void setJaasConfiguration() throws ServiceException, IOException {
String keytabFile = Services.get().getConf().get(KERBEROS_KEYTAB,
System.getProperty("user.home") + "/oozie.keytab").trim();
if (keytabFile.length() == 0) {
throw new ServiceException(ErrorCode.E0026, KERBEROS_KEYTAB);
}
String principal = Services.get().getConf().get(KERBEROS_PRINCIPAL,
"oozie/localhost@LOCALHOST");
if (principal.length() == 0) {
throw new ServiceException(ErrorCode.E0026, KERBEROS_PRINCIPAL);
}
// This is equivalent to writing a jaas.conf file and setting the
system property, "java.security.auth.login.config", to
// point to it (but this way we don't have to write a file, and it
works better for the tests)
JaasConfiguration.addEntry("Client", principal, keytabFile);
Configuration.setConfiguration(JaasConfiguration.getInstance());
}
{code}
Compare how the principal is set without any string replacement with how it is
done in class
[org.apache.oozie.service.HadoopAccessorService|https://git-wip-us.apache.org/repos/asf?p=oozie.git;a=blob;f=core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java;h=5377127736cf189bcf4aa3d62bbbeab6b3a29d52;hb=HEAD#l226]
{code}
private void kerberosInit(Configuration serviceConf) throws
ServiceException {
try {
String keytabFile = ConfigurationService.get(serviceConf,
KERBEROS_KEYTAB).trim();
if (keytabFile.length() == 0) {
throw new ServiceException(ErrorCode.E0026,
KERBEROS_KEYTAB);
}
String principal = SecurityUtil.getServerPrincipal(
serviceConf.get(KERBEROS_PRINCIPAL,
"oozie/localhost@LOCALHOST"),
InetAddress.getLocalHost().getCanonicalHostName());
if (principal.length() == 0) {
throw new ServiceException(ErrorCode.E0026,
KERBEROS_PRINCIPAL);
}
Configuration conf = new Configuration();
conf.set("hadoop.security.authentication", "kerberos");
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab(principal, keytabFile);
LOG.info("Got Kerberos ticket, keytab [{0}], Oozie principal
principal [{1}]",
keytabFile, principal);
}
catch (ServiceException ex) {
throw ex;
}
catch (Exception ex) {
throw new ServiceException(ErrorCode.E0100,
getClass().getName(), ex.getMessage(), ex);
}
}
{code}
In the latter case, the SecurityUtil class is used to replace "_HOST"
occurrences with an actual hostname.
I guess something similar should be done in the ZKUtils class as well.
Best Regards
/Thomas
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
