Jonathan Hurley created AMBARI-14377:
----------------------------------------
Summary: HiveServer start fails after enabling security post EU
from 2.1 to 2.3 on non-HA cluster
Key: AMBARI-14377
URL: https://issues.apache.org/jira/browse/AMBARI-14377
Project: Ambari
Issue Type: Bug
Components: ambari-server
Affects Versions: 2.2.0
Reporter: Jonathan Hurley
Assignee: Jonathan Hurley
Priority: Critical
Fix For: 2.2.0
*Steps:*
# Try EU from HDP 2.1 to 2.3.4 on an unsecure and non-HA cluster
# Let the EU succeed
# Enable security on the cluster
Result:
After enabling security HiveServer2 goes down
Looked at the value of three properties:
hive.cluster.delegation.token.store.zookeeper.connectString": "localhost:2181"
hive.zookeeper.quorum": "localhost:2181"
hive.cluster.delegation.token.store.class:org.apache.hadoop.hive.thrift.ZooKeeperTokenStore
It appears that values of first and second property are wrongly set (compared
this with a fresh non-HA cluster where Kerberos was enabled after install)
Logs on HiveServer node:
{code}
2015-12-14 17:41:48,495 FATAL [Thread-9]: thrift.ThriftCLIService
(ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could not
start ThriftBinaryCLIService
veserorg.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
Error creating path /hive/cluster/delegationHIVESERVER2/keys
at
org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
at
org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
at
org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.startDelegationTokenSecretManager(HadoopThriftAuthBridge.java:444)
at
org.apache.hive.service.auth.HiveAuthFactory.<init>(HiveAuthFactory.java:124)
at
org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:57)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$AuthFailedException:
KeeperErrorCode = AuthFailed for /hive/cluster/delegationHIVESERVER2/keys
at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at
org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691)
at
org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:675)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at
org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:672)
at
org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
at
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
at
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
at
org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
at
org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
at
org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
... 6 more
2015-12-14 17:41:48,497 INFO [Thread-4]: server.HiveServer2
(HiveStringUtils.java:run(709)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at
os-r6-udmbts-baikaltom20unsecr-re-3.novalocal/172.22.76.200
************************************************************/
2015-12-14 17:41:58,398 INFO [main]: service.AbstractService
(AbstractService.java:stop(125)) - Service:CLIService is stopped.
2015-12-14 17:41:58,398 INFO [main]: service.AbstractService
(AbstractService.java:stop(125)) - Service:HiveServer2 is stopped.
2015-12-14 17:41:58,403 INFO [main]: server.HiveServer2
(HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance
removed from ZooKeeper.
2015-12-14 17:41:58,403 INFO [Thread-7]: server.HiveServer2
(HiveServer2.java:stop(371)) - Shutting down HiveServer2
2015-12-14 17:41:58,403 INFO [Thread-7]: server.HiveServer2
(HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance
removed from ZooKeeper.
{code}
As it turns out, the installation wizard sets these at Hive installation time,
even though they are only used for a Kerberized Hive. Therefore, the
Kerberization wizard doesn't set these at all.
When upgrading from HDP 2.1 Hive, the installation wizard hasn't set these
since they first appeared in HDP 2.2. Therefore, they are never set. The right
way to fix this is to have the Kerberos Wizard determine that it needs to
calculate and set them. But that's an architectural change mostly. The faster
fix is to make Hive consistent - just set them on upgrade from HDP 2.1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
