[
https://issues.apache.org/jira/browse/AMBARI-14377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hurley updated AMBARI-14377:
-------------------------------------
Attachment: AMBARI-14377.patch
> HiveServer start fails after enabling security post EU from 2.1 to 2.3 on
> non-HA cluster
> ----------------------------------------------------------------------------------------
>
> Key: AMBARI-14377
> URL: https://issues.apache.org/jira/browse/AMBARI-14377
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.2.0
> Reporter: Jonathan Hurley
> Assignee: Jonathan Hurley
> Priority: Critical
> Fix For: 2.2.0
>
> Attachments: AMBARI-14377.patch
>
>
> *Steps:*
> # Try EU from HDP 2.1 to 2.3.4 on an unsecure and non-HA cluster
> # Let the EU succeed
> # Enable security on the cluster
> Result:
> After enabling security HiveServer2 goes down
> Looked at the value of three properties:
> hive.cluster.delegation.token.store.zookeeper.connectString": "localhost:2181"
> hive.zookeeper.quorum": "localhost:2181"
> hive.cluster.delegation.token.store.class:org.apache.hadoop.hive.thrift.ZooKeeperTokenStore
> It appears that values of first and second property are wrongly set (compared
> this with a fresh non-HA cluster where Kerberos was enabled after install)
> Logs on HiveServer node:
> {code}
> 2015-12-14 17:41:48,495 FATAL [Thread-9]: thrift.ThriftCLIService
> (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could
> not start ThriftBinaryCLIService
> veserorg.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
> Error creating path /hive/cluster/delegationHIVESERVER2/keys
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.startDelegationTokenSecretManager(HadoopThriftAuthBridge.java:444)
> at
> org.apache.hive.service.auth.HiveAuthFactory.<init>(HiveAuthFactory.java:124)
> at
> org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:57)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.zookeeper.KeeperException$AuthFailedException:
> KeeperErrorCode = AuthFailed for /hive/cluster/delegationHIVESERVER2/keys
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:675)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:672)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
> ... 6 more
> 2015-12-14 17:41:48,497 INFO [Thread-4]: server.HiveServer2
> (HiveStringUtils.java:run(709)) - SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down HiveServer2 at
> os-r6-udmbts-baikaltom20unsecr-re-3.novalocal/172.22.76.200
> ************************************************************/
> 2015-12-14 17:41:58,398 INFO [main]: service.AbstractService
> (AbstractService.java:stop(125)) - Service:CLIService is stopped.
> 2015-12-14 17:41:58,398 INFO [main]: service.AbstractService
> (AbstractService.java:stop(125)) - Service:HiveServer2 is stopped.
> 2015-12-14 17:41:58,403 INFO [main]: server.HiveServer2
> (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance
> removed from ZooKeeper.
> 2015-12-14 17:41:58,403 INFO [Thread-7]: server.HiveServer2
> (HiveServer2.java:stop(371)) - Shutting down HiveServer2
> 2015-12-14 17:41:58,403 INFO [Thread-7]: server.HiveServer2
> (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance
> removed from ZooKeeper.
> {code}
> As it turns out, the installation wizard sets these at Hive installation
> time, even though they are only used for a Kerberized Hive. Therefore, the
> Kerberization wizard doesn't set these at all.
> When upgrading from HDP 2.1 Hive, the installation wizard hasn't set these
> since they first appeared in HDP 2.2. Therefore, they are never set. The
> right way to fix this is to have the Kerberos Wizard determine that it needs
> to calculate and set them. But that's an architectural change mostly. The
> faster fix is to make Hive consistent - just set them on upgrade from HDP 2.1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
