After upgrading to HDP 2.5.3.x we are no longer able to correctly use the HiveCLI with Atlas Hive Hook. Some assumptions specifically that the only access method is HiveServer2.
We need the ability to split option in Ambari so HiveCLI AtlasHook uses the following options:
atlas.jaas.KafkaClient.loginModuleControlFlag=required
atlas.jaas.KafkaClient.loginModuleName=com.sun.security.auth.module.Krb5LoginModule
atlas.jaas.KafkaClient.option.serviceName=kafka
atlas.jaas.KafkaClient.option.renewTicket=True
atlas.jaas.KafkaClient.option.useTicketCache=True
and HiveServer2 using:
atlas.jaas.KafkaClient.loginModuleControlFlag=required
atlas.jaas.KafkaClient.loginModuleName=com.sun.security.auth.module.Krb5LoginModule
atlas.jaas.KafkaClient.option.keyTab=/etc/security/keytabs/hive.service.keytab
atlas.jaas.KafkaClient.option.principal=hive/[email protected]
atlas.jaas.KafkaClient.option.serviceName=kafka
atlas.jaas.KafkaClient.option.storeKey=True
atlas.jaas.KafkaClient.option.useKeyTab=True
If this is not done HiveCLI will fail to post to Kafka:
ve/warehouse/nyse_stocks_test"}}}}}, endTime=Mon Jan 30 11:42:38 EST 2017}}]] after 3 retries. Quitting
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:188)
at org.apache.atlas.kafka.KafkaNotification.createProducer(KafkaNotification.java:311)
at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:220)
at org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:84)
at org.apache.atlas.hook.AtlasHook.notifyEntitiesInternal(AtlasHook.java:129)
at org.apache.atlas.hook.AtlasHook.notifyEntities(AtlasHook.java:114)
at org.apache.atlas.hook.AtlasHook.notifyEntities(AtlasHook.java:167)
at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:282)
at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:82)
at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:193)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:71)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:83)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:277)
After adjusting atlas-application.properties:
-b558-4b02d509d888
2017-01-30 23:13:41,053 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=PostHook.org.apache.atlas.hive.hook.HiveHook from=org.apache.hadoop.hive.ql.Driver>
2017-01-30 23:13:41,062 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=PostHook.org.apache.atlas.hive.hook.HiveHook start=1485836021053 end=1485836021062 duration=9 from=org.apache.hadoop.hive.ql.Driver>
2017-01-30 23:13:41,062 INFO [Atlas Logger 1]: hook.HiveHook (HiveHook.java:fireAndForget(209)) - Entered Atlas hook for hook type POST_EXEC_HOOK operation CREATETABLE_AS_SELECT
2017-01-30 23:13:41,062 INFO [main]: ql.Driver (Driver.java:execute(1635)) - Resetting the caller context to
|
