Hi I am using Spark 1.5.1 with CDH 5.4.2. My cluster is kerberos protected.
Here is pseudocode for what I am trying to do. ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI(“foo", “…") ugi.doAs( new PrivilegedExceptionAction() { val sparkConf: SparkConf = createSparkConf(…) val sparkContext = new JavaSparkContext(sparkConf) new HiveContext(sparkContext.sc) // failed }) Spark context boots up fine with UGI, but HiveContext creation failed with following message. If I manually do kinit within same shell, this code works. Any thoughts? 15/12/08 04:12:27 INFO hive.HiveContext: Initializing execution hive, version 1.2.1 15/12/08 04:12:27 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0 15/12/08 04:12:27 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0 15/12/08 04:12:27 INFO cluster.YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkexecu...@ip-10-222-0-230.us-west-2.compute.internal:48544/user/Executor#684269808]) with ID 5 15/12/08 04:12:27 INFO hive.metastore: Trying to connect to metastore with URI thrift://ip-10-222-0-145.us-west-2.compute.internal:9083 15/12/08 04:12:27 INFO hive.metastore: Connected to metastore. 15/12/08 04:12:27 INFO session.SessionState: Created local directory: /tmp/01726939-f7f4-45a4-a027-feb0ab9b0b68_resources 15/12/08 04:12:27 INFO session.SessionState: Created HDFS directory: /tmp/hive/dev.baseline/01726939-f7f4-45a4-a027-feb0ab9b0b68 15/12/08 04:12:27 INFO session.SessionState: Created local directory: /tmp/developer/01726939-f7f4-45a4-a027-feb0ab9b0b68 15/12/08 04:12:27 INFO cluster.YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkexecu...@ip-10-222-0-230.us-west-2.compute.internal:47786/user/Executor#-468362093]) with ID 9 15/12/08 04:12:27 INFO session.SessionState: Created HDFS directory: /tmp/hive/dev.baseline/01726939-f7f4-45a4-a027-feb0ab9b0b68/_tmp_space.db 15/12/08 04:12:27 INFO storage.BlockManagerMasterEndpoint: Registering block manager ip-10-222-0-230.us-west-2.compute.internal:38220 with 530.0 MB RAM, BlockManagerId(1, ip-10-222-0-230.us-west-2.compute.internal, 38220) 15/12/08 04:12:28 INFO hive.HiveContext: default warehouse location is /user/hive/warehouse 15/12/08 04:12:28 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes. 15/12/08 04:12:28 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0 15/12/08 04:12:28 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0 15/12/08 04:12:28 INFO storage.BlockManagerMasterEndpoint: Registering block manager ip-10-222-0-230.us-west-2.compute.internal:56661 with 530.0 MB RAM, BlockManagerId(5, ip-10-222-0-230.us-west-2.compute.internal, 56661) 15/12/08 04:12:28 INFO storage.BlockManagerMasterEndpoint: Registering block manager ip-10-222-0-230.us-west-2.compute.internal:43290 with 530.0 MB RAM, BlockManagerId(9, ip-10-222-0-230.us-west-2.compute.internal, 43290) 15/12/08 04:12:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/12/08 04:12:28 INFO hive.metastore: Trying to connect to metastore with URI thrift://ip-10-222-0-145.us-west-2.compute.internal:9083 15/12/08 04:12:28 ERROR transport.TSaslTransport: SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024) at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174) at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503) at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:171) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.spark.sql.hive.client.IsolatedClientLoader.liftedTree1$1(IsolatedClientLoader.scala:183) at org.apache.spark.sql.hive.client.IsolatedClientLoader.<init>(IsolatedClientLoader.scala:179) at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:226) at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:185) at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:392) at org.apache.spark.sql.SQLContext$$anonfun$5.apply(SQLContext.scala:235) at org.apache.spark.sql.SQLContext$$anonfun$5.apply(SQLContext.scala:234) at scala.collection.Iterator$class.foreach(Iterator.scala:742) at scala.collection.AbstractIterator.foreach(Iterator.scala:1194) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:234) at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:72) at com.workday.bds.dfserver.SparkServiceProvider$.com$workday$bds$dfserver$SparkServiceProvider$$createHiveContext(SparkServiceProvider.scala:86) at com.workday.bds.dfserver.SparkServiceProvider$$anon$1.run(SparkServiceProvider.scala:52) at com.workday.bds.dfserver.SparkServiceProvider$$anon$1.run(SparkServiceProvider.scala:50) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at com.workday.bds.dfserver.SparkServiceProvider$.init(SparkServiceProvider.scala:50) at com.workday.bds.dfserver.Spas$.delayedEndpoint$com$workday$bds$dfserver$Spas$1(Spas.scala:72) at com.workday.bds.dfserver.Spas$delayedInit$body.apply(Spas.scala:20) at scala.Function0$class.apply$mcV$sp(Function0.scala:34) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.App$class.main(App.scala:76) at com.workday.bds.dfserver.Spas$.main(Spas.scala:20) at com.workday.bds.dfserver.Spas.main(Spas.scala) Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 62 more 15/12/08 04:12:28 WARN hive.metastore: Failed to connect to the MetaStore Server...