Hi all
I see UGI credentials (ex sparkCookie) shared from driver to executor is being
lost on driver side in yarn mode. Below is the analysis on start of
thriftserver,
Step 1. SparkSubmit create submit env which does a loginUserFromKeytab
"main@1" prio=5 tid=0x1 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1041)
- locked <0x582> (a java.lang.Class)
at
org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:336)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:156)
at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:122)
at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala:-1)
Step 2. HiveThriftServer does SparkSQLEnv.init which will Create SparkContext,
and hence calls createDriverEnv which will generate secret key and add to UGI
by Step 1 as credentials
"main@1" prio=5 tid=0x1 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at
org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:429)
at
org.apache.spark.SecurityManager.<init>(SecurityManager.scala:228)
at
org.apache.spark.SparkEnv$.create(SparkEnv.scala:237)
at
org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
at
org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:258)
at
org.apache.spark.SparkContext.<init>(SparkContext.scala:433)
at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2521)
- locked <0x10a5> (a java.lang.Object)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:923)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:915)
at scala.Option.getOrElse(Option.scala:121)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:915)
- locked <0x1091> (a
org.apache.spark.sql.SparkSession$Builder)
- locked <0x10a6> (a
org.apache.spark.sql.SparkSession$)
at
org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:48)
at
org.apache.spark.sql.hive.thriftserver.a$.main(HiveThriftServer2.scala:86)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:798)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:183)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:208)
at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:122)
at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala:-1)
Step 3. Next Application is submitted which will create container launch
context using UGI passed from Step 2
"main@1" prio=5 tid=0x1 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at
org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:429)
at
org.apache.spark.SecurityManager.<init>(SecurityManager.scala:228)
at
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:999)
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:194)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)
at
org.apache.spark.SparkContext.<init>(SparkContext.scala:510)
at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2521)
- locked <0x10a5> (a java.lang.Object)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:923)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:915)
at scala.Option.getOrElse(Option.scala:121)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:915)
- locked <0x1091> (a
org.apache.spark.sql.SparkSession$Builder)
- locked <0x10a6> (a
org.apache.spark.sql.SparkSession$)
at
org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:48)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:86)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:798)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:183)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:208)
at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:122)
at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala:-1)
Step 4. Lastly initCompositeService will invoke SparkSQLCLIService which does a
loginUserFromKeytab again which will override UGI created by Step 1
"main@1" prio=5 tid=0x1 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1041)
- locked <0x582> (a java.lang.Class)
at
org.apache.hive.service.auth.HiveAuthFactory.loginFromKeytab(HiveAuthFactory.java:199)
at
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIService.init(SparkSQLCLIService.scala:53)
at
org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$$anonfun$initCompositeService$1.apply(SparkSQLCLIService.scala:79)
at
org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$$anonfun$initCompositeService$1.apply(SparkSQLCLIService.scala:79)
at
scala.collection.Iterator$class.foreach(Iterator.scala:893)
at
scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at
scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$class.initCompositeService(SparkSQLCLIService.scala:79)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.initCompositeService(HiveThriftServer2.scala:283)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.init(HiveThriftServer2.scala:303)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:99)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:798)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:183)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:208)
at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:122)
at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala:-1)
Is this the right behavior? as it looks like UGI created on JDBCServer startup
which is passed to containerLaunchContext becomes stale.? please correct me if
i am wrong
Regards
Ajith
