Liu Shaohui created KYLIN-3726:
----------------------------------
Summary: KylinSession should load spark properties from
spark-defaults.conf
Key: KYLIN-3726
URL: https://issues.apache.org/jira/browse/KYLIN-3726
Project: Kylin
Issue Type: Sub-task
Components: Storage - Parquet
Reporter: Liu Shaohui
When testing parquet storage, the spark session job failed to be submit for no
JAVA_HOME in executor env. This config is set in the spark default property
file: spark-defaults.conf.
{code}
2018-12-18,15:13:15,466 ERROR org.apache.spark.deploy.yarn.YarnAllocator:
Failed to launch executor 6 on container
container_e823_1541646991414_1025309_01_000007
java.util.NoSuchElementException: key not found: JAVA_HOME at
scala.collection.MapLike$class.default(MapLike.scala:228) at
scala.collection.AbstractMap.default(Map.scala:59) at
scala.collection.mutable.HashMap.apply(HashMap.scala:65) at
org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3$$anonfun$apply$3.apply(ExecutorRunnable.scala:286)
at
org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3$$anonfun$apply$3.apply(ExecutorRunnable.scala:275)
at scala.Option.foreach(Option.scala:257) at
org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3.apply(ExecutorRunnable.scala:275)
at
org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3.apply(ExecutorRunnable.scala:274)
at scala.Option.foreach(Option.scala:257) at
org.apache.spark.deploy.yarn.ExecutorRunnable.prepareEnvironment(ExecutorRunnable.scala:274)
at
org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:92)
at
org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:69) at
org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$runAllocatedContainers$1$$anon$1.run(YarnAllocator.scala:556)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
