Hello, This week I have been testing 1.6 (#d509194b) in our HDP 2.3 platform and its been working pretty ok, at the exception of the YARN cluster deployment mode. Note that with 1.5 using the same "spark-props.conf" and "spark-env.sh" config files the cluster mode works as expected.
Has anyone else also tried the cluster mode in 1.6? Problem reproduction: ---- # spark-submit --master yarn --deploy-mode cluster --num-executors 1 --properties-file $PWD/spark-props.conf --class org.apache.spark.examples.SparkPi /opt/spark/lib/spark-examples-1.6.0-SNAPSHOT-hadoop2.7.1.jar Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster spark-props.conf ----------------------------- spark.driver.extraJavaOptions -Dhdp.version=2.3.2.0-2950 spark.driver.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64 spark.executor.extraJavaOptions -Dhdp.version=2.3.2.0-2950 spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64 ----------------------------- I will try to do some more debugging on this issue. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-6-YARN-Cluster-Mode-tp25729.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
