Hi, I'm trying to run the SparkPi example using `--deploy-mode cluster`. Not having much luck. Looks like the JAVA_HOME on my Mac is used for launching the app on the Spark cluster running on CentOS. How can I set the JAVA_HOME or better yet, how can I configure Spark to use the JAVA_HOME configured for the cluster?
Here is the sysout from launching the job: $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode cluster --master spark://borneo:7077 hdfs://borneo:8020/xd/spark/spark-examples_2.10-1.2.0-SNAPSHOT.jar 100 Spark assembly has been built with Hive, including Datanucleus jars on classpath Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/10/27 10:19:13 INFO SecurityManager: Changing view acls to: trisberg, 14/10/27 10:19:13 INFO SecurityManager: Changing modify acls to: trisberg, 14/10/27 10:19:13 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(trisberg, ); users with modify permissions: Set(trisberg, ) 14/10/27 10:19:13 INFO Slf4jLogger: Slf4jLogger started 14/10/27 10:19:14 INFO Utils: Successfully started service 'driverClient' on port 49999. Sending launch command to spark://borneo:7077 Driver successfully submitted as driver-20141027091913-0007 ... waiting before polling master for driver state ... polling master for driver state State of driver-20141027091913-0007 is ERROR Exception from cluster was: java.io.IOException: Cannot run program "/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/bin/java" (in directory "/home/hadoop/spark-1.1.0-bin-hadoop2.4/work/driver-20141027091913-0007"): error=2, No such file or directory java.io.IOException: Cannot run program "/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/bin/java" (in directory "/home/hadoop/spark-1.1.0-bin-hadoop2.4/work/driver-20141027091913-0007"): error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047) at org.apache.spark.deploy.worker.ProcessBuilderLike$$anon$4.start(DriverRunner.scala:238) at org.apache.spark.deploy.worker.DriverRunner.runCommandWithRetry(DriverRunner.scala:200) at org.apache.spark.deploy.worker.DriverRunner.org $apache$spark$deploy$worker$DriverRunner$$launchDriver(DriverRunner.scala:183) at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:88) Caused by: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:186) at java.lang.ProcessImpl.start(ProcessImpl.java:130) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028) ... 4 more Thanks, Thomas
