I am trying to submit a simple SparkPi application from a windows machine which has spark 1.0.2 to a hadoop 2.3.0 cluster running on Linux. SparkPi application can be launched and executed successfully when running on the Linux machine, however, I get the following error when I launch from Windows.
*On Windows:* spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode client --master yarn c:\Users \windows_user\spark-1.0.2\examples\target\scala-2.10\spark-examples-1.0.2-hadoop2.3.0.jar *Exception on Windows:* Exception in thread "main" org.apache.spark.SparkException: Yarn application already ended,might be killed or not able to launch application master. *On Linux Hadoop Cluster:* /opt/hadoop/hadoop/logs/userlogs/application_1411473500741_0002/container_1411473500741_0002_01_000001/stderr Error: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher * Resource Manager Logs* /2014-09-23 12:28:21,479 DEBUG org.apache.hadoop.yarn.event.AsyncDispatcher: Dispatching the event org.apache.hadoop.yarn.server.resourcemanager.RMAppManagerEvent.EventType: APP_COMPLETED 2014-09-23 12:28:21,479 DEBUG org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: RMAppManager processing event for application_1411473500741_0002 of type APP_COMPLETED 2014-09-23 12:28:21,479 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=spark OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1411473500741_0002 failed 2 times due to AM Container for appattempt_1411473500741_0002_000002 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:511) at org.apache.hadoop.util.Shell.run(Shell.java:424) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 .Failing this attempt.. Failing the application. APPID=application_1411473500741_0002 2014-09-23 12:28:21,479 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1411473500741_0002,name=Spark Pi,user=spark,queue=root.spark,state=FAILED,trackingUrl=linux_machine1:50088/cluster/app/application_1411473500741_0002,appMasterHost=N/A,startTime=1411475289940,finishTime=1411475301443,finalStatus=FAILED 2014-09-23 12:28:21,513 DEBUG org.apache.hadoop.ipc.Server: IPC Server idle connection scanner for port 8030: task running 2014-09-23 12:28:21,646 DEBUG org.apache.hadoop.ipc.Server: IPC Server idle connection scanner for port 8032: task running 2014-09-23 12:28:21,723 DEBUG org.apache.hadoop.ipc.Server: got #1778/ The spark assembly as well as the application jar spark-examples-1.0.2-hadoop2.3.0.jar seems to get uploaded successfully to hadoop hdfs, but the application launcher cannot find the spark jars. For troubleshooting purposes, I even copied spark-yarn_2.10-1.0.2.jar (which contains ExecutorLauncher class) into share lib folders of Hadoop cluster without luck. Any help to resolve this issue would be highly appreciated. Also, I was able to grab the shell command that is executed by the launcher (/*launch_container.sh*/) / export HADOOP_TOKEN_FILE_LOCATION="/tmp/hadoop-hadoop/nm-local-dir/usercache/spark/appcache/application_1411467581902_0007/container_1411467581902_0007_01_000001/container_tokens" export CLASSPATH="$PWD/__spark__.jar;$HADOOP_CONF_DIR;$HADOOP_COMMON_HOME/share/hadoop/common/*;$HADOOP_COMMON_HOME/share/hadoop/common/lib/*;$HADOOP_HDFS_HOME/share/hadoop/hdfs/*;$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*;$HADOOP_YARN_HOME/share/hadoop/yarn/*;$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*>;%HADOOP_MAPRED_HOME%\share\hadoop\mapreduce\*;%HADOOP_MAPRED_HOME%\share\hadoop\mapreduce\lib\*;$PWD/__app__.jar;$PWD/;$PWD;$PWD/*" export USER="spark" export HADOOP_HDFS_HOME="/opt/hadoop/hadoop-2.3.0-cdh5.1.2" export CONTAINER_ID="container_1411467581902_0007_01_000001" export HOME="/home/" export HADOOP_CONF_DIR="/opt/hadoop/hadoop/etc/hadoop" ln -sf "/tmp/hadoop-hadoop/nm-local-dir/usercache/spark/filecache/14/spark-assembly-1.0.2-hadoop2.3.0-cdh5.1.0.jar" "__spark__.jar" ln -sf "/tmp/hadoop-hadoop/nm-local-dir/filecache/10/spark-examples-1.0.2-hadoop2.2.0.jar" "spark-examples-1.0.2-hadoop2.2.0.jar" exec /bin/bash -c "$JAVA_HOME/bin/java -server -Xmx512m -Djava.io.tmpdir=$PWD/tmp -Dspark.tachyonStore.folderName=\"spark-0497b165-b1d5-48ff-8a06-c9145d68f3a6\" -Dspark.yarn.dist.files=\"hdfs://linux_machine1:9000/user/spark/share/lib/spark-examples-1.0.2-hadoop2.2.0.jar\" -Dspark.yarn.secondary.jars=\"\" -Dspark.driver.host=\"WS_1560\" -Dspark.app.name=\"Spark Pi\" -Dspark.jars=\"file:/c:/Users/windows_user/spark-1.0.2/spark-1.0.2/examples/target/scala-2.10/spark-examples-1.0.2-hadoop2.3.0.jar\" -Dspark.fileserver.uri=\"http://10.138.62.137:58756\" -Dspark.master=\"yarn-client\" -Dspark.driver.port=\"58751\" -Dspark.httpBroadcast.uri=\"http://10.138.62.137:58755\" -Dlog4j.configuration=log4j-spark-container.properties org.apache.spark.deploy.yarn.ExecutorLauncher --class notused --jar null --args 'WS_1560:58751' --executor-memory 1024 --executor-cores 1 --num-executors 2 1> /opt/hadoop/hadoop-2.3.0-cdh5.1.2/logs/userlogs/application_1411467581902_0007/container_1411467581902_0007_01_000001/stdout 2> /opt/hadoop/hadoop-2.3.0-cdh5.1.2/logs/userlogs/application_1411467581902_0007/container_1411467581902_0007_01_000001/stderr"/ -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Error-launching-spark-application-from-Windows-to-Linux-YARN-Cluster-Could-not-find-or-load-main-clar-tp14888.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org