Hi, I am using Spark on Amazon EMR. So far I have not succeeded to submit the application successfully, not sure what's problem. In the log file I see the followings. java.io.FileNotFoundException: File does not exist: hdfs://ipx-x-x-x:8020/user/hadoop/.sparkStaging/application_123344567_0018/spark-assembly-1.4.1-hadoop2.6.0-amzn-0.jar
However, even putting spark-assembly-1.4.1-hadoop2.6.0-amzn-0.jar in the fat jar file didn't solve the problem. I am out of clue now. I want to submit a spark application, using aws web console, as a step. I submit the application as : spark-submit --deploy-mode cluster --class mypack.MyMainClass --master yarn-cluster s3://mybucket/MySparkApp.jar Is there any one who has similar problem with EMR? best, /Shahab
