Re: Error submitting Spark Job in yarn-cluster mode on EMR

2018-05-08 Thread Marco Mistroni
Did you by any chances left a sparkSession.setMaster("local") lurking in your code? Last time i checked, to run on yarn you have to package a 'fat jar'. could you make sure the spark depedencies in your jar matches the version you are running on Yarn? alternatively please share code including h

Error submitting Spark Job in yarn-cluster mode on EMR

2018-05-08 Thread SparkUser6
I have a simple program that works fine in the local mode. But I am having issues when I try to run the program in yarn-cluster mode. I know usually no such method happens when compile and run version mismatch but I made sure I took the same version. 205 [main] INFO org.spark_project.jetty.ser