I'm trying to get a SparkContext going in a web container which is being submitted through yarn-client. I'm trying two different approaches and both seem to be resulting in the same error from the yarn nodemanagers:
1) I'm newing up a spark context direct, manually adding all the lib jars from Spark and Hadoop to the setJars() method on the SparkConf. 2) I'm using SparkSubmit,main() to pass the classname and jar containing my code. When yarn tries to create the container, I get an exception in the driver "Yarn application already ended, might be killed or not able to launch application master". When I look into the logs for the nodemanager, I see "NoClassDefFoundError: org/apache/spark/Logging. Looking closer @ the contents of the nodemanagers, I see that the spark yarn jar was renamed to __spark__.jar and placed in the app cache while the rest of the libraries I specified via setJars() were all placed in the file cache. Any ideas as to what may be happening? I even tried adding the spark-core dependency and uber-jarring my own classes so that the dependencies would be there when Yarn tries to create the container.
