For Spark streaming, you must always set *--executor-cores* to a value which is >= 2. Or else it will not do any processing.
Thanks Best Regards On Sat, Nov 22, 2014 at 8:39 AM, pankaj channe <pankajc...@gmail.com> wrote: > I have seen similar posts on this issue but could not find solution. > Apologies if this has been discussed here before. > > I am running a spark streaming job with yarn on a 5 node cluster. I am > using following command to submit my streaming job. > > spark-submit --class class_name --master yarn-cluster --num-executors 1 > --driver-memory 1g --executor-memory 1g --executor-cores 1 my_app.jar > > > After running for some time, the job stops. The application log shows > following two errors: > > 14/11/21 22:05:04 WARN yarn.ApplicationMaster: Unable to retrieve > SparkContext in spite of waiting for 100000, maxNumTries = 10 > Exception in thread "main" java.lang.NullPointerException > at > org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkContextInitialized(ApplicationMaster.scala:218) > at > org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:107) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:410) > at > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53) > at > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) > at > org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52) > at > org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:409) > at > org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) > > > and later... > > Failed to list files for dir: > /data2/hadoop/yarn/local/usercache/user_name/appcache/application_1416332002106_0009/spark-local-20141121220325-b529/20 > at org.apache.spark.util.Utils$.listFilesSafely(Utils.scala:673) > at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:685) > at > org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:686) > at > org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:685) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34) > at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:685) > at > org.apache.spark.storage.DiskBlockManager$$anonfun$stop$1.apply(DiskBlockManager.scala:181) > at > org.apache.spark.storage.DiskBlockManager$$anonfun$stop$1.apply(DiskBlockManager.scala:178) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > at > org.apache.spark.storage.DiskBlockManager.stop(DiskBlockManager.scala:178) > at > org.apache.spark.storage.DiskBlockManager$$anon$1$$anonfun$run$1.apply$mcV$sp(DiskBlockManager.scala:171) > at > org.apache.spark.storage.DiskBlockManager$$anon$1$$anonfun$run$1.apply(DiskBlockManager.scala:169) > at > org.apache.spark.storage.DiskBlockManager$$anon$1$$anonfun$run$1.apply(DiskBlockManager.scala:169) > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1311) > at > org.apache.spark.storage.DiskBlockManager$$anon$1.run(DiskBlockManager.scala:169) > > > Note: I am building my jar on my local with spark dependency added in > pom.xml and running it on cluster running spark. > > > -Pankaj >
