Did you go through http://spark.apache.org/docs/latest/job-scheduling.html#configuration-and-setup for yarn, i guess you will have to copy the spark-1.5.1-yarn-shuffle.jar to the classpath of all nodemanagers in your cluster.
Thanks Best Regards On Fri, Oct 30, 2015 at 7:41 PM, Tom Stewart < stewartthom...@yahoo.com.invalid> wrote: > I am running the following command on a Hadoop cluster to launch Spark > shell with DRA: > spark-shell --conf spark.dynamicAllocation.enabled=true --conf > spark.shuffle.service.enabled=true --conf > spark.dynamicAllocation.minExecutors=4 --conf > spark.dynamicAllocation.maxExecutors=12 --conf > spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=120 --conf > spark.dynamicAllocation.schedulerBacklogTimeout=300 --conf > spark.dynamicAllocation.executorIdleTimeout=60 --executor-memory 512m > --master yarn-client --queue default > > This is the code I'm running within the Spark Shell - just demo stuff from > teh web site. > > import org.apache.spark.mllib.clustering.KMeans > import org.apache.spark.mllib.linalg.Vectors > > // Load and parse the data > val data = sc.textFile("hdfs://ns/public/sample/kmeans_data.txt") > > val parsedData = data.map(s => Vectors.dense(s.split(' > ').map(_.toDouble))).cache() > > // Cluster the data into two classes using KMeans > val numClusters = 2 > val numIterations = 20 > val clusters = KMeans.train(parsedData, numClusters, numIterations) > > This works fine on Spark 1.4.1 but is failing on Spark 1.5.1. Did > something change that I need to do differently for DRA on 1.5.1? > > This is the error I am getting: > 15/10/29 21:44:19 WARN YarnScheduler: Initial job has not accepted any > resources; check your cluster UI to ensure that workers are registered and > have sufficient resources > 15/10/29 21:44:34 WARN YarnScheduler: Initial job has not accepted any > resources; check your cluster UI to ensure that workers are registered and > have sufficient resources > 15/10/29 21:44:49 WARN YarnScheduler: Initial job has not accepted any > resources; check your cluster UI to ensure that workers are registered and > have sufficient resources > > That happens to be the same error you get if you haven't followed the > steps to enable DRA, however I have done those and as I said if I just flip > to Spark 1.4.1 on the same cluster it works with my YARN config. > >