Did you go through
http://spark.apache.org/docs/latest/job-scheduling.html#configuration-and-setup
for yarn, i guess you will have to copy the spark-1.5.1-yarn-shuffle.jar to
the classpath of all nodemanagers in your cluster.
Thanks
Best Regards
On Fri, Oct 30, 2015 at 7:41 PM, Tom Stewart <
stewartthom...@yahoo.com.invalid> wrote:
> I am running the following command on a Hadoop cluster to launch Spark
> shell with DRA:
> spark-shell --conf spark.dynamicAllocation.enabled=true --conf
> spark.shuffle.service.enabled=true --conf
> spark.dynamicAllocation.minExecutors=4 --conf
> spark.dynamicAllocation.maxExecutors=12 --conf
> spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=120 --conf
> spark.dynamicAllocation.schedulerBacklogTimeout=300 --conf
> spark.dynamicAllocation.executorIdleTimeout=60 --executor-memory 512m
> --master yarn-client --queue default
>
> This is the code I'm running within the Spark Shell - just demo stuff from
> teh web site.
>
> import org.apache.spark.mllib.clustering.KMeans
> import org.apache.spark.mllib.linalg.Vectors
>
> // Load and parse the data
> val data = sc.textFile("hdfs://ns/public/sample/kmeans_data.txt")
>
> val parsedData = data.map(s => Vectors.dense(s.split('
> ').map(_.toDouble))).cache()
>
> // Cluster the data into two classes using KMeans
> val numClusters = 2
> val numIterations = 20
> val clusters = KMeans.train(parsedData, numClusters, numIterations)
>
> This works fine on Spark 1.4.1 but is failing on Spark 1.5.1. Did
> something change that I need to do differently for DRA on 1.5.1?
>
> This is the error I am getting:
> 15/10/29 21:44:19 WARN YarnScheduler: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient resources
> 15/10/29 21:44:34 WARN YarnScheduler: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient resources
> 15/10/29 21:44:49 WARN YarnScheduler: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient resources
>
> That happens to be the same error you get if you haven't followed the
> steps to enable DRA, however I have done those and as I said if I just flip
> to Spark 1.4.1 on the same cluster it works with my YARN config.
>
>
