Hi, As per spark documentation for spark's Dynamic Resource Allocation. I did the following to enable shuffle/ Dynamic allocation service:
A) Added the following lines to "spark-defaults.conf" enabling dynamic resource allocation and shuffle service spark.dynamicAllocation.enabled=true spark.dynamicAllocation.initialExecutors=10 spark.dynamicAllocation.minExecutors=5 spark.shuffle.service.enabled=true B) Added the following to "yarn-site.xml" <property> <name>yarn.nodemanager.aux-services</name> <value>spark_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.spark_shuffle.class</name> <value>org.apache.spark.network.yarn.YarnShuffleService</value> </property> C) removed "--num-executors" from the arguments, while running the spark job My spark version I am using to fire jobs is 1.4 while the spark yarn shuffle on the cluster is at version 1.3. After running the job I found that the jobs being run are not accepting set number of initialExecutors executor as well as uses only 3 executors while the minimum I've provided in the default config is 5. Does that mean that my Dynamic allocation service is not working correctly? What must I do to configure it correctly? Thanks! Varuni
