Re: Running Hive and Spark together with Dynamic Resource Allocation
It seems like the best solution is to set: yarn.nodemanager.aux-services to mapred_shuffle,spark_shuffle -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Hive-and-Spark-together-with-Dynamic-Resource-Allocation-tp27968p27978.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Running Hive and Spark together with Dynamic Resource Allocation
This works for us yarn.nodemanager.aux-services mapreduce_shuffle,spark_shuffle yarn.nodemanager.aux-services.mapreduce_shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.nodemanager.aux-services.spark_shuffle.class org.apache.spark.network.yarn.YarnShuffleService On Thu, Oct 27, 2016 at 8:13 AM, rachmaninovquartet < rachmaninovquar...@gmail.com> wrote: > oblem is that for Spark 1.5.2 with dynamic resource allocation to > function properly we needed to set y >
Running Hive and Spark together with Dynamic Resource Allocation
Hi, My team has a cluster running HDP, with Hive and Spark. We setup spark to use dynamic resource allocation, for benefits such as not having to hard code the number of executors and to free resources after using. Everything is running on YARN. The problem is that for Spark 1.5.2 with dynamic resource allocation to function properly we needed to set yarn.nodemanager.aux-services in yarn-site.xml to spark_shuffle, but this breaks hive (1.2.1), since it is looking for auxService:mapreduce_shuffle. Does any one know of a way to configure in order to have both services running smoothly? Thanks, Ian -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Hive-and-Spark-together-with-Dynamic-Resource-Allocation-tp27968.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
