This article recommends setting spark.locality.wait to 10 (milliseconds) in the case of using Spark Streaming and gives an explanation of why they chose that value. If using batch Spark, that value should still be a good starting place https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/
On Sat, Jan 21, 2017 at 5:06 AM, Cesar <ces...@gmail.com> wrote: > > I am working with datasets of the order of 200 GB using 286 cores divided > across 143 executor. Each executor has 32 Gb (which makes every core 15 > Gb). And I am using Spark 1.6. > > > I would like to tune the spark.locality.wait. Does anyone can give me a > range on the values of spark.locality wait that I can try ? > > > > Thanks a lot ! > -- > Cesar Flores >
