Re: spilling in-memory map of 5.1 MB to disk (272 times so far)

Here's a bunch of configuration for that https://spark.apache.org/docs/latest/configuration.html#shuffle-behavior Thanks Best Regards On Fri, Jun 26, 2015 at 10:37 PM, igor.berman igor.ber...@gmail.com wrote: Hi, wanted to get some advice regarding tunning spark application I see for some of

spilling in-memory map of 5.1 MB to disk (272 times so far)

Hi, wanted to get some advice regarding tunning spark application I see for some of the tasks many log entries like this Executor task launch worker-38 ExternalAppendOnlyMap: Thread 239 spilling in-memory map of 5.1 MB to disk (272 times so far) (especially when inputs are considerable) I