So I've seen in the documentation that (after the overhead memory is
subtracted), the memory allocations of each executor are as follows (assume
default settings):

60% for cache
40% for tasks to process data


Reading about how Spark implements shuffling, I've also seen it say "20% of
executor memory is utilized for shuffles" Does this 20% cut into the 40%
for tasks to process data or the 60% for the data cache?

Reply via email to