Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21527
yes it is debatable, but with it being configurable it gives you the option
to change. the only other way is to change the # of partitions which could be
more costly. There have been various talks about this number at spark summits
where people have told customers to increase it to be 2001 partitions. Note
if you just do a search for spark 2000 partitions you will fine various things
all talking about this number.
Personally I would like to see it configurable. If there is enough that
disagree we can just close. I have seen a few users bump up the number of
partitions when they were close to 2001 to get the memory saving, but I don't
have any hard numbers.
@squito @vanzin have you seen people changing # of partitions where this
config would be useful?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
