[
https://issues.apache.org/jira/browse/SPARK-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15673843#comment-15673843
]
Sean Owen commented on SPARK-18479:
-----------------------------------
Yes, that's not a good hash function for this purpose, but it is indeed a
default hash function for types like Long or Timestamp. In user code, you can
avoid this by partitioning by something different (like, a hash function with
better properties, or its string representation), but you can't control that in
SQL.
Yes, 199 vs 200 partitions doesn't matter much with 50 or 1000 cores. It would
matter when they're close. The number of partitions is user-controllable (and
can be made prime if desired). But you're commonly going to set this to the
number of cores available, which I suppose makes this moot in a lot of cases
anyway. I think it'd be surprising to most people to find an odd number of
partitions pop out.
> spark.sql.shuffle.partitions defaults should be a prime number
> --------------------------------------------------------------
>
> Key: SPARK-18479
> URL: https://issues.apache.org/jira/browse/SPARK-18479
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.2
> Reporter: Hamel Ajay Kothari
>
> For most hash bucketing use cases it is my understanding that a prime value,
> such as 199, would be a safer value than the existing value of 200. Using a
> non-prime value makes the likelihood of collisions much higher when the hash
> function isn't great.
> Consider the case where you've got a Timestamp or Long column with
> millisecond times at midnight each day. With the default value for
> spark.sql.shuffle.partitions, you'll end up with 120/200 partitions being
> completely empty.
> Looking around there doesn't seem to be a good reason why we chose 200 so I
> don't see a huge risk in changing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
