[
https://issues.apache.org/jira/browse/SPARK-24076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451743#comment-16451743
]
yucai commented on SPARK-24076:
-------------------------------
1. When shuffle.partition = 8192, tuples in the same partition follows the
connection like below:
hash(tuple x) = hash(tuple y) + n * 8192
2. In the next HashAggregate stage, tuples from the same partition need put
into a 16K BytesToBytesMap (unsafeRowAggBuffer).
Here, the HashAggregate uses the same hash algorithm and seed as shuffle, it
leads to all tuples will be hashed to only 2 different places actually. That's
why hash conflict happens.
> very bad performance when shuffle.partition = 8192
> --------------------------------------------------
>
> Key: SPARK-24076
> URL: https://issues.apache.org/jira/browse/SPARK-24076
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: yucai
> Priority: Major
> Attachments: image-2018-04-25-14-29-39-958.png, p1.png, p2.png
>
>
> We see very bad performance when shuffle.partition = 8192 on some cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
