caican created SPARK-43526:
------------------------------
Summary: when shuffle hash join is enabled, q95 performance
deteriorates
Key: SPARK-43526
URL: https://issues.apache.org/jira/browse/SPARK-43526
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.2.0, 3.1.2
Reporter: caican
Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when
shuffle hash join is enabled and the performance is better when sortMergeJoin
is used.
>From 8.1min(shuffledHashJoin) to 3.9min(sortMergeJoin).
enable shuffledHashJoin, the execution plan is as follows:
!image-2023-05-16-21-01-53-423.png!
!image-2023-05-16-21-16-37-376.png!
disable shuffledHashJoin, the execution plan is as follows:
!image-2023-05-16-21-05-45-361.png!
!image-2023-05-16-21-16-13-128.png!
And When shuffledHashJoin is enabled, gc is very serious
!image-2023-05-16-21-12-24-618.png!
But sortMergeJoin executes without this problem
!image-2023-05-16-21-15-21-047.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
