[
https://issues.apache.org/jira/browse/SPARK-38853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated SPARK-38853:
--------------------------------
Attachment: enable.png
> optimizeSkewsInRebalancePartitions has performance issue
> --------------------------------------------------------
>
> Key: SPARK-38853
> URL: https://issues.apache.org/jira/browse/SPARK-38853
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Yuming Wang
> Priority: Major
> Attachments: Disable.png, enable.png
>
>
> How to reproduce this issue:
> {code:sql}
> CREATE TABLE t USING PARQUET
> AS
> SELECT
> /*+ REBALANCE */
> A.SESSION_START_DT
> , COALESCE(A.SITE_ID,0) AS SITE_ID
> , A.GUID
> , COALESCE(CAST(A.SESSION_SKEY AS BIGINT),0) AS SESSION_SKEY
> , COALESCE(CAST(A.SEQNUM AS INT),0) AS SEQNUM
>
> , COALESCE(A.IMP_PAGE_ID,0) AS IMP_PAGE_ID
> , COALESCE(A.PLACEMENT_ID,0) AS PLACEMENT_ID
> , A.PRODUCT_LINE_CODE
> , A.ALGORITHM_ID
> , A.MEID
> , A.ALGO_OUTPUT_ITEMS
> , A.CLICKS
> , A.GMV_7D
> FROM big_partition_table A
> WHERE
> DT BETWEEN DATE_FORMAT(DATE_SUB(CURRENT_DATE,11), 'yyyyMMdd') AND
> DATE_FORMAT(DATE_ADD(DATE_SUB(CURRENT_DATE,11),0), 'yyyyMMdd')
> AND TO_DATE(from_unixtime(unix_timestamp(A.SESSION_START_DT,
> 'yyyy/MM/dd'))) = DATE_SUB(CURRENT_DATE,11)
> AND ICFBOT = '00';
> {code}
> Enabling optimizeSkewsInRebalancePartitions takes more than 2 hours and the
> driver hangs:
> Disabling optimizeSkewsInRebalancePartitions takes only 29 minutes:
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
