[
https://issues.apache.org/jira/browse/SPARK-46295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794081#comment-17794081
]
Kazuyuki Tanimura commented on SPARK-46295:
-------------------------------------------
I realized that I am using the default {{numPartitions=100}} while the CI
(GHA) is using {{numPartitions=1}}
So it looks this issue happens when {{numPartitions!=1}}
> TPCDS q39a and a39b have correctness issues with broadcast hash join and
> shuffled hash join
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-46295
> URL: https://issues.apache.org/jira/browse/SPARK-46295
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.2, 3.5.0, 4.0.0
> Reporter: Kazuyuki Tanimura
> Priority: Major
> Labels: correctness
>
> {{TPCDSQueryTestSuite}} fails for q39a and a39b with
> {{broadcastHashJoinConf}} and {{shuffledHashJoinConf}}. It works fine with
> {{sortMergeJoinConf}}
> {code}SPARK_TPCDS_DATA=<path of TPCDS SF=1 data> build/sbt "~sql/testOnly
> *TPCDSQueryTestSuite -- -z q39a"{code}
> {code}
> [info] - q39a *** FAILED *** (19 seconds, 139 milliseconds)
> [info] java.lang.Exception: Expected "...25 1.022382911080458[8 ..." but
> got "...25 1.022382911080458[5 ..."
> {code}
> {code}SPARK_TPCDS_DATA=<path of TPCDS SF=1 data> build/sbt "~sql/testOnly
> *TPCDSQueryTestSuite -- -z q39b"{code}
> {code}
> [info] - q39b *** FAILED *** (19 seconds, 351 milliseconds)
> [info] java.lang.Exception: Expected "...34 1.563403519178623[3 3
> 10427 2 381.25 1.0623056061004696
> [info] 3 3315 1 271.75 1.555976998814345 3 3315
> 2 393.75 1.0196319345405949
> [info] 3 3393 1 260.0 1.5009563026568116 3 3393
> 2 470.25 1.129275872154205
> [info] 4 16211 1 257.6666666666667 1.6381074811154002]
> 4 16211 2 352.25 1....", but got "...34 1.563403519178623[5
> 3 10427 2 381.25 1.0623056061004696
> [info] 3 3315 1 271.75 1.555976998814345 3 3315
> 2 393.75 1.0196319345405949
> [info] 3 3393 1 260.0 1.5009563026568118 3 3393
> 2 470.25 1.129275872154205
> [info] 4 16211 1 257.6666666666667 1.6381074811154]
> 4 16211 2 352.25 1...." Result did not match
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
