wangyum commented on a change in pull request #25891: [SPARK-29203][SQL][TESTS]
Reduce shuffle partitions in SQLQueryTestSuite
URL: https://github.com/apache/spark/pull/25891#discussion_r326953006
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -133,6 +133,10 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
private val notIncludedMsg = "[not included in comparison]"
private val clsName = this.getClass.getCanonicalName
+ override def sparkConf: SparkConf = super.sparkConf
+ // Reduce shuffle partitions to reduce testing time.
+ .set(SQLConf.SHUFFLE_PARTITIONS, 5)
Review comment:
`=4 ` has another issue: `--SET
spark.sql.autoBroadcastJoinThreshold=10485760` will success, but `--SET
spark.sql.autoBroadcastJoinThreshold=-1,spark.sql.join.preferSortMergeJoin=true`
will failed:
```
22:31:31.233 ERROR org.apache.spark.sql.SQLQueryTestSuite: Error using
configs:
spark.sql.autoBroadcastJoinThreshold=-1,spark.sql.join.preferSortMergeJoin=true,spark.sql.codegen.wholeStage=true,spark.sql.codegen.factoryMode=CODEGEN_ONLY
[info] - subquery/in-subquery/not-in-joins.sql *** FAILED *** (32 seconds,
609 milliseconds)
[info] subquery/in-subquery/not-in-joins.sql
[info] Expected "1 16 12 [21
[info] 1 16 12 10]
[info] 1 10 NULL 12
[info] 1 6 8 ...", but got "1 16 12 [10
[info] 1 16 12 21]
[info] 1 10 NULL 12
[info] 1 6 8 ..." Result did not match for query #6
[info] SELECT Count(DISTINCT( t1a )),
[info] t1b,
[info] t1c,
[info] t1d
[info] FROM t1
[info] WHERE t1a NOT IN (SELECT t2a
[info] FROM t2
[info] JOIN t1
[info] WHERE t2b <> t1b)
[info] GROUP BY t1b,
[info] t1c,
[info] t1d
[info] HAVING t1d NOT IN (SELECT t2d
[info] FROM t2
[info] WHERE t1d = t2d)
[info] ORDER BY t1b DESC (SQLQueryTestSuite.scala:383)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
