viirya commented on pull request #33564: URL: https://github.com/apache/spark/pull/33564#issuecomment-889492218
I see. lgtm. I am not in front of laptop. Will merge this after I come back. On Thu, Jul 29, 2021, 15:06 Chao Sun ***@***.***> wrote: > ***@***.**** commented on this pull request. > ------------------------------ > > In > sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitionsSuite.scala > <https://github.com/apache/spark/pull/33564#discussion_r679516263>: > > > @@ -116,7 +107,7 @@ class PruneFileSourcePartitionsSuite extends PrunePartitionSuiteBase { > // Force datasource v2 for parquet > withSQLConf((SQLConf.USE_V1_SOURCE_LIST.key, "")) { > withTempPath { dir => > - spark.range(10).selectExpr("id", "id % 3 as p") > + spark.range(10).coalesce(1).selectExpr("id", "id % 3 as p") > > @viirya <https://github.com/viirya> @huaxingao > <https://github.com/huaxingao> this is because TestHiveSingleton uses > local[1] while SharedSparkSession uses local[2] so it will generate 2 > Parquet files. As result, ParquetScan will generate 2 input splits which > makes the test fail since the provided value for expectedPartitionCount > is 1. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/spark/pull/33564#discussion_r679516263>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAAQZ54KNSERYH4IFNPIYE3T2HGGDANCNFSM5BFLC3HA> > . > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
