HyukjinKwon commented on a change in pull request #28466:
URL: https://github.com/apache/spark/pull/28466#discussion_r421182591
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
##########
@@ -952,18 +952,23 @@ class ParquetIOSuite extends QueryTest with ParquetTest
with SharedSparkSession
.write
.parquet(path)
}
- // The file metadata indicates if it needs rebase or not, so we
can always get the
- // correct result regardless of the "rebaseInRead" config.
- Seq(true, false).foreach { rebase =>
- withSQLConf(SQLConf.LEGACY_AVRO_REBASE_DATETIME_IN_READ.key ->
rebase.toString) {
- checkAnswer(spark.read.parquet(path),
Row(Timestamp.valueOf(tsStr)))
- }
- }
- // Force to not rebase to prove the written datetime values are
rebased and we will get
- // wrong result if we don't rebase while reading.
- withSQLConf("spark.test.forceNoRebase" -> "true") {
- checkAnswer(spark.read.parquet(path),
Row(Timestamp.valueOf(nonRebased)))
+ Seq(false, true).foreach { vectorized =>
+ withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key ->
vectorized.toString) {
+ // The file metadata indicates if it needs rebase or not, so
we can always get the
+ // correct result regardless of the "rebaseInRead" config.
+ Seq(true, false).foreach { rebase =>
+ withSQLConf(SQLConf.LEGACY_AVRO_REBASE_DATETIME_IN_READ.key
-> rebase.toString) {
Review comment:
Looks it should be Parquet instead as you fixed. Looks good otherwise.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
