vinodkc commented on code in PR #56422:
URL: https://github.com/apache/spark/pull/56422#discussion_r3398277109
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala:
##########
@@ -694,7 +704,8 @@ class SparkToParquetSchemaConverter(
case _: TimeType =>
Types.primitive(INT64, repetition)
- .as(LogicalTypeAnnotation.timeType(false,
TimeUnit.MICROS)).named(field.name)
+ .as(LogicalTypeAnnotation.timeType(timeIsAdjustedToUTC,
TimeUnit.MICROS))
Review Comment:
Thanks!
Now, replaced all 8 `case ParquetTimeMicrosType =>` sites with `case
ParquetSchemaType(_: TimeLogicalTypeAnnotation, INT64, 0) =>` as suggested, and
removed the now-unused `ParquetTimeMicrosType` constant.
Also removed the write-path change from SparkToParquetSchemaConverter
entirely , the config now only controls the read path, so ParquetScanBuilder
always gets timeType(false, MICROS) from the converter.
Added a test in ParquetFilterSuite that writes a raw TIME(MICROS, true)
Parquet file and verifies all predicate types (eq, notEq, lt, gt, ltEq, gtEq,
in) push down correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
