uros-b commented on code in PR #58693:
URL: https://github.com/apache/spark/pull/58693#discussion_r3978223650
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTypeWideningSuite.scala:
##########
@@ -225,9 +226,30 @@ class ParquetTypeWideningSuite
}
}
+ // The same widening from a legacy INT96 timestamp column. INT96 has no
logical unit; it decodes
+ // to microseconds and is promoted to (epochMicros, 0). INT96 is only
produced for the LTZ family;
+ // the write/read INT96 rebase are pinned to CORRECTED so
proleptic-Gregorian values round-trip.
for {
- outputTimestampType <-
- Seq(ParquetOutputTimestampType.INT96,
ParquetOutputTimestampType.TIMESTAMP_MILLIS)
+ toType: DataType <- Seq(
+ TimestampLTZNanosType(TimestampLTZNanosType.NANOS_PRECISION),
+ TimestampLTZNanosType(7))
+ }
+ test(s"parquet widening conversion TimestampType (int96) -> $toType") {
+ withSQLConf(
+ SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true",
+ SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key ->
ParquetOutputTimestampType.INT96.toString,
+ SQLConf.PARQUET_INT96_REBASE_MODE_IN_WRITE.key ->
LegacyBehaviorPolicy.CORRECTED.toString,
+ SQLConf.PARQUET_INT96_REBASE_MODE_IN_READ.key ->
LegacyBehaviorPolicy.CORRECTED.toString) {
Review Comment:
Tests pin INT96 rebase to CORRECTED and use 2020 / 5138. The MICROS widening
loop documents pre-1582 LEGACY rebase; INT96 needs the same, plus EXCEPTION
failIfRebase. Dictionary decode is implemented, keep checkAllParquetReaders
dict on/off.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java:
##########
@@ -227,6 +227,17 @@ public ParquetVectorUpdater getUpdater(ColumnDescriptor
descriptor, DataType spa
int96RebaseTz);
}
}
+ } else if (sparkType instanceof TimestampNTZNanosType) {
Review Comment:
Spark writes NTZ as INT64 MICROS, but the read path still accepts INT96 as
TimestampNTZNanosType. If that widening is intentional, please add a read test;
otherwise drop the arm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
