stevomitric commented on code in PR #56407:
URL: https://github.com/apache/spark/pull/56407#discussion_r3421496460
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala:
##########
@@ -268,6 +281,18 @@ class ParquetWriteSupport extends
WriteSupport[InternalRow] with Logging {
// MICROS time unit.
(row: SpecializedGetters, ordinal: Int) =>
recordConsumer.addLong(row.getLong(ordinal))
+ // TIMESTAMP(NANOS) values are always proleptic Gregorian and are exempt
from datetime
+ // rebasing; see the TIMESTAMP(NANOS) converters in
`ParquetRowConverter` for details.
Review Comment:
Confirmed it produces a clear error, so I don't think a dedicated case is
needed. When the annotation isn't NANOS the nanos guards fail and makeConverter
falls through to its default case t => throw
cannotCreateParquetConverterForDataTypeError(t, parquetType), which raises
PARQUET_CONVERSION_FAILURE.UNSUPPORTED naming both the requested Spark type and
the actual Parquet type. This is the same fall-through the existing types use -
TimestampType/TimestampNTZTyp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
