awdavidson commented on code in PR #38312:
URL: https://github.com/apache/spark/pull/38312#discussion_r1000927574
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala:
##########
@@ -271,6 +271,8 @@ class ParquetToSparkSchemaConverter(
} else {
TimestampNTZType
}
+ case timestamp: TimestampLogicalTypeAnnotation if timestamp.getUnit
== TimeUnit.NANOS =>
Review Comment:
I agree it's weird and if it is unintended behaviour, from what I know it's
been unintended since spark 2.3 so quite a long time (I have not checked
versions before 2.3). However, will let you guys take the call on whether is
should be considered a regression.
I assume the change to support nanos will be a breaking change; I am
completely for the support of nanos but would be nice if the original behaviour
still existed up until the support of nanos is developed and released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
