stevomitric commented on code in PR #56941:
URL: https://github.com/apache/spark/pull/56941#discussion_r4368107643
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala:
##########
@@ -1034,6 +1035,20 @@ object CatalogColumnStat extends Logging {
forTimestampNTZ = forTimestampNTZ)
}
+ /**
+ * Returns a [[TimestampFormatter]] with nanosecond precision (9 fractional
digits) for
+ * lossless serialization of nanosecond-precision timestamp column
statistics.
+ */
+ private def getTimestampNanosFormatter(
+ isParsing: Boolean,
+ forTimestampNTZ: Boolean = false): TimestampFormatter = {
+ TimestampFormatter(
+ format = "yyyy-MM-dd HH:mm:ss.SSSSSSSSS",
Review Comment:
will this throw under `timeParserPolicy=LEGACY` ?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/EstimationUtils.scala:
##########
@@ -145,6 +151,11 @@ object EstimationUtils {
case BooleanType => double.toInt == 1
case DateType => double.toInt
case TimestampType => double.toLong
+ case _: AnyTimestampNanoType =>
+ // Lossy: the input Double may have lost precision in toDouble (see
above).
+ // Acceptable for CBO estimation only.
+ val nanos = double.toLong
Review Comment:
what happens outside `1677–2262` range ? nanos can reach year 9999 per sql
standard
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
