vrjdev opened a new pull request, #58946:
URL: https://github.com/apache/spark/pull/58946
### What changes were proposed in this pull request?
This PR extends `CatalogColumnStat.toExternalString`/`fromExternalString` to
support `TimestampLTZNanosType`/`TimestampNTZNanosType` (nanosecond-precision
`TIMESTAMP_LTZ(p)`/`TIMESTAMP_NTZ(p)` columns), reusing the existing
`TimestampFormatter.{formatNanos, parseNanos, formatWithoutTimeZoneNanos,
parseWithoutTimeZoneNanos}` API added for the nanosecond-timestamp SPIP
(SPARK-56822). `DESCRIBE TABLE EXTENDED` is updated to render nanosecond LTZ
column stats in the session time zone, matching the existing microsecond
behavior.
Making these stats collectible immediately exposes them to CBO code paths
(`EstimationUtils`, `FilterEstimation`, `JoinEstimation`, `UnionEstimation`,
`CommandUtils.supportsHistogram`) that previously never saw a
nanosecond-timestamp value and would crash (`MatchError`) or silently drop
stats once `ANALYZE` could produce them. This PR adds the minimal handling
needed so those paths don't crash, then (self-review follow-up) fixes a
precision bug in that handling: the initial `toDouble` conversion projected a
nanosecond value down to its epoch-microseconds component only, which could
make two distinct nanosecond values collapse to the same `Double` and corrupt
CBO min/max/selectivity estimation. The follow-up encodes the sub-microsecond
remainder as a fractional component instead, reconstructing it losslessly in
`fromDouble`, and extends `UnionEstimation.isTypeSupported` and
`JoinEstimation.computeByHistogram` to the same types. Histogram collection is
explicitly excluded for these
types rather than taught a new composite value type; basic min/max/ndv stats
are unaffected.
This also touches SPARK-57839 (CBO filter/selectivity estimation for
nanosecond timestamps) and SPARK-57805 (CBO `MatchError` for
`TimestampNTZ`/interval/`TIME` columns) -- the crash-prevention and precision
work here was a prerequisite for SPARK-57812 to be safely mergeable, but does
not claim to fully resolve either of those broader tickets.
### Why are the changes needed?
Before this change, `ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS` on a
`TIMESTAMP_LTZ(p)`/`TIMESTAMP_NTZ(p)` column threw
`columnStatisticsSerializationNotSupportedError`, so these newer
nanosecond-precision types (SPARK-56822) couldn't get column statistics at all.
### Does this PR introduce _any_ user-facing change?
Yes.
- `ANALYZE TABLE ... FOR COLUMNS` and `DESCRIBE TABLE EXTENDED` now work for
`TIMESTAMP_LTZ(p)`/`TIMESTAMP_NTZ(p)` columns instead of throwing
`columnStatisticsSerializationNotSupportedError`.
- Queries with `spark.sql.cbo.enabled=true` that filter, join, or `ANALYZE`
on such columns no longer throw `MatchError`, and estimate cardinality using
full nanosecond precision rather than crashing or silently dropping min/max on
`UNION ALL`.
- `ANALYZE ... FOR COLUMNS` with histograms enabled skips histogram
collection for these columns (min/max/ndv/null-count stats still collected)
instead of failing.
### How was this patch tested?
Added `CatalogColumnStatSuite` (new), `EstimationUtilsSuite` (new), and new
cases in `StatisticsCollectionSuite` covering DESC round-trip, histogram-skip,
CBO estimation over nanosecond predicates (LTZ and NTZ), and UNION ALL min/max
propagation.
Ran locally with JDK 17 (build/sbt):
- `catalyst/testOnly CatalogColumnStatSuite EstimationUtilsSuite
FilterEstimationSuite JoinEstimationSuite UnionEstimationSuite` -- 117/117
passed.
- `sql/testOnly StatisticsCollectionSuite CommandUtilsSuite` -- 47/47 passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code claude-sonnet-5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
