johntomcat7408-cmyk opened a new pull request, #57526:
URL: https://github.com/apache/spark/pull/57526
### What changes were proposed in this pull request?
This PR extends CBO filter and join statistics estimation to support
TimestampNTZType, TimeType, YearMonthIntervalType, and DayTimeIntervalType.
It updates EstimationUtils to convert these Catalyst internal numeric values
to and from Double, and routes range and IN predicates for these types
through
the existing numeric estimation path.
It also adds regression coverage for equality, range, IN, and equi-join
estimation. UnionEstimation is intentionally excluded because TimeType
support
there is handled by apache/spark#53312.
### Why are the changes needed?
When CBO is enabled and column statistics exist, filters or equi-joins on
TimestampNTZ, ANSI interval, or TIME columns can reach non-exhaustive type
matches in the statistics-estimation layer and throw scala.MatchError during
query optimization.
These types use numeric Catalyst internal representations, so they can reuse
the existing NumericValueInterval estimation path once conversion and
dispatch
recognize them.
### Does this PR introduce _any_ user-facing change?
Yes. Queries using CBO and collected statistics on TimestampNTZ, ANSI
interval,
or TIME columns no longer fail with scala.MatchError during query planning.
There is no new API or configuration.
### How was this patch tested?
On CentOS 7 with JDK 21:
- `catalyst/Test/compile`
- `catalyst/testOnly
org.apache.spark.sql.catalyst.statsEstimation.FilterEstimationSuite`
(86 tests passed)
- `catalyst/testOnly
org.apache.spark.sql.catalyst.statsEstimation.JoinEstimationSuite`
(19 tests passed)
### Was this patch authored or co-authored using generative AI tooling?
Yes. Generated-by: OpenAI Codex (GPT-5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
