MaxGekk commented on PR #53312: URL: https://github.com/apache/spark/pull/53312#issuecomment-4854129451
Filed [SPARK-57805](https://issues.apache.org/jira/browse/SPARK-57805) for the CBO estimation-side gap noted in my review above ("CBO filter/join estimation throws `MatchError` for TimestampNTZ, ANSI interval, and TIME columns"). It's an umbrella bug: `EstimationUtils.toDouble`/`fromDouble`, `FilterEstimation`, and `JoinEstimation` only enumerate NUMERIC/DATE/TIMESTAMP/BOOLEAN, so once stats are collected on a TIME (or TimestampNTZ / ANSI-interval) column, a `WHERE time_col <op> lit` / `IN (...)` / join under `spark.sql.cbo.enabled=true` hits a `scala.MatchError`. TimestampNTZ has carried the identical latent gap since SPARK-42777, so this is best fixed for all three families together rather than blocking this PR. For *this* PR, the one clearly in-scope item is `UnionEstimation.isTypeSupported` (`UnionEstimation.scala:38-43`), which lists `TimestampNTZType` + `_: AnsiIntervalType` (added in SPARK-37468) but omits `TimeType` — a safe one-liner worth folding in here. The `FilterEstimation`/`JoinEstimation` fix and its regression tests are tracked in SPARK-57805. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
