[
https://issues.apache.org/jira/browse/SPARK-57839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093007#comment-18093007
]
Anupam Yadav commented on SPARK-57839:
--------------------------------------
I would like to work on this. Plan: extend CBO filter/selectivity estimation to
nanosecond-precision timestamp types, mirroring the existing
microsecond-precision handling (follows on from the TIME CBO work in
SPARK-57784). Will put up a PR shortly.
> Support CBO filter/selectivity estimation for nanosecond-precision timestamps
> -----------------------------------------------------------------------------
>
> Key: SPARK-57839
> URL: https://issues.apache.org/jira/browse/SPARK-57839
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
> precision).
> h2. Problem
> {{FilterEstimation}} (execution/.../FilterEstimation.scala ~L294, ~L416)
> matches microsecond {{TimestampType}} only (not even {{TimestampNTZType}}),
> and {{EstimationUtils.toDouble}} handles {{TimestampType}} only, so
> nanosecond columns get no range-based selectivity and produce silent
> mis-estimates.
> h2. Goal
> Include nanosecond timestamp columns in range/selectivity estimation,
> converting {{TimestampNanosVal}} min/max to the numeric domain used by the
> estimator.
> h2. Scope
> Broaden the type matches to include {{AnyTimestampNanoType}} (ideally via
> {{DatetimeType}}); add nanosecond value<->double conversion in
> {{EstimationUtils}}.
> h2. Acceptance criteria
> * Filter/range predicate selectivity on nanosecond columns is estimated (not
> defaulted); plans reflect stats.
> h2. Testing
> {{FilterEstimationSuite}}.
> h2. Dependencies
> Do AFTER SPARK-57812 (catalog column-statistics serialization) - needs
> persisted nanosecond min/max to be meaningful.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
