Max Gekk created SPARK-57812:
--------------------------------
Summary: Support catalog column-statistics serialization for
nanosecond-precision timestamps
Key: SPARK-57812
URL: https://issues.apache.org/jira/browse/SPARK-57812
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
precision).
h2. Problem
{{CatalogColumnStat.toExternalString}} and {{fromExternalString}}
(catalyst/catalog/interface.scala ~L1040-1081) handle only {{TimestampType}} /
{{TimestampNTZType}}; nanosecond min/max hit
{{columnStatisticsSerializationNotSupportedError}} /
{{columnStatisticsDeserializationNotSupportedError}}. In-query stat aggregation
already works ({{CommandUtils.statExprs}} uses {{case _: DatetimeType}}); the
failure is on persisting/reading catalog stats.
h2. Goal
Serialize and deserialize nanosecond timestamp min/max column statistics
to/from the catalog at full precision.
h2. Scope
Add nanosecond arms to {{toExternalString}} / {{fromExternalString}} using the
nanos-aware formatter/parser (a round-trippable text form including the
fractional digits).
h2. Acceptance criteria
* {{ANALYZE TABLE ... FOR COLUMNS <nanos_col>}} persists and reloads min/max
without error; values round-trip.
h2. Testing
{{StatisticsCollectionSuite}} / {{StatisticsSuite}}.
h2. Dependencies
None - independent. Enables the CBO FilterEstimation sub-task (which needs
persisted nanosecond min/max).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
