[
https://issues.apache.org/jira/browse/SPARK-57550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-57550:
----------------------------------
Affects Version/s: 4.4.0
(was: 4.3.0)
> Extend support for the TIME data type
> -------------------------------------
>
> Key: SPARK-57550
> URL: https://issues.apache.org/jira/browse/SPARK-57550
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
>
> Umbrella for follow-up work extending support for the TIME data type
> (TimeType) introduced by the SPIP SPARK-51162. Collects TIME-related tasks
> such as casts to/from other types, columnar/Arrow support, Parquet/Variant
> interop, and statistics collection.
> h2. Prioritization
> Suggested ordering of the open sub-tasks by dependency, correctness impact,
> ANSI/user value, and existing momentum (open PRs).
> h3. Dependencies / blockers
> * SPARK-57551 (TIME precision -> 9) gates SPARK-57552 and SPARK-57554.
> * SPARK-57552 and SPARK-57554 additionally depend on the nanosecond TIMESTAMP
> types (TimestampNTZNanosType / TimestampLTZNanos), tracked outside this
> umbrella.
> * SPARK-54203 (RowToColumnConverter) gates the columnar follow-ons
> SPARK-57567, SPARK-57568, SPARK-57569, SPARK-57570 (each "is blocked by" it).
> h3. Tier 1 - do first (correctness gaps; small, no dependencies)
> TIME currently throws/fails in these paths, so they behave like bugs:
> * SPARK-54203 - RowToColumnConverter: TIME hits unsupportedDataTypeError in
> row->column conversion (caching/vectorized paths). Best single first ticket.
> * SPARK-54582 - stats serialization: CatalogColumnStat.toExternalString
> throws for TIME, so ANALYZE TABLE min/max persistence is broken.
> * SPARK-57559 - add a TimeType case to PhysicalDataType: trivial robustness
> fix.
> h3. Tier 2 - high ANSI/user value, no deps, momentum (PRs exist)
> * SPARK-52617 - TIME <-> TIMESTAMP_NTZ (micros): ANSI-mandatory cast, highest
> everyday value (PR open).
> * SPARK-54281 - numeric -> TIME: completes cast symmetry (PR open).
> * SPARK-57553 - TIME <-> TIMESTAMP_LTZ (micros): finishes the ANSI cast
> matrix for the common timestamp type.
> * SPARK-52621 - TIME <-> VARIANT (PR open); needs the encoding decision first.
> h3. Tier 3 - foundational enabler for the nanosecond line
> * SPARK-57551 - precision -> 9: highest-leverage enabler; unblocks
> SPARK-57552 / SPARK-57554 and aligns TIME with the in-flight nanos TIMESTAMP
> work and ANSI's "TIME and TIMESTAMP share the same max precision" rule. Start
> early if the nanosecond direction is a release priority.
> * Then SPARK-57552 and SPARK-57554 once 57551 and the nanos TIMESTAMP types
> are in.
> h3. Tier 4 - valuable but independent / can run anytime
> * SPARK-57555 - JDBC data source: biggest migration payoff (the SPIP
> motivation), but a larger multi-dialect effort; parallelize on its own track.
> * SPARK-54507 - time_bucket (PR open), SPARK-57558 - LOCALTIME (small, ANSI),
> SPARK-57557 - quantile/sketch aggregates.
> * SPARK-57562 - TIME benchmarks: addresses the SPIP Q6 performance-regression
> risk; worth running alongside the feature work rather than deferring.
> * SPARK-57574 - TIME in pandas API on Spark: user-facing PySpark surface;
> independent of the others.
> h3. Tier 5 - lower priority / niche / polish
> * SPARK-53368 - Parquet isAdjustedToUTC=true (PR open, minor), SPARK-57560 -
> TRY-mode arithmetic, SPARK-57556 - Hive interop (Hive has no TIME; mostly a
> documented-limitation task), SPARK-51403 - ordered/atomic tests (starter),
> and docs SPARK-57030 / SPARK-57031 (do last, once behavior is settled).
> * SPARK-57571 - TIME predicate pushdown to ORC (parity with the done Parquet
> pushdown).
> * SPARK-57572 - infer TIME during CSV/JSON schema inference (guard against
> ambiguity).
> * SPARK-57573 - TIME in the Protobuf connector (parity with Avro).
> * SPARK-57575 - TIME in to_char/to_varchar (complements time_format
> SPARK-54588).
> h3. Testing and benchmarks (cross-cutting)
> Coverage to reach parity with DATE/TIMESTAMP. Feature tickets carry their own
> unit tests; these track the remaining gaps:
> * SPARK-57562 - TIME benchmarks (also in Tier 4; prioritize per SPIP Q6).
> * SPARK-57561 - verify datetime functions reject TIME with a clear error (no
> silent coercion).
> * SPARK-57563 - SQL golden-file parity (try_cast,
> datetime-parsing/formatting, postgreSQL/time.sql, ...).
> * SPARK-57564 - catalyst/core unit-test parity (ExpressionEncoderSuite,
> DDLParserSuite, DataTypeWriteCompatibilitySuite).
> * SPARK-57565 - PySpark TIME test coverage.
> * SPARK-57566 - Spark Connect TIME test coverage.
> h3. Columnar follow-ons (blocked by SPARK-54203)
> These become possible only once SPARK-54203 lands; each may also need TIME
> support in its own
> layer (Arrow type mapping, Parquet/ORC logical types, Variant encoding).
> Ordered by value:
> * SPARK-57567 - TIME in Arrow-based Python/pandas UDF evaluation (user-facing
> PySpark value).
> * SPARK-57570 - TIME in vectorized-reader column population
> (partition/missing/constant columns; read correctness).
> * SPARK-57568 - TIME in Parquet/ORC aggregate push-down (performance; niche).
> * SPARK-57569 - TIME in Parquet Variant shredding (niche; relates to
> SPARK-52621).
> h3. Bottom line
> * Implement first: SPARK-54203 (smallest, no deps, closes a real failure
> path).
> * In parallel, kick off: SPARK-57551 (foundational blocker for the nanosecond
> cast branch).
> * Then drive to done: the ANSI cast tickets with existing PRs (SPARK-52617,
> SPARK-54281).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
