[
https://issues.apache.org/jira/browse/SPARK-57581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57581:
-----------------------------
Affects Version/s: 4.2.0
(was: 4.3.0)
> Encode the TIME data type in Avro with a unit-correct logical type
> ------------------------------------------------------------------
>
> Key: SPARK-57581
> URL: https://issues.apache.org/jira/browse/SPARK-57581
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Max Gekk
> Priority: Major
> Labels: pull-request-available
>
> h2. What
> Spark's Avro encoding for the TIME data type writes the internal value
> (nanoseconds since midnight, INT64) but annotates the column with the Avro
> {{time-micros}} logical type. The declared unit (microseconds) does not match
> the stored unit (nanoseconds).
> Relevant code:
> * {{SchemaConverters.scala}}: {{case t: TimeType =>
> LogicalTypes.timeMicros().addToSchema(...)}} plus a {{spark.sql.catalyst.type
> = time(p)}} property.
> * {{AvroSerializer.scala}}: {{case (_: TimeType, LONG) =>
> getter.getLong(ordinal)}} (writes raw nanos).
> * {{AvroDeserializer.scala}}: {{case (LONG, _: TimeType) =>
> updater.setLong(ordinal, value)}} (reads raw nanos).
> Spark-to-Spark round-trips are correct because both the writer and reader
> treat the value as a raw Long and recover the precision from the
> {{spark.sql.catalyst.type}} property. However, the value is mislabeled for
> any consumer that honors the Avro {{time-micros}} logical type.
> h2. Why
> An external Avro reader (Hive, Trino, Flink, fastavro, etc.) decoding a
> Spark-written TIME column interprets the INT64 as
> microseconds-since-midnight, while it is actually nanoseconds-since-midnight
> - a 1000x error that also falls outside the valid micros-of-day range. This
> affects *all* precisions (0-9), not only 7-9, because even TIME(6) stores
> nanoseconds under {{time-micros}}.
> For comparison, the Parquet path is unit-correct: it emits TIME(MICROS) with
> a microsecond value for precision 0-6 and TIME(NANOS) with a nanosecond value
> for precision 7-9 (SPARK-57551), so Parquet files are interpretable by
> external tools.
> h2. Scope
> * Write path: emit a logical type whose unit matches the written value:
> ** precision 0-6: {{time-micros}}, write microseconds (nanos -> micros).
> ** precision 7-9: {{time-nanos}}, write nanoseconds (if the bundled Avro
> version exposes a nanosecond TIME logical type; otherwise document/choose an
> alternative encoding).
> * Read path: convert by the declared unit - {{time-micros}} -> micros ->
> nanos, {{time-nanos}} -> nanos (truncated to the requested precision). Keep
> honoring the {{spark.sql.catalyst.type}} property for precision fidelity.
> * Tests: assert that a plain (non-Spark) Avro decode yields the correct
> hour/minute/second/fraction, in addition to the existing Spark round-trip
> tests.
> h2. Backward compatibility (important)
> Changing the on-disk encoding is a format change. Avro files already written
> by current Spark store nanoseconds under {{time-micros}}; a new unit-correct
> reader would misread those legacy files (treating nanoseconds as
> microseconds). The fix must define how legacy files are detected/handled -
> e.g. keying off the {{spark.sql.catalyst.type}} property or a writer-version
> marker, or accepting the break given how recently TIME-in-Avro shipped. This
> migration question is the main design decision for the ticket.
> h2. Out of scope
> * The TIME precision extension itself (SPARK-57551).
> * Non-Avro datasources (Parquet/ORC already correct).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
