Max Gekk created SPARK-57581:
--------------------------------
Summary: Encode the TIME data type in Avro with a unit-correct
logical type
Key: SPARK-57581
URL: https://issues.apache.org/jira/browse/SPARK-57581
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
## What
Spark's Avro encoding for the TIME data type writes the internal value
(nanoseconds since midnight, INT64) but annotates the column with the Avro
`time-micros` logical type. The declared unit (microseconds) does not match the
stored unit (nanoseconds).
Relevant code:
- SchemaConverters.scala: `case t: TimeType =>
LogicalTypes.timeMicros().addToSchema(...)`
plus a `spark.sql.catalyst.type = time(p)` property.
- AvroSerializer.scala: `case (_: TimeType, LONG) => getter.getLong(ordinal)`
(writes raw nanos).
- AvroDeserializer.scala: `case (LONG, _: TimeType) => updater.setLong(ordinal,
value)` (reads raw nanos).
Spark-to-Spark round-trips are correct because both the writer and reader treat
the value as a raw Long and recover the precision from the
`spark.sql.catalyst.type`
property. However, the value is mislabeled for any consumer that honors the Avro
`time-micros` logical type.
## Why
An external Avro reader (Hive, Trino, Flink, fastavro, etc.) decoding a
Spark-written TIME column interprets the INT64 as microseconds-since-midnight,
while it is actually nanoseconds-since-midnight - a 1000x error that also falls
outside the valid micros-of-day range. This affects ALL precisions (0-9), not
only 7-9, because even TIME(6) stores nanoseconds under `time-micros`.
For comparison, the Parquet path is unit-correct: it emits TIME(MICROS) with a
microsecond value for precision 0-6 and TIME(NANOS) with a nanosecond value for
precision 7-9 (SPARK-57551), so Parquet files are interpretable by external
tools.
## Scope
- Write path: emit a logical type whose unit matches the written value:
- precision 0-6: `time-micros`, write microseconds (nanos -> micros).
- precision 7-9: `time-nanos`, write nanoseconds (if the bundled Avro version
exposes a nanosecond TIME logical type; otherwise document/choose an
alternative encoding).
- Read path: convert by the declared unit - `time-micros` -> micros -> nanos,
`time-nanos` -> nanos (truncated to the requested precision). Keep honoring
the
`spark.sql.catalyst.type` property for precision fidelity.
- Tests: assert that a plain (non-Spark) Avro decode yields the correct
hour/minute/second/fraction, in addition to the existing Spark round-trip
tests.
## Backward compatibility (important)
Changing the on-disk encoding is a format change. Avro files already written by
current Spark store nanoseconds under `time-micros`; a new unit-correct reader
would misread those legacy files (treating nanoseconds as microseconds). The fix
must define how legacy files are detected/handled - e.g. keying off the
`spark.sql.catalyst.type` property or a writer-version marker, or accepting the
break given how recently TIME-in-Avro shipped. This migration question is the
main design decision for the ticket.
## Out of scope
- The TIME precision extension itself (SPARK-57551).
- Non-Avro datasources (Parquet/ORC already correct).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
