[
https://issues.apache.org/jira/browse/SPARK-57581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57581:
-----------------------------
Description:
h2. What
Spark's Avro encoding for the TIME data type writes the internal value
(nanoseconds since midnight, INT64) but annotates the column with the Avro
{{time-micros}} logical type. The declared unit (microseconds) does not match
the stored unit (nanoseconds).
Relevant code:
* {{SchemaConverters.scala}}: {{case t: TimeType =>
LogicalTypes.timeMicros().addToSchema(...)}} plus a {{spark.sql.catalyst.type =
time(p)}} property.
* {{AvroSerializer.scala}}: {{case (_: TimeType, LONG) =>
getter.getLong(ordinal)}} (writes raw nanos).
* {{AvroDeserializer.scala}}: {{case (LONG, _: TimeType) =>
updater.setLong(ordinal, value)}} (reads raw nanos).
Spark-to-Spark round-trips are correct because both the writer and reader treat
the value as a raw Long and recover the precision from the
{{spark.sql.catalyst.type}} property. However, the value is mislabeled for any
consumer that honors the Avro {{time-micros}} logical type.
h2. Why
An external Avro reader (Hive, Trino, Flink, fastavro, etc.) decoding a
Spark-written TIME column interprets the INT64 as microseconds-since-midnight,
while it is actually nanoseconds-since-midnight - a 1000x error that also falls
outside the valid micros-of-day range. This affects *all* precisions (0-9), not
only 7-9, because even TIME(6) stores nanoseconds under {{time-micros}}.
For comparison, the Parquet path is unit-correct: it emits TIME(MICROS) with a
microsecond value for precision 0-6 and TIME(NANOS) with a nanosecond value for
precision 7-9 (SPARK-57551), so Parquet files are interpretable by external
tools.
h2. Scope
* Write path: emit a logical type whose unit matches the written value:
** precision 0-6: {{time-micros}}, write microseconds (nanos -> micros).
** precision 7-9: {{time-nanos}}, write nanoseconds (if the bundled Avro
version exposes a nanosecond TIME logical type; otherwise document/choose an
alternative encoding).
* Read path: convert by the declared unit - {{time-micros}} -> micros -> nanos,
{{time-nanos}} -> nanos (truncated to the requested precision). Keep honoring
the {{spark.sql.catalyst.type}} property for precision fidelity.
* Tests: assert that a plain (non-Spark) Avro decode yields the correct
hour/minute/second/fraction, in addition to the existing Spark round-trip tests.
h2. Backward compatibility (important)
Changing the on-disk encoding is a format change. Avro files already written by
current Spark store nanoseconds under {{time-micros}}; a new unit-correct
reader would misread those legacy files (treating nanoseconds as microseconds).
The fix must define how legacy files are detected/handled - e.g. keying off the
{{spark.sql.catalyst.type}} property or a writer-version marker, or accepting
the break given how recently TIME-in-Avro shipped. This migration question is
the main design decision for the ticket.
h2. Out of scope
* The TIME precision extension itself (SPARK-57551).
* Non-Avro datasources (Parquet/ORC already correct).
was:
## What
Spark's Avro encoding for the TIME data type writes the internal value
(nanoseconds since midnight, INT64) but annotates the column with the Avro
`time-micros` logical type. The declared unit (microseconds) does not match the
stored unit (nanoseconds).
Relevant code:
- SchemaConverters.scala: `case t: TimeType =>
LogicalTypes.timeMicros().addToSchema(...)`
plus a `spark.sql.catalyst.type = time(p)` property.
- AvroSerializer.scala: `case (_: TimeType, LONG) => getter.getLong(ordinal)`
(writes raw nanos).
- AvroDeserializer.scala: `case (LONG, _: TimeType) => updater.setLong(ordinal,
value)` (reads raw nanos).
Spark-to-Spark round-trips are correct because both the writer and reader treat
the value as a raw Long and recover the precision from the
`spark.sql.catalyst.type`
property. However, the value is mislabeled for any consumer that honors the Avro
`time-micros` logical type.
## Why
An external Avro reader (Hive, Trino, Flink, fastavro, etc.) decoding a
Spark-written TIME column interprets the INT64 as microseconds-since-midnight,
while it is actually nanoseconds-since-midnight - a 1000x error that also falls
outside the valid micros-of-day range. This affects ALL precisions (0-9), not
only 7-9, because even TIME(6) stores nanoseconds under `time-micros`.
For comparison, the Parquet path is unit-correct: it emits TIME(MICROS) with a
microsecond value for precision 0-6 and TIME(NANOS) with a nanosecond value for
precision 7-9 (SPARK-57551), so Parquet files are interpretable by external
tools.
## Scope
- Write path: emit a logical type whose unit matches the written value:
- precision 0-6: `time-micros`, write microseconds (nanos -> micros).
- precision 7-9: `time-nanos`, write nanoseconds (if the bundled Avro version
exposes a nanosecond TIME logical type; otherwise document/choose an
alternative encoding).
- Read path: convert by the declared unit - `time-micros` -> micros -> nanos,
`time-nanos` -> nanos (truncated to the requested precision). Keep honoring
the
`spark.sql.catalyst.type` property for precision fidelity.
- Tests: assert that a plain (non-Spark) Avro decode yields the correct
hour/minute/second/fraction, in addition to the existing Spark round-trip
tests.
## Backward compatibility (important)
Changing the on-disk encoding is a format change. Avro files already written by
current Spark store nanoseconds under `time-micros`; a new unit-correct reader
would misread those legacy files (treating nanoseconds as microseconds). The fix
must define how legacy files are detected/handled - e.g. keying off the
`spark.sql.catalyst.type` property or a writer-version marker, or accepting the
break given how recently TIME-in-Avro shipped. This migration question is the
main design decision for the ticket.
## Out of scope
- The TIME precision extension itself (SPARK-57551).
- Non-Avro datasources (Parquet/ORC already correct).
> Encode the TIME data type in Avro with a unit-correct logical type
> ------------------------------------------------------------------
>
> Key: SPARK-57581
> URL: https://issues.apache.org/jira/browse/SPARK-57581
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> h2. What
> Spark's Avro encoding for the TIME data type writes the internal value
> (nanoseconds since midnight, INT64) but annotates the column with the Avro
> {{time-micros}} logical type. The declared unit (microseconds) does not match
> the stored unit (nanoseconds).
> Relevant code:
> * {{SchemaConverters.scala}}: {{case t: TimeType =>
> LogicalTypes.timeMicros().addToSchema(...)}} plus a {{spark.sql.catalyst.type
> = time(p)}} property.
> * {{AvroSerializer.scala}}: {{case (_: TimeType, LONG) =>
> getter.getLong(ordinal)}} (writes raw nanos).
> * {{AvroDeserializer.scala}}: {{case (LONG, _: TimeType) =>
> updater.setLong(ordinal, value)}} (reads raw nanos).
> Spark-to-Spark round-trips are correct because both the writer and reader
> treat the value as a raw Long and recover the precision from the
> {{spark.sql.catalyst.type}} property. However, the value is mislabeled for
> any consumer that honors the Avro {{time-micros}} logical type.
> h2. Why
> An external Avro reader (Hive, Trino, Flink, fastavro, etc.) decoding a
> Spark-written TIME column interprets the INT64 as
> microseconds-since-midnight, while it is actually nanoseconds-since-midnight
> - a 1000x error that also falls outside the valid micros-of-day range. This
> affects *all* precisions (0-9), not only 7-9, because even TIME(6) stores
> nanoseconds under {{time-micros}}.
> For comparison, the Parquet path is unit-correct: it emits TIME(MICROS) with
> a microsecond value for precision 0-6 and TIME(NANOS) with a nanosecond value
> for precision 7-9 (SPARK-57551), so Parquet files are interpretable by
> external tools.
> h2. Scope
> * Write path: emit a logical type whose unit matches the written value:
> ** precision 0-6: {{time-micros}}, write microseconds (nanos -> micros).
> ** precision 7-9: {{time-nanos}}, write nanoseconds (if the bundled Avro
> version exposes a nanosecond TIME logical type; otherwise document/choose an
> alternative encoding).
> * Read path: convert by the declared unit - {{time-micros}} -> micros ->
> nanos, {{time-nanos}} -> nanos (truncated to the requested precision). Keep
> honoring the {{spark.sql.catalyst.type}} property for precision fidelity.
> * Tests: assert that a plain (non-Spark) Avro decode yields the correct
> hour/minute/second/fraction, in addition to the existing Spark round-trip
> tests.
> h2. Backward compatibility (important)
> Changing the on-disk encoding is a format change. Avro files already written
> by current Spark store nanoseconds under {{time-micros}}; a new unit-correct
> reader would misread those legacy files (treating nanoseconds as
> microseconds). The fix must define how legacy files are detected/handled -
> e.g. keying off the {{spark.sql.catalyst.type}} property or a writer-version
> marker, or accepting the break given how recently TIME-in-Avro shipped. This
> migration question is the main design decision for the ticket.
> h2. Out of scope
> * The TIME precision extension itself (SPARK-57551).
> * Non-Avro datasources (Parquet/ORC already correct).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
