MaxGekk commented on PR #3125: URL: https://github.com/apache/avro/pull/3125#issuecomment-4757474467
From the Apache Spark side, +1 on adding a `time-nanos` logical type - this would close a real interoperability gap for us. Spark is about to support `TIME(p)` for `p` in `[0, 9]` (i.e. up to nanosecond precision), to match its existing nanosecond `TIMESTAMP`/`TIMESTAMP_NTZ` support and to stay ANSI-consistent (ISO/IEC 9075-2 requires the maximum `<time precision>` to equal the maximum `<timestamp precision>`, which Spark already has at 9). The problem: Avro today has `timestamp-nanos` and `local-timestamp-nanos`, but no nanosecond time-of-day type. The most precise standard option is `time-micros`. That means Spark cannot store `TIME(7)`, `TIME(8)`, or `TIME(9)` values into Avro without truncating them to microseconds - the sub-microsecond digits are lost (or, worse, mislabeled if written as raw nanos under `time-micros`). By contrast, Parquet already offers `TIME(NANOS)`, so Avro ends up strictly lower-fidelity than the equivalent Parquet file for the same column. Adding `time-nanos` (a `long` of nanoseconds-since-midnight; max time-of-day 86,400,000,000,000 ns fits comfortably in `int64`, as noted in AVRO-4043) would let Spark round-trip `TIME(7-9)` through Avro losslessly and in a way that external readers can interpret correctly. For reference, the Spark-side tracking is SPARK-57581 (making the Avro TIME encoding unit-correct); the nanosecond precision extension is SPARK-57551. Happy to help validate from the Spark consumer side once a Java implementation lands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
