[
https://issues.apache.org/jira/browse/AVRO-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18090361#comment-18090361
]
Max Gekk commented on AVRO-4043:
--------------------------------
A use case from Apache Spark in support of this proposal.
Spark is extending its TIME data type to support TIME(p) for p in [0, 9] (up to
nanosecond precision), matching its existing nanosecond TIMESTAMP/TIMESTAMP_NTZ
support and ANSI SQL, which requires the maximum <time precision> to equal the
maximum <timestamp precision>.
Avro currently provides timestamp-nanos and local-timestamp-nanos, but no
nanosecond time-of-day logical type. The most precise standard option for
time-of-day is time-micros, so Spark cannot store TIME(7..9) values into Avro
without truncating them to microseconds (losing the sub-microsecond digits).
Parquet already offers TIME(NANOS), so Avro ends up lower-fidelity than the
equivalent Parquet file for the same column.
Adding time-nanos (a long of nanoseconds-since-midnight) would let Spark
round-trip TIME(7..9) through Avro losslessly and interpretably for external
readers. Spark-side tracking: SPARK-57581 (unit-correct Avro TIME encoding) and
SPARK-57551 (TIME nanosecond precision). See also the spec PR apache/avro#3125.
+1 from a downstream consumer.
> Add time-nanos logical type
> ---------------------------
>
> Key: AVRO-4043
> URL: https://issues.apache.org/jira/browse/AVRO-4043
> Project: Apache Avro
> Issue Type: New Feature
> Components: logical types, spec
> Affects Versions: 1.12.0
> Reporter: Guillaume Labourey
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> To align with nanoseconds precision of timestamps and local timestamps, I
> would propose adding this precision to the time logical type Avro
> specification as well.
> Maximum time of day is 86 400 000 000 000 nano seconds. It can easily be
> encoded in a {{long}} type because the maximum signed value is commonly 9 223
> 372 036 854 775 807 as describe in the following table.
> ||Language||Maximum||
> |C++|[std::numeric_limits|http://en.cppreference.com/w/cpp/types/numeric_limits]<long>::max()|
> |Java|Long.MAX_VALUE|
> |Rust|i64::MAX|
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
