Hi all, I'd like to discuss how Avro timestamps should be represented in V4. Personally, I'm in favor of following the Avro 1.12.0 spec and dropping the Iceberg-specific conventions, but I'd love to hear other ideas on this.
The goals are: 1. Make it easier to write Avro-encoded data into Iceberg tables with ingestion tools like the Kafka Iceberg Sink and the Flink Iceberg Sink 2. Allow standard Avro files to be added to a table with no modifications needed 3. Generally avoid diverging from the Avro spec when there's no real need to :) Currently, [the Appendix A - Avro](https://iceberg.apache.org/spec/#avro) mandates the following mappings: |Type|Avro type|Notes| |--- |--- |--- | |**`timestamp`** | `{ "type": "long", "logicalType": "timestamp-micros", "adjust-to-utc": false }` | Stores microseconds from 1970-01-01 00:00:00.000000. [1] | |**`timestamptz`** | `{ "type": "long", "logicalType": "timestamp-micros", "adjust-to-utc": true }` | Stores microseconds from 1970-01-01 00:00:00.000000 UTC. [1] | |**`timestamp_ns`** | `{ "type": "long", "logicalType": "timestamp-nanos" , "adjust-to-utc": false }` | Stores nanoseconds from 1970-01-01 00:00:00.000000000. [1], [2] | |**`timestamptz_ns`** | `{ "type": "long", "logicalType": "timestamp-nanos" , "adjust-to-utc": true }` | Stores nanoseconds from 1970-01-01 00:00:00.000000000 UTC. [1], [2] | Notes: 1. Avro type annotation `adjust-to-utc` is an Iceberg convention; default value is `false` if not present. 2. Avro logical type `timestamp-nanos` is an Iceberg convention; the Avro specification does not define this type. Avro 1.10.0 (released 2020-07-01) added `local-timestamp-millis` and `local-timestamp-micros`, and 1.12.0 (2024-08-05) added `timestamp-nanos` and `local-timestamp-nanos` on top of that. One option would be to simply add support for `local-timestamp-(micros|nanos)` for timestamps without a zone, on top of the existing mappings. The catch is that timestamp logical types have been UTC-adjusted by definition since they were introduced in Avro 1.8.0 (2016-01-29), whereas the Iceberg convention treats `timestamp-(micros|nanos)` without `adjust-to-utc` as local. In practice this is probably only a spec-level inconsistency, since the core library has always [set `adjust-to-utc` explicitly, going back at least to 0.7.0]( https://github.com/apache/iceberg/blob/apache-iceberg-0.7.0-incubating/core/src/main/java/org/apache/iceberg/avro/TypeToSchema.java#L53-L54). The only place timestamp types can actually appear in v1-v3 manifests is in partition data, when the partition spec applies an `identity` or `void` transform to a timestamp field - there are no static timestamp fields defined anywhere else in the spec. Since Avro has had everything Iceberg needs for over two years now, rather than growing the spec further, I'd rather follow upstream and drop `adjust-to-utc` entirely in V4: |Type|Avro type|Notes| |--- |--- |--- | |**`timestamp`** | `{ "type": "long", "logicalType": "local-timestamp-micros" }` | Stores microseconds from 1970-01-01 00:00:00.000000. | |**`timestamptz`** | `{ "type": "long", "logicalType": "timestamp-micros" }` | Stores microseconds from 1970-01-01 00:00:00.000000 UTC. | |**`timestamp_ns`** | `{ "type": "long", "logicalType": "local-timestamp-nanos" }` | Stores nanoseconds from 1970-01-01 00:00:00.000000000. | |**`timestamptz_ns`** | `{ "type": "long", "logicalType": "timestamp-nanos" }` | Stores nanoseconds from 1970-01-01 00:00:00.000000000 UTC. | No notes needed. According to [the ongoing work](https://github.com/apache/iceberg/pull/16025), manifests will get a per-entry `format_version` `int` field (0: PRE-V4, 4: V4). Readers could use this field to interpret `timestamp-*`: - `format_version=0`: read `adjust-to-utc`, defaulting to `false` - `format_version>=4`: always read as UTC-adjusted, ignore `adjust-to-utc` This would resolve the conflict and remove any ambiguity. It would also let users add external Avro files to a table without rewriting them - a file gets `format_version=0` if its embedded writer schema is Iceberg v1-3 spec compliant, or `format_version=4` if it's a plain Avro schema. Does this seem like a valid use of the `format_version` field? The work to support `local-timestamp-*` and to read `timestamp-*` as UTC-adjusted is already underway in [PR 17196]( https://github.com/apache/iceberg/pull/17196), and I'm starting this discussion to help move it forward. There are several other related issues and PRs linked from that PR's description, so this isn't a new topic - users have wanted "new" Avro logical types integrated into Iceberg for a while now. I'm looking forward to your thoughts! Best regards, Sergei Nikolaev
