talatuyarer opened a new pull request, #27770:
URL: https://github.com/apache/flink/pull/27770
## What is the purpose of the change
Currently, the Flink Avro format support for `TIMESTAMP` and
`TIMESTAMP_WITH_LOCAL_TIME_ZONE` is capped at microsecond precision (6).
Standard Apache Avro (up to 1.11) only defines logical types for
`timestamp-millis` and `timestamp-micros`.
When Flink tables utilize **nanosecond precision (`TIMESTAMP(9)`)**, there
is no native Avro logical type mapping, leading to either fallback behaviour
(treating as raw `BIGINT`) or unsupported type exceptions during automatic
schema conversion.
This pull request adds support for **nanosecond-precision logical types**
(`timestamp-nanos` and `local-timestamp-nanos`) in the Flink Avro format. It
also upgrades **Avro to version 1.12.1** and enables **decimal logical types**
handling for improved compatibility and numeric fidelity.
Specifically:
- Supports mapping Flink's `TIMESTAMP(9)` and
`TIMESTAMP_WITH_LOCAL_TIME_ZONE(9)` back-and-forth with Avro's
`timestamp-nanos` and `local-timestamp-nanos` logical types.
- Enables high-precision temporal calculations without losing precision at
microsecond or millisecond boundaries.
## Brief change log
- **`AvroToRowDataConverters.java`**:
- Replaced the static timestamp converter with a precision-aware
`createTimestampConverter(int precision)` that handles millisecond,
microsecond, and nanosecond cases into `TimestampData`.
- **`RowDataToAvroConverters.java`**:
- Updated conversion for `TIMESTAMP_WITHOUT_TIME_ZONE` and
`TIMESTAMP_WITH_LOCAL_TIME_ZONE` to respect higher precision values up to 9
(nanoseconds) when packing epoch values.
- **`AvroSchemaConverter.java`**:
- Mapped `LogicalTypes.timestampNanos()` and
`LogicalTypes.localTimestampNanos()` schemas into equivalent Flink SQL styles
(`TIMESTAMP(9)` / `TIMESTAMP_WITH_LOCAL_TIME_ZONE(9)`).
- Updated conversion from Flink data types to Avro schema to support
precision up to 9 for timestamp types.
- **`pom.xml`**:
- Upgraded Avro dependency to **1.12.1**.
## Verifying this change
This change added tests and can be verified as follows:
- Updated unit tests within `AvroSchemaConverterTest.java` (e.g., asserting
fields like `type_timestamp_nanos`, `type_local_timestamp_nanos` map correctly).
- Ran pre-existing Avro suite (`AvroOutputFormatTest`,
`AvroTypeExtractionTest`) to ensure backwards compatibility with upgraded
baseline versions.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **yes** (Upgraded
Avro to 1.12.1)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **yes**
(Adjusts converter inner-loops, standard for type feature additions)
- Anything that affects deployment or recovery: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **yes**
- If yes, how is the feature documented? **not applicable** (Automatic
type mapping addition, validated by updated test cases)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
