talatuyarer opened a new pull request, #16268:
URL: https://github.com/apache/iceberg/pull/16268
I found a user-visible chain issue/bug.
1. If user defines a partition spec with `identity(ts_nano)` or
`truncate(N, ts_nano)`.
2. IcebergSink enables range distribution, calls
`Partitioning.sortOrderFor(spec)` to derive a sort order.
3. The sort order has a `TIMESTAMP_NANO` field.
4. Flink shuffles records → serializes the sort key via
`SortKeySerializer` → boom, `UnsupportedOperationException`.
In addition to that user case:
1. User configures a Flink streaming source watermark on a
`TIMESTAMP_NANO` column.
2. `ColumnStatsWatermarkExtractor` constructor → boom,
`IllegalArgumentException` because column type not in the allowlist.
Both are now fixed, and each fix has a test that would have failed against
the old code. Patched Flink v2.1 nanosecond gaps in `SortKeySerializer` and
`ColumnStatsWatermarkExtractor`, and added partition-transform tests.
fyi @pvary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
