OIiveirra opened a new pull request, #68276:
URL: https://github.com/apache/doris/pull/68276
[fix](iceberg) Preserve timestamp partition micros
### What problem does this PR solve?
Issue Number: N/A (internal: DORIS-29054, DORIS-29055)
Related PR: N/A
Problem Summary:
Iceberg partition timestamps are epoch microseconds. Splitting negative
values
with truncating division can produce an invalid nanosecond value during scan
planning. Iceberg `$files` metadata conversion also went through
milliseconds,
which lost microsecond precision and applied the session time zone to
timestamp-without-zone values.
Use floor-based microsecond decomposition for partition timestamp conversion
and metadata-table conversion. Timestamp-without-zone now preserves its UTC
wall-clock value; timestamp-with-zone remains converted from its instant in
UTC.
### Release note
Fix Iceberg timestamp partition reads around the Unix epoch and preserve
microsecond precision in metadata tables.
### Check List (For Author)
- Test
- [x] Unit Test
-
`IcebergUtilsTest#testPartitionTimestampSerializationPreservesNegativeMicros`
- `IcebergSysTableColumnValueTest`
- [x] Manual test
- Iceberg external `test_external_catalog_iceberg_partition` passed
with the personal Iceberg profile.
- `$files` external coverage requires the community pipeline because
the persistent personal BE uses a newer JNI plugin ABI than 4.1.
- Behavior changed:
- [x] Yes. Negative epoch-microsecond partition timestamps no longer
fail;
metadata timestamps retain microseconds and timestamp-without-zone is
not
shifted by the session time zone.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
