OIiveirra opened a new pull request, #68985:
URL: https://github.com/apache/doris/pull/68985
### What problem does this PR solve?
Issue Number: N/A
Related PR: #68276 (supersedes its metadata-scanner fix; this PR targets
master)
Problem Summary:
Iceberg metadata table timestamps are encoded as epoch microseconds. The JNI
scanner converts them through milliseconds, losing fractional precision. It
also applies the session time zone to timestamp-without-zone partition values,
changing their wall-clock value.
For example, a `$files.partition` timestamp of `1234567` microseconds is
truncated to `1970-01-01 00:00:01.234000` rather than `1970-01-01
00:00:01.234567`. Switching from UTC to Asia/Shanghai incorrectly shifts
timestamp-without-zone values by eight hours.
This PR:
- Decodes signed epoch microseconds with floor division/modulus, preserving
pre-epoch fractions.
- Propagates Iceberg types through struct/list/map values to distinguish
wall-clock timestamps from instants.
- Resolves requested field types by name while continuing to read projected
rows by position.
- Adds unit tests and a `$files.partition` regression covering `-1000001`,
`-1`, `1`, `999`, `1000`, `1234567`, and NULL, UTC/Asia/Shanghai, both
timestamp mapping settings, and a projected partition child.
Scope is the Java Iceberg metadata scanner. This PR does not modify regular
data-table timestamp reads or the FE/native `$position_deletes` serialization
path. There are no storage-format or configuration changes.
### Release note
Fix microsecond precision loss and incorrect session-time-zone shifts for
timestamp-without-zone values in Iceberg metadata tables.
### Check List (For Author)
- Test
- [x] Regression test
- `test_iceberg_files_timestamp_partition`: generated the golden
output, checked its timestamp values, and reran normal comparison successfully
(1 suite, 0 failures/skips).
- `test_iceberg_system_table_projection`: passed.
- `test_iceberg_position_deletes_sys_table`: passed.
- [x] Unit Test
- `IcebergSysTableColumnValueTest`: 2 tests passed; the
FE/Java-extension reactor completed successfully.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Local External tests used an isolated Iceberg fixture stack and the newly
packaged metadata-scanner plugin deployed to the personal Doris BE. The full
FE/BE deployment build was interrupted during C++ submodule downloads; FE/BE
executables were not rebuilt for these runs. These are targeted local results,
not full community External/P0/coverage results.
- Behavior changed:
- [ ] No.
- [x] Yes. Metadata timestamps retain microseconds;
timestamp-without-zone values no longer shift with the session time zone.
Timestamp-with-zone values preserve their instant semantics.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
