Gabriel39 opened a new pull request, #68934:
URL: https://github.com/apache/doris/pull/68934
### What problem does this PR solve?
Iceberg `timestamp_ns` was unsupported, and Paimon `timestamp(9)` was
exposed as `DATETIMEV2(6)`, losing the final three fractional digits. Map these
unzoned types bidirectionally to Doris `TIMESTAMP_NS` and preserve all nine
digits through reads, inserts and CTAS, including arrays, maps and structs.
- Add nanosecond materialization to Parquet/ORC readers and Iceberg writers.
Nanosecond Iceberg schemas use Parquet 2.6; ORC uses a UTC wall-clock carrier
to avoid historical timezone-offset differences at the signed nanosecond
boundaries.
- Support Iceberg identity, calendar and bucket partitions, including
partition-value commit serialization. Omit unsafe ORC timestamp bounds and
derived MTMV RANGE boundaries; Iceberg's own partition pruning remains
available.
- Select Paimon's JNI reader for projected or filter-only high-precision
fields, including nested fields, and use nanosecond Arrow transport for writes.
Paimon precisions 7 and 8 also map to `TIMESTAMP_NS`.
Iceberg nanosecond schemas require format version 3. High-precision Paimon
reads use JNI, which may cost more than native scans. The existing ORC-645
rejection of ambiguous pre-epoch fractions is retained; Parquet supports those
values.
### Release note
Support reading and writing Iceberg `timestamp_ns` and Paimon `timestamp(9)`
as Doris `TIMESTAMP_NS`, preserving nanosecond precision.
### Validation
- Full FE and ASAN BE builds; FE Checkstyle, clang-format 16 and clang-tidy
over all 29 changed C++ files (initial findings fixed and rechecked).
- 2,112 relevant Java tests: zero failures, five existing skips.
- 117 relevant BE tests under ASAN, including independent Arrow/ORC file
read-back, signed bounds, dictionary decoding and overflow handling.
- Three end-to-end regression suites passed against the generated goldens:
`test_iceberg_timestamp_ns`, `test_paimon_timestamp_ns`, and
`paimon_timestamp_types`. Coverage includes both scanners, session time zones,
nested values, filters, signed bounds, CTAS, Iceberg partition
transforms/overwrite, and Paimon primary-key upsert.
- Independent Flink read-back of four Doris-written Paimon tables: 28 rows
across Parquet/ORC and append/primary-key tables, with exact nanosecond bounds
and upserts.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (independent Flink read-back described above)
- Behavior changed:
- [x] Yes. External unzoned nanosecond timestamps retain nine fractional
digits.
- Does this need documentation?
- [x] Yes. The external type-mapping tables should describe
`TIMESTAMP_NS`.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
