Gabriel39 opened a new pull request, #68441:
URL: https://github.com/apache/doris/pull/68441
### What problem does this PR solve?
Related PR: #67784. This is the third split, following #68297 and #68381,
targeting `branch-4.1`.
Parquet reads and writes can conflate wall-clock timestamps with UTC
instants. ORC decoding truncates nanoseconds while pruning and writer
statistics use different precision, which can shift values or discard matching
rows.
- Keep unannotated INT96 as DATETIMEV2 and carry an explicit
`hive.parquet.time-zone` contract through schema inference, scanning,
nested/filter-only projections and Hive writes. An unset or empty property
selects wall-clock INT96; a named zone decodes values normalized in that zone.
Scanner selection and condition-cache keys preserve this interpretation.
- Default Parquet OUTFILE/EXPORT to INT64 logical timestamps: DATETIMEV2 has
`isAdjustedToUTC=false`, while TIMESTAMPTZ retains instant semantics. Keep
explicit `enable_int96_timestamps=true`, validate its boolean value, and pass
it through EXPORT. Hive writes retain INT96.
- Round ORC nanoseconds to microseconds consistently, distinguish civil-time
carry from instant carry at DST transitions, and reject invalid nanoseconds or
years before narrowing into packed storage. Align aggregate statistics,
conservative SARG bounds and writer bounds; reject the pre-epoch fractional
interval that ORC cannot write losslessly.
This split preserves existing external type-mapping defaults and leaves
table-specific Iceberg/Paimon bindings to the next split. It adds no binary
computation support.
### Release note
Parquet OUTFILE/EXPORT now defaults to INT64 logical timestamps; set
`enable_int96_timestamps=true` when INT96 output is required.
`hive.parquet.time-zone` explicitly controls INT96 normalization. ORC timestamp
rounding and pruning now agree at microsecond and DST boundaries; lossy
pre-epoch writes fail explicitly.
### Validation
- Built the official BE unit-test target with ASAN; all 622 selected tests
in 21 suites passed, covering converters, writers, readers, nested mappings,
pruning, condition caching, scanner selection and row-id reads.
- All 43 selected FE tests in eight classes passed.
- FE Checkstyle and clang-format 16 checks passed for all affected files.
- Parsed all 21 changed regression Groovy scripts successfully.
- External SQL/Hive/HDFS regressions were not executed locally; full
regression validation is pending CI.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- Behavior changed:
- [x] Yes. Timestamp representation, interpretation and precision changes
are described above.
- Does this need documentation?
- [x] Yes. The INT64 default and `hive.parquet.time-zone` should be
reflected in the file-format documentation.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
