Gabriel39 opened a new pull request, #68604:
URL: https://github.com/apache/doris/pull/68604
### What problem does this PR solve?
Flight SQL can publish out-of-range timestamps from legacy INT96 date
materialization. For example, a non-null zero DATETIME becomes `timestamp[us]`
with value `-62169984000000000`: Arrow validation succeeds, but PyArrow scalar
conversion raises `OverflowError`.
Validate timestamps in `ArrowFlightArrowBlockConvertor` before publishing
each batch. Check the actual timestamp unit and both UTC/local calendar bounds
for zoned values. Recursively check arrays, map keys/values, and structs while
skipping values masked by a NULL parent. Reject unsupported timestamps with an
error identifying the field path and block row. Other Arrow consumers and
Parquet scan compatibility remain unchanged.
Related PR: #68596 adds separate UTF-8 validation. This PR is independent of
that change; both checks must be retained when integrating the shared Flight
conversion entry point.
### Testing
- Added seven BE tests covering zero dates, years 0/10000,
seconds/milliseconds/microseconds, valid calendar boundaries, pre-epoch
fractions, timezone offsets, slices/subsequent batches, nested timestamps, NULL
parents, and other consumers.
- Before the fix: four new test groups failed because malformed timestamps
were published; three valid/NULL groups passed.
- After the fix: all 36 tests in
`ArrowFlightTimestampTest.*:DataTypeSerDeArrowTest.*` passed under ASAN.
Recompiled the changed converter and new tests against existing ASAN BE
libraries.
- Added a Flight JDBC regression suite and a separate HDFS INT96 suite using
five existing group4 fixtures. The HDFS suite requires `enableHiveTest` and the
external test environment.
- Clang-format 16 and Groovy compilation passed. Live SQL/HDFS regression
execution is pending CI; no patched cluster was deployed locally.
### Release note
Flight SQL now returns a server-side error for timestamps outside the
supported 0001–9999 calendar range instead of publishing values that fail
client conversion. Valid timestamps and NULLs retain their existing values.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- Behavior changed:
- [x] Yes: reject out-of-range Flight timestamps before publishing a
batch.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
