morningman opened a new pull request, #69872:
URL: https://github.com/apache/doris/pull/69872
### What problem does this PR solve?
Issue Number: close #69858
Related PR: None
Problem Summary:
The shared Arrow BOOLEAN and DECIMAL SerDes dereference unchecked
`dynamic_cast` results. Passing an `Int32Array` to the BOOLEAN reader or a
`StringArray` to the DECIMAL64 reader can therefore terminate the BE instead of
returning a load error. MAP and STRUCT have the same unchecked input-cast
pattern.
Reject incompatible arrays before accessing buffers, decimal metadata, or
values, independently of `enable_arrow_input_validation`. Reject mismatched
STRUCT field counts before indexing children. Add the column name and
source/destination types to V1/V2 Arrow load errors.
The current V2 Stream Load reader already binds supported input types with
safe Arrow casts. Preserve those conversions, including valid
integer-to-BOOLEAN and string-to-DECIMAL inputs, and test the shared SerDes
directly for incompatible arrays.
### Release note
Reject incompatible Arrow arrays with an error instead of dereferencing a
null pointer, and include column and type details in Arrow Stream Load
conversion errors.
### Check List (For Author)
- Test
- [x] Regression test: the new `test_arrow_stream_load_type_validation`
matrix and existing `test_arrow_stream_load_uppercase_column` suite passed with
`enable_arrow_input_validation=true` and `false`, restarting the isolated BE
between configurations.
- [x] Unit Test: 65 ASAN tests passed across
`DataTypeSerDeArrowValidationTest`, the scalar/collection Arrow round-trip
controls, and `LoadArrowPipeTest`, including eight new named tests.
- [ ] Manual test
- [ ] No need to test or manual test.
Local validation used macOS aarch64 and a cluster built from this
checkout. The new HTTP matrix runs 114 loads across BOOLEAN,
DECIMAL32/64/128/256, ARRAY, MAP, and STRUCT with nullable/required targets. It
covers valid conversions, invalid values/container types, nested NULLs, decimal
precision/scale failures, empty/multiple batches, and failure after a preceding
valid IPC stream. All 63 failed-load visibility checks returned zero rows, and
subsequent correct loads succeeded on the same BE with HTTP retries disabled.
The 36 IPC fixtures have a reproducible PyArrow generator; the regression suite
does not require Python. The `.out` file was generated by the regression runner
and then verified in comparison mode.
FE build, BE ASAN build, clang-format v16, build hygiene, and clang-tidy
checks passed.
- Behavior changed:
- [ ] No.
- [x] Yes. Incompatible SerDe array types and STRUCT field counts return
`InvalidArgument`. Supported V2 safe casts are preserved.
- Does this need documentation?
- [x] No. This fixes invalid-input handling and error context without
adding syntax or configuration.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
