Gabriel39 opened a new pull request, #68396:
URL: https://github.com/apache/doris/pull/68396
### What problem does this PR solve?
Related PR: #68381. This is the master version of the second split from
#67784, based on the primitives merged in #68301.
Arrow batch conversion mixes protocol serialization with table-specific UUID
handling, while writers construct schemas separately. Introduce explicit Doris,
Python, Arrow Flight, Parquet, Hive, Iceberg and Paimon convertors with
instance-owned schema parameters and timezone. Move schema
construction/decoding into the convertors and route nested SerDe writes through
the selected format.
Separate Parquet, Hive and Iceberg writers and migrate existing callers.
Preserve master's tracked Arrow memory pools, Iceberg statistics and
timestamp-nanosecond support. Master does not yet contain the Paimon write
backend or physical Variant table writes present on branch-4.1; this pick adds
the converter interfaces without importing those features. Parquet timestamp
encoding and external type mappings remain unchanged.
Include the Python timezone regression correction from #68381: the single
string output uses ARRAY<STRING>, so the lateral-view comparison reaches
execution instead of failing on a STRUCT-versus-STRING comparison. Retain
coverage for four session timezones, microseconds, pre-epoch values, NULLs,
UDF, UDTF and UDAF.
### Release note
Fix Python UDF timestamp conversion to preserve wall-clock values when the
Arrow protocol declares a fixed-offset timezone.
### Check List (For Author)
- Test
- [x] Unit Test: explicit schemas and independent converter instances,
nested/null values, UUID and fixed binary bytes, timestamp bindings, and
Iceberg writer statistics.
- [x] Regression test: Python UDF/UDTF/UDAF timezone comparisons and the
existing timestamp snapshot corrections.
- Behavior changed:
- [x] Yes: align Python UDF conversion with its Arrow timezone
declaration; reject invalid nested bindings before casts.
- Does this need documentation?
- [x] No.
Validation: ASAN BE build and 310 focused tests passed (53 suites), covering
Arrow conversion, Parquet/ORC, Variant SerDe and Python. All 42 affected C++
files passed clang-format 16, and header hygiene passed. The original UDTF
declaration reproduced the SQL analysis error on an isolated FE; the corrected
declaration passed the same analysis. Groovy and embedded Python checks passed.
Full Python SQL and external-catalog regressions remain for CI. clang-tidy was
attempted: the new converter's size warning was resolved; analysis remains
blocked by a pre-existing unmatched NOLINTEND in core/types.h.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
