Gabriel39 opened a new pull request, #65674:
URL: https://github.com/apache/doris/pull/65674
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: FileScannerV2 Parquet scans currently pay avoidable
allocation and materialization costs around Arrow record batches, selection
handling, and nested-column reconstruction. The reader also needs stricter
contracts for nested projection, dictionary indices, logical-row selection, and
profiling before the physical decoder can be migrated away from Arrow safely.
This PR preserves nested projections during lazy demotion, tightens
predicate placement and profile accounting, introduces an Arrow-independent
physical column schema and a selection-aware flat decode contract in the Doris
Parquet kernel, batches string materialization, and retains scalar and
LIST/MAP/STRUCT scratch buffers across batches. The existing complex-reader API
remains a migration facade; the design document defines the target
shared-level-plan interface for the native decoder. The legacy v1 production
read path is not changed.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- Remote `./build.sh --be` with ASAN: passed
- Remote targeted Parquet BE unit tests: 120 passed
- clang-format 16 check: passed
- clang-tidy was attempted but the remote toolchain could not resolve
system `stddef.h` and also reported a pre-existing unmatched `NOLINTEND` in
`be/src/core/types.h`
- Behavior changed: No. This adds migration contracts, correctness
hardening, profiling, and internal allocation optimizations.
- Does this need documentation: Yes. Updated the FileScannerV2 Parquet
design document and the format_v2 review guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
