xylaaaaa opened a new pull request, #65560:
URL: https://github.com/apache/doris/pull/65560
Issue Number: None
Related PR: None
Problem Summary: COUNT(nullable complex column) over Parquet or ORC file
TVFs cannot use COUNT(*) row metadata because top-level NULL values must be
excluded. With FileScannerV2, the query therefore decoded the full complex
column through Arrow and then materialized Doris columns, retaining decoded
binary chunks and causing excessive peak memory or intermittent OOM. This
change introduces an explicit COUNT_NON_NULL operation and target slot from FE
to BE. Parquet counts top-level shape from bounded definition/repetition-level
batches without Arrow builders or Doris value columns, while ORC uses non-null
stripe statistics. It also emits valid non-NULL synthetic target rows and
releases file-block ownership before mutable output conversion to avoid
unnecessary COW copies.
Reduce memory usage for COUNT on nullable columns read from Parquet and ORC
file TVFs with FileScannerV2.
- Test: Unit Test
- FE PhysicalStorageLayerAggregateTest: 4 tests passed
- BE targeted Parquet, ORC, and TableReader tests: 14 tests passed
- HDFS TVF regression assertion added but not run locally because this
worktree has no Doris cluster or external HDFS test environment
- Behavior changed: Yes. Eligible nullable file-column counts use explicit
COUNT_NON_NULL pushdown instead of materializing the full column.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
