Gabriel39 opened a new pull request, #65624:
URL: https://github.com/apache/doris/pull/65624
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: FileScannerV2 had several correctness, fallback-contract,
lazy-materialization, profiling, and repeated schema-work gaps. Scanner slot
ids could be mistaken for table-global ordinals, invalid Parquet dictionary ids
could silently reject rows, dictionary selection could request a clean fallback
after advancing the stream, filters that could not be localized still forced
eager reads, and every split counted all row groups in the file.
Equality-delete loading also rebuilt its schema-shaped block for each batch.
This change:
- rejects unresolved scanner slot ids instead of mixing identity namespaces;
- reports invalid Parquet dictionary ids as corruption;
- permits dictionary fallback only before any skip or read advances the
record reader;
- returns visible columns referenced only by unlocalizable filters to the
lazy non-predicate set;
- counts only split-owned row groups in pruning Profile counters;
- builds the equality-delete Block schema once and clones the empty template
per batch;
- adds focused unit coverage for the changed contracts.
### Related JIRAs
- DORIS-27067
- DORIS-27068
- DORIS-27069
- DORIS-27071
- DORIS-27081
- DORIS-27082
### Release note
Fix FileScannerV2 filter correctness and Parquet row-group profile
accounting.
### Check List (For Author)
- Test: Unit Test coverage added. `run-be-ut.sh` was attempted twice but
stopped before compilation because this worktree's `thirdparty/installed` is
missing `protoc` and Snappy.
- `build-support/clang-format.sh`
- `build-support/check-format.sh`
- clang-format v16 dry-run on all changed C++ files
- `git diff --check`
- Behavior changed: Yes. Corrupt dictionary ids and unresolved slot mappings
now fail explicitly; lazy materialization and split-scoped Profile counters are
corrected.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
