hubgeter opened a new pull request, #66941:
URL: https://github.com/apache/doris/pull/66941
### What problem does this PR solve?
Problem Summary:
Reading a shredded Iceberg Variant was 2.16x slower than reading the same
data unshredded. The shredded scan read 38% fewer bytes but spent all of it
back on Variant reconstruction: every requested leaf fell through
find_typed_value() to materialized_column(), so VariantReconstructedRows
equalled RawRowsRead for the whole scan.
find_typed_value() gave up in three situations it did not have to:
1. A STRING leaf returned nullopt whenever the state was complete, even
though normalize_projected_primitive_leaf() already reconstructs exactly that
leaf from its ParquetColumnSchema and is used for the same types once a leaf
projection exists. Its result is byte-identical to rebuilding the canonical
root and extracting the leaf, so the complete-state guard only bought a full
reconstruction.
2. A residual beside any ancestor on the path rejected the whole batch.
Reaching a path segment proves that key belongs to the shredding schema, and
shredding keeps an object's residual keys disjoint from its shredded fields, so
an ancestor residual can never supply the requested path. This already is the
invariant collect_variant_terminal_fallback_leaf_ids() relies on when it
validates row groups.
3. A residual anywhere on the path rejected the whole batch even when a
single row carried it. Rows are now resolved individually: the first null typed
value on the path hands the row to the encoded residual beside it, everything
else keeps the shredded leaf. Rejecting 8192 rows because one of them is stored
unshredded is what kept q08 on the slow path even for its BIGINT leaf.
Walking the path per row also closes a correctness gap. find_typed_value()
never inspected the root `value`, so a row whose root typed_value is null and
whose complete Variant lives in the root residual reported the path as absent,
while the materialized fallback returned its real value. The root is now level
0 of the same walk and resolves like every other level.
VariantDirectLeafResidualMergedRows reports how many rows took the merged
path.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
