morningman opened a new pull request, #69016:
URL: https://github.com/apache/doris/pull/69016
### What problem does this PR solve?
Issue Number: None
Related PR: #66575, #67022, #66413, #67675
Problem Summary:
**In short.** This ports branch-4.1 #66575 and its follow-up #67022 to
master. The native Parquet reader of File Scanner V2 can now read a deep
Variant object path such as `payload['profile']['address']['zip']` from its
shredded leaf instead of rebuilding the whole Variant, and it decides this per
row group instead of once per file. On the FE side, `explode` over an ordinary
Variant keeps the whole input container in its access path, while a NestedGroup
Variant keeps the exact leaf paths that NestedGroup storage needs.
**Background.** A shredded Parquet Variant stores each shredded field as a
wrapper with a `typed_value` (the field in its own Parquet column) and a
`value` (a binary residual for rows whose value does not fit the shredded
type). FE sends BE the paths a query reads as lists of segments, and BE can
then read only the leaves of those paths. On master that projection is limited:
- `column_mapper` projects only a single top-level key, so a path of two or
more object keys reads the whole Variant root and rebuilds every row.
- `AccessPathParser` has a separate branch for a root Variant that sorts and
flattens its paths instead of merging them in the shared access-path tree.
- `ParquetReader::open` keeps a projection only when every `value` along the
path is NULL in the whole file; otherwise it rewrites the request to the full
column for every row group of the file.
- `AccessPathPlanCollector` narrows the input of `explode(v['arr'])` to the
fields read from the output (`[v, arr, x]`). A generator can change the
container kind of a Variant, which such a path cannot express; once BE projects
multi-segment paths, the narrowed path could hide residual values of those
rows. That is why #66575 changes FE and BE together.
**The problem, and what it cost.**
| Reading `payload['profile']['address']['zip']` and `['rank']` from a
shredded Paimon table with the native reader | master | with this PR |
|---|---|---|
| Access paths FE sends | `[payload.profile.address.rank,
payload.profile.address.zip]` | the same |
| What BE reads | the whole `payload` Variant, rebuilt row by row:
`VariantLeafProjections 0`, `VariantReconstructedRows 2` on the 2-row fixture |
the `address` leaves: `VariantLeafProjectionRowGroupColumns 2`,
`VariantFullProjectionRowGroupColumns 0`, `VariantReconstructedRows 0` |
| A file where one row group has residual values on the path | every row
group of the file reads the whole Variant | only that row group does |
**How this PR fixes it.**
- BE, `access_path_parser.cpp`: a root Variant goes through the generic
`AccessPathNode` tree, which merges sibling and prefix paths (`[profile]`
absorbs `[profile, city]`); a `project_all` node clears the Variant paths.
- BE, `column_mapper.cpp`: `build_variant_leaf_path_projection` follows
object paths of any depth. An empty path, a missing file column or a numeric
segment (an array index; TODO: typed path segments) still reads the whole
Variant, and every intermediate `typed_value` must be a STRUCT, so a path
cannot cross a repeated node.
- BE, `parquet_reader.cpp` / `parquet_scan.cpp`: `open` only counts the
candidate projections (`VariantLeafProjections`). Each row group decides on its
own: the projection is kept when the `value` beside each projected terminal
leaf is all NULL in that row group, otherwise that row group reads the complete
Variant. The decision is made when the row group plan is built
(`prepare_row_group_physical_projection`) and the scheduler opens the row group
with the resulting physical request; footer pruning and the filtered-bytes
statistics use the leaf set of each row group. The reader and the scheduler now
share one `RowGroupScanPlan`. Two counters report the outcome per opened
column: `VariantLeafProjectionRowGroupColumns` and
`VariantFullProjectionRowGroupColumns`.
- BE, `variant_column_reader.cpp`: when a projected Variant is asked for a
complex prefix of a validated leaf (a binary `element_at` chain), it serializes
only the retained descendants, so a field that was projected out never becomes
visible.
- FE, `AccessPathPlanCollector`: `explode` / `explode_outer` over an
ordinary Variant collects the whole input container, for both the unary and the
multi-argument form (#66575). Over a NestedGroup Variant it keeps the exact
suffixes (#67022).
Two places differ from branch-4.x on purpose:
- Statistics pruning keeps master's rule from #66413: a shredded leaf's
min/max prunes a row group or page only when the `value` of every wrapper on
the path is all NULL (`ShreddedVariantAncestorFallbackDisablesPruning`).
branch-4.x only checks the terminal `value`; the three `parquet_statistics.cpp`
hunks of #66575 that switch to that rule are not included. The tests of #66575
pass under master's rule.
- master #67675 makes `element_at` keep only the max sub-column count of its
Variant type, so `v['arr']` no longer carries the NestedGroup flag. A straight
port of #67022 would send `[v, arr]` instead of `[v, arr, x]` for a NestedGroup
column whenever the explode argument is still an `element_at` (sub-path pruning
disabled, or the load path). `isNestedGroupVariant` follows the `element_at`
chain to the column it reads instead, and
`testExplodeNestedGroupVariantAccessPathsWithoutSubPathPruning` covers that
case.
The residual-column pruning that branch-4.x added on top of these files
later (#66941) is not part of this PR.
**Results.**
- BE UT: the six test files of #66575 pass, 376 tests
(`AccessPathParserTest`, `ColumnMapper*`, `NewParquetReaderTest`,
`FileReaderTest`, `ParquetVariantProjectionTest`, `ParquetScan*`,
`ParquetRuntimeFilterDirectReaderTest`, `NativeParquetStatisticsTest`,
`ParquetBloomFilterPruningTest`, `VariantColumnReaderTest`). Limiting the leaf
projection to one segment again fails
`ColumnMapperTest.VariantDeepObjectPathProjectsPhysicalTypedLeaf`; making every
row group read the complete Variant fails 5 tests
(`SwitchesVariantLeafProjectionPerRowGroup`,
`ReadsFullyShreddedVariantTypedLeafProjection` and three
`ParquetVariantProjectionTest` cases).
- FE UT: `PruneNestedColumnTest` (71) and `VariantPruningLogicTest` (17)
pass. With master's `AccessPathPlanCollector`, exactly 12 fail (the 11 changed
expectations and `testMultiArgumentExplodePreservesVariantContainers`); without
the NestedGroup check, the 3 NestedGroup cases fail; a straight port of #67022
fails only `testExplodeNestedGroupVariantAccessPathsWithoutSubPathPruning`,
which passes on master.
- Regression: `test_paimon_catalog_variant` and `test_iceberg_variant_read`
pass against the iceberg docker environment, and their `.out` files were
generated by the script. The Paimon suite's profile check (leaf projections >
0, full projections = 0) also shows that the docker Spark with Paimon 1.3.1
writes the three-level shredding schema of `run13.sql`.
Original authors: @Gabriel39 (#66575), @eldenmoon (#67022)
### Release note
Iceberg and Paimon native Parquet scans read deep shredded Variant object
paths from their leaves, deciding per row group, instead of rebuilding the
whole Variant.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `test_paimon_catalog_variant` (deep projection results and profile
counters on the shredded fixture) and `test_iceberg_variant_read` (a deep path
on an unshredded table); `.out` files generated by the script.
- [x] Unit Test
- BE: the six test files of #66575 (376 tests). FE:
`PruneNestedColumnTest`, `VariantPruningLogicTest`, including a NestedGroup
case with sub-path pruning disabled.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- Deep shredded Variant object paths are read from their leaves, per
row group. `VariantLeafProjections` now counts candidate projections, and two
new counters report the per-row-group outcome.
- `explode` over an ordinary Variant column reads the whole input
container instead of the output-field leaves.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
