eldenmoon opened a new pull request, #66310:
URL: https://github.com/apache/doris/pull/66310
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
A future Iceberg external-table Variant rewrite needs `select
v['metric']['x']` to materialize a dedicated Path Slot without reading or
returning it as an unrelated root slot. `FileScannerV2` previously dropped the
existing `TSlotDescriptor.column_paths`, so BE could not preserve the path, use
the Iceberg root field id, merge multiple logical paths into one physical root
request, or materialize each logical output.
This PR adds the BE mapping/materialization contract while deliberately
reusing existing descriptor fields:
- `TSlotDescriptor.col_unique_id` identifies the Iceberg Variant root field.
- `TSlotDescriptor.column_paths` (`list<string>`) carries exact object-key
segments.
- No `TVariantPathSlot`, new Thrift field, or gensrc change is introduced.
- Path Slots sharing one root field id share one file-local root carrier;
predicate/output requests union their paths.
- `TableReader` extracts each logical Path Slot with the existing Variant V2
`element_at` kernel during finalization.
- Field id is authoritative across schema evolution; a missing id does not
fall back to a reused name.
- Legacy `ColumnVariant`, non-field-id mapping, and non-nullable Path Slots
fail fast.
Important scope boundary:
- This is the BE Path Slot mapping and materialization contract, not
complete physical typed-stream I/O pruning.
- The production Format V2 Parquet reader does not yet consume
`LocalColumnIndex.variant_paths`. Until a Variant-aware decoder is added, it
may read the full root.
- That future reader must use `variant_paths` to select typed streams and
reconstruct an encoded, possibly partial, root `ColumnVariantV2` at the shared
root local position. Returning a direct typed leaf would violate the contract
because `TableReader` performs the logical path extraction.
- `column_paths` entries are exact object keys. Array indices are
intentionally unsupported because `list<string>` cannot distinguish index `0`
from object key `"0"`; the future FE external rewrite must leave integer
`element_at` expressions unreplaced.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `GLIBC_COMPATIBILITY=OFF ./run-be-ut.sh --run
--filter=ColumnMapperTest.VariantPathSlotsShareOneRootCarrierAndFinalizeOnly:ColumnMapperTest.VariantPathSlotsRejectLegacyAndNonFieldIdMappings:TableReaderTest.MaterializesTwoVariantPathSlotsFromSharedEncodedRootCarrier:TableReaderTest.MissingVariantPathRootMaterializesSqlNulls:TableReaderTest.MissingVariantPathRootExtractsFromDefaultCarrier:FileScannerV2Test.BuildsVariantPathColumnFromExistingSlotFields:FileScanRequestBuilderTest.MergesVariantPathsAcrossPredicateAndOutputSlots:TableReaderTest.AnnotateVariantPathSlotFindsRootByFieldId:TableReaderTest.AnnotateVariantPathSlotDoesNotFallbackAfterMissingFieldId:LocalColumnIndexTest.MergeUnionsPartialChildrenAndFullProjectionDominates`
- 10/10 focused BE ASAN unit tests passed.
- `build-support/clang-format.sh`, `build-support/check-format.sh`, and
`git diff --check` passed.
- Targeted clang-tidy checks on changed-line findings passed; the full
repository script was limited by the local 30-minute execution window.
- Behavior changed: Yes (adds the BE mapping/materialization contract for
future external Variant Path Slots; existing root slots retain their path-empty
behavior)
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
