jacktengg opened a new pull request, #64123:
URL: https://github.com/apache/doris/pull/64123
### What problem does this PR solve?
Problem Summary:
When a table function (explode/unnest/lateral view) sits below a projection
that prunes away the exploded value and/or the parent (child) columns, the
TABLE_FUNCTION operator still materialized full-width output columns that are
immediately discarded by the projection. For the benchmark `SELECT COUNT(*)
FROM (SELECT t.id, u.unnest FROM t_explode t, unnest(t.arr) AS u(unnest) WHERE
t.id > 100)` over 500k rows x 1234-element arrays (~616M output rows), this
operator dominated runtime.
This change adds several pruning-aware fast paths, all driven by FE's
existing `outputSlotIds` pruning (`_slot_need_copy()` / `_output_slot_indexs` /
`_useless_slot_indexs`):
1. Block fast path, partial prune (`_get_expanded_block_block_fast_path`):
- `need_row_ids`: skip building the `seg_row_ids` replication vector
entirely when no child column needs replicating (it was built and never used).
- `materialize_values`: when the table-function value is not read
downstream, skip copying the nested values / null map (`insert_range_from`).
- `tf_value_as_const`: when the pruned value column also has a downstream
projection (`has_output_row_desc()`), don't grow it per-row with
`insert_many_defaults()` (memsetting a full nullable column over hundreds of
millions of rows); leave it untouched during the walk and size it once after
the loop with an O(1) `ColumnConst` placeholder.
2. Count-only path, full prune (`_output_no_columns_eligible` ->
`_get_expanded_block_no_columns` / `_count_only_fast_path`): for queries such
as `SELECT COUNT(*)` where every output slot is pruned and the only consumer is
a constant projection (FE adds `final projections: 1`), skip building the full
`_output_slots` block and emit a single lightweight UInt8 `__tf_count__`
placeholder column carrying just the expanded row count. Counting is
independent of nested contiguity, so it covers explode / explode_outer too.
3. Pruned "useless" child columns (the lateral-view input `arr` in `SELECT
COUNT(unnest) FROM t, unnest(t.arr)`): represent them with an O(1)
`ColumnConst` (`mock_column_size`) instead of `insert_many_defaults()`, instead
of allocating/memsetting large array-offset buffers only to be discarded by the
projection.
All `ColumnConst` placeholders are guarded by `has_output_row_desc()`, so
the const lives only in the internal `_origin_block` consumed by
`do_projections()` and never propagates downstream (where
`check_type_and_column()` or a downstream `assert_cast` would reject it). The
shared `mock_column_size` helper is hoisted from
`process_hash_table_probe_impl.h` into `column_const.h` so both hash join and
the table function use the same canonical pattern.
Because the block fast path is chosen per child block, a fast batch could
leave the reused table-function output column as a `ColumnConst` that a later
slow batch would feed to `get_value()` (`assert_cast<ColumnNullable*>`),
crashing. To preserve the slow path's invariant, `get_expanded_block`
normalizes any const table-function output column back to a fresh empty mutable
column at the start of each batch; the fast path re-derives the const each
batch.
posexplode and the slow (row-wise) path are unchanged for value
materialization; posexplode always materializes its struct output and never
takes the const path.
### Release note
None
### Check List (For Author)
- Test: Regression test
-
regression-test/suites/query_p0/sql_functions/table_function/explode_unnest_prune.groovy
covers fully-pruned COUNT(*), partial-prune (value/id pruned), value
materialized via filter/aggregate, posexplode, and explode_outer.
- Behavior changed: No (pure performance optimization; result-preserving)
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
