[
https://issues.apache.org/jira/browse/SPARK-58063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolina Vraneš updated SPARK-58063:
------------------------------------
Description:
A Project over BinBy is already column-pruned by the generic Project(_, child)
fallback in ColumnPruning: BinBy is not intercepted by
GeneratorNestedColumnAliasing or NestedColumnAliasing (it is in neither
canProjectPushThrough nor canPruneOn), so it falls through to the fallback,
which trims the unused pass-through columns while keeping the range and
DISTRIBUTE inputs the kernel reads (BinBy.references lists exactly those, given
the produced-attributes shape from SPARK-57858).
An earlier revision added a dedicated ColumnPruning arm for Project over BinBy;
it was found redundant with the generic fallback (identical pruning, same
guard) and removed. This subtask now only adds test coverage locking in that
BinBy is pruned, plus a binBy test DSL helper. No production optimizer change.
Subtask of SPARK-57133.
was:
Add Generate-style column pruning to the BinBy logical operator: an
unrequiredChildIndex field plus a requiredChildOutput lazy val (mirroring
Generate), and a ColumnPruning arm for Project over BinBy that trims forwarded
pass-through columns while keeping the range and DISTRIBUTE inputs the kernel
reads. Enabled by the produced-attributes shape from SPARK-57858. Subtask of
SPARK-57133.
Summary: Test that BIN BY is column-pruned by the generic ColumnPruning
fallback (was: Add column pruning for BIN BY)
> Test that BIN BY is column-pruned by the generic ColumnPruning fallback
> -----------------------------------------------------------------------
>
> Key: SPARK-58063
> URL: https://issues.apache.org/jira/browse/SPARK-58063
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Nikolina Vraneš
> Priority: Major
> Labels: pull-request-available
>
> A Project over BinBy is already column-pruned by the generic Project(_,
> child) fallback in ColumnPruning: BinBy is not intercepted by
> GeneratorNestedColumnAliasing or NestedColumnAliasing (it is in neither
> canProjectPushThrough nor canPruneOn), so it falls through to the fallback,
> which trims the unused pass-through columns while keeping the range and
> DISTRIBUTE inputs the kernel reads (BinBy.references lists exactly those,
> given the produced-attributes shape from SPARK-57858).
> An earlier revision added a dedicated ColumnPruning arm for Project over
> BinBy; it was found redundant with the generic fallback (identical pruning,
> same guard) and removed. This subtask now only adds test coverage locking in
> that BinBy is pruned, plus a binBy test DSL helper. No production optimizer
> change. Subtask of SPARK-57133.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
