[
https://issues.apache.org/jira/browse/SPARK-60758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60758:
-----------------------------------
Labels: pull-request-available (was: )
> Require the same broadcast mode when a DPP filter reuses a broadcast without
> AQE
> --------------------------------------------------------------------------------
>
> Key: SPARK-60758
> URL: https://issues.apache.org/jira/browse/SPARK-60758
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Peter Toth
> Priority: Major
> Labels: pull-request-available
>
> Without AQE, {{PlanDynamicPruningFilters.reusableBroadcast}} keeps an
> {{onlyInBroadcast}} DPP filter when a broadcast hash join broadcasts a plan
> with the same result as the filter's build plan. For a direct reuse it
> compares only the plans. It then builds a {{BroadcastExchangeExec}} keyed by
> the filter's own build keys. When the join broadcasts that plan with another
> hash mode, {{ReuseExchangeAndSubquery}} cannot share the two exchanges. So
> the build plan runs and is broadcast twice. This goes against
> {{spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly}}, which
> applies DPP only where a broadcast can be reused. Results are not affected.
> The hash mode differs when:
> * the join broadcasts the plan on other keys;
> * the join is a null-aware anti join, which broadcasts with {{isNullAware =
> true}}.
> Under AQE, {{PlanAdaptiveDynamicPruningFilters}} compares the broadcast
> exchanges, mode included, and skips null-aware anti joins. There such a
> filter is planned as {{true}}. Without AQE, the broadcast value projection of
> SPARK-58265 also requires the same mode.
> A scratch run with AQE off, where the same filtered plan is the build plan of
> a DPP filter in one union branch and is broadcast by a join in the other:
> * the join on the DPP filter's key: one broadcast exchange, which the DPP
> filter reuses;
> * the join on another key: two broadcast exchanges, the DPP filter builds its
> own;
> * a null-aware anti join: two broadcast exchanges.
> With AQE on, the second case is planned as {{true}}.
> A possible fix is to require the same mode for a direct reuse too, as the AQE
> rule does. SPARK-60702 counts such a filter as kept, so its key gets no
> runtime bloom filter. With this fix, it can count only a broadcast with the
> same mode.
> Found while working on SPARK-60702.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]