Peter Toth created SPARK-60758:
----------------------------------
Summary: Require the same broadcast mode when a DPP filter reuses
a broadcast without AQE
Key: SPARK-60758
URL: https://issues.apache.org/jira/browse/SPARK-60758
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: Peter Toth
Without AQE, {{PlanDynamicPruningFilters.reusableBroadcast}} keeps an
{{onlyInBroadcast}} DPP filter when a broadcast hash join broadcasts a plan
with the same result as the filter's build plan. For a direct reuse it compares
only the plans. It then builds a {{BroadcastExchangeExec}} keyed by the
filter's own build keys. When the join broadcasts that plan with another hash
mode, {{ReuseExchangeAndSubquery}} cannot share the two exchanges. So the build
plan runs and is broadcast twice. This goes against
{{spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly}}, which
applies DPP only where a broadcast can be reused. Results are not affected.
The hash mode differs when:
* the join broadcasts the plan on other keys;
* the join is a null-aware anti join, which broadcasts with {{isNullAware =
true}}.
Under AQE, {{PlanAdaptiveDynamicPruningFilters}} compares the broadcast
exchanges, mode included, and skips null-aware anti joins. There such a filter
is planned as {{true}}. Without AQE, the broadcast value projection of
SPARK-58265 also requires the same mode.
A scratch run with AQE off, where the same filtered plan is the build plan of a
DPP filter in one union branch and is broadcast by a join in the other:
* the join on the DPP filter's key: one broadcast exchange, which the DPP
filter reuses;
* the join on another key: two broadcast exchanges, the DPP filter builds its
own;
* a null-aware anti join: two broadcast exchanges.
With AQE on, the second case is planned as {{true}}.
A possible fix is to require the same mode for a direct reuse too, as the AQE
rule does. SPARK-60702 counts such a filter as kept, so its key gets no runtime
bloom filter. With this fix, it can count only a broadcast with the same mode.
Found while working on SPARK-60702.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
