[
https://issues.apache.org/jira/browse/SPARK-60703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60703:
-----------------------------------
Labels: pull-request-available (was: )
> Make the broadcast value projection for DPP work under AQE when the broadcast
> plan has no shuffle
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-60703
> URL: https://issues.apache.org/jira/browse/SPARK-60703
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Peter Toth
> Priority: Major
> Labels: pull-request-available
>
> SPARK-58265 lets an {{onlyInBroadcast}} DPP filter take its values from the
> rows of an earlier broadcast
> ({{spark.sql.optimizer.dynamicPartitionPruning.broadcastProjection.enabled}},
> off by default). Under AQE this does not happen when the plan under that
> broadcast has no shuffle. That is the usual case, for example a filtered
> scan. The DPP filter is planned as {{true}} instead, so the query gets no
> pruning, as before SPARK-58265. Results are not affected.
> Cause:
> * {{PlanAdaptiveDynamicPruningFilters.projectedBroadcast}} plans the source
> with {{QueryExecution.prepareExecutedPlan(projection.sourcePlan, context)}}.
> It goes on only when it gets an {{AdaptiveSparkPlanExec}} back.
> * {{InsertAdaptiveSparkPlan.apply}} passes {{isSubquery = false}}. So a plan
> with no exchange, no distribution requirement and no subquery comes back
> unwrapped, and {{projectedBroadcast}} returns {{None}}.
> The tests miss it:
> * {{EnableAdaptiveExecutionSuite}} sets
> {{spark.sql.adaptive.forceApply=true}}, so the {{AEOn}} suites always wrap
> the plan.
> * The V2 projection tests in
> {{DataSourceV2EnhancedRuntimePartitionFilterSuite}} run with AQE off.
> With {{forceApply = false}} in {{EnableAdaptiveExecutionSuite}}, all 3
> projection tests of {{DynamicPartitionPruningV1SuiteAEOn}} fail. The plan has
> no {{ProjectedBroadcastValueSubqueryExec}}.
> Found while working on SPARK-60702.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]