ewoodbury commented on code in PR #2848:
URL:
https://github.com/apache/datafusion-ballista/pull/2848#discussion_r4582238614
##########
ballista/scheduler/src/state/aqe/optimizer_rule/distributed_exchange.rs:
##########
@@ -60,21 +61,18 @@ impl DistributedExchangeRule {
config: &datafusion::config::ConfigOptions,
) -> datafusion::error::Result<Transformed<Arc<dyn ExecutionPlan>>> {
// DataFusion's null-aware hash join coordinates visited rows and
- // probe-side NULL state with in-process atomics. A CollectLeft join
- // running once per probe partition therefore produces duplicate or
- // incorrect output. This is the final plan-mutating rule, so enforce
an
- // explicit probe-side coalesce here after DataFusion's optimizers have
- // finished. Add an exchange when the subtree has no existing boundary.
- if let Some(hash_join) = execution_plan.downcast_ref::<HashJoinExec>()
- && hash_join.null_aware
- && *hash_join.partition_mode() == PartitionMode::CollectLeft
- && hash_join
- .right()
- .downcast_ref::<CoalescePartitionsExec>()
- .is_none()
+ // probe-side NULL state with in-process atomics, and a nested-loop
join
+ // tracks matched build rows the same way. Such a join running once per
+ // probe partition therefore produces duplicate or incorrect output.
+ // This is the final plan-mutating rule, so enforce an explicit
+ // probe-side coalesce here after DataFusion's optimizers have
finished.
+ // Add an exchange when the subtree has no existing boundary.
+ if probe_needs_one_task(&execution_plan)
+ && let [left, right] = execution_plan.children().as_slice()
+ && right.downcast_ref::<CoalescePartitionsExec>().is_none()
{
- let left = hash_join.left().clone();
- let right = hash_join.right().clone();
+ let left = Arc::clone(left);
+ let right = Arc::clone(right);
let right = if right.downcast_ref::<ExchangeExec>().is_none()
Review Comment:
Done for small scans, a probe that `is_inlinable_scan_pipeline` accepts is
now read directly in the join task. Null-aware joins share this code so they
get it too. I skipped the one-partition check because some scans report one
partition at planning time and split later, which could put a large scan in the
single join task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]