andygrove commented on code in PR #2848:
URL: 
https://github.com/apache/datafusion-ballista/pull/2848#discussion_r4579274589


##########
ballista/scheduler/src/state/aqe/optimizer_rule/distributed_exchange.rs:
##########
@@ -60,21 +61,18 @@ impl DistributedExchangeRule {
         config: &datafusion::config::ConfigOptions,
     ) -> datafusion::error::Result<Transformed<Arc<dyn ExecutionPlan>>> {
         // DataFusion's null-aware hash join coordinates visited rows and
-        // probe-side NULL state with in-process atomics. A CollectLeft join
-        // running once per probe partition therefore produces duplicate or
-        // incorrect output. This is the final plan-mutating rule, so enforce 
an
-        // explicit probe-side coalesce here after DataFusion's optimizers have
-        // finished. Add an exchange when the subtree has no existing boundary.
-        if let Some(hash_join) = execution_plan.downcast_ref::<HashJoinExec>()
-            && hash_join.null_aware
-            && *hash_join.partition_mode() == PartitionMode::CollectLeft
-            && hash_join
-                .right()
-                .downcast_ref::<CoalescePartitionsExec>()
-                .is_none()
+        // probe-side NULL state with in-process atomics, and a nested-loop 
join
+        // tracks matched build rows the same way. Such a join running once per
+        // probe partition therefore produces duplicate or incorrect output.
+        // This is the final plan-mutating rule, so enforce an explicit
+        // probe-side coalesce here after DataFusion's optimizers have 
finished.
+        // Add an exchange when the subtree has no existing boundary.
+        if probe_needs_one_task(&execution_plan)
+            && let [left, right] = execution_plan.children().as_slice()
+            && right.downcast_ref::<CoalescePartitionsExec>().is_none()
         {
-            let left = hash_join.left().clone();
-            let right = hash_join.right().clone();
+            let left = Arc::clone(left);
+            let right = Arc::clone(right);
             let right = if right.downcast_ref::<ExchangeExec>().is_none()

Review Comment:
   Small one. When the probe already has a single partition, or is a cheap scan 
that `is_inlinable_scan_pipeline` would inline, this still adds an 
`ExchangeExec` and a stage. Would it be worth just wrapping it in 
`CoalescePartitionsExec` in that case? The null-aware path does the same thing 
today, so feel free to leave it.



##########
ballista/scheduler/src/planner.rs:
##########
@@ -202,6 +202,35 @@ impl DefaultDistributedPlanner {
             stages.append(&mut child_stages);
         }
 
+        // A nested-loop join tracks which build rows matched in memory shared 
by
+        // its probe partitions, so a join type that emits build rows is only
+        // correct in one task. Give its probe side a one-partition stage of 
its
+        // own: an inline `CoalescePartitionsExec` would not hold, as
+        // `EnsureRequirements` removes one the join does not require.
+        if let Some(join) = execution_plan.downcast_ref::<NestedLoopJoinExec>()
+            && !collect_left_broadcast_safe(*join.join_type())
+            && children[1]
+                .properties()
+                .output_partitioning()
+                .partition_count()
+                > 1
+        {
+            let probe = 
Arc::new(CoalescePartitionsExec::new(children[1].clone()));

Review Comment:
   Nice catch on the `EnsureRequirements` interaction, the comment explains it 
well. One question on the static path. The probe stage here is 
`CoalescePartitionsExec(children[1])`, so the probe scan and its filters now 
run in a single task on one executor, while the adaptive path keeps that stage 
parallel. Could the static path write `children[1]` as an ordinary N-partition 
stage and read it back through a one-partition reader instead? 
`UnresolvedShuffleExec::new_coalesced` with a single group covering all 
upstream partitions looks like it would do that. A reader is a leaf, so 
`EnsureRequirements` can't strip it. Happy for this to be a follow-up if you'd 
rather keep this PR focused on correctness.



##########
ballista/scheduler/src/state/aqe/optimizer_rule/distributed_exchange.rs:
##########
@@ -478,6 +487,40 @@ mod tests {
         ");
     }
 
+    #[test]
+    fn nested_loop_join_coalesces_probe_only_when_it_emits_build_rows() {

Review Comment:
   The tests look great, and I checked that every case fails on `main` under 
both planners. Two optional additions. Could this test loop over all `JoinType` 
variants and assert the coalesce appears for exactly the five unsafe ones? And 
a static-planner shape test using `stages_for_sql` in `planner.rs` would pin 
the new stage without needing a standalone cluster.



##########
ballista/scheduler/src/planner.rs:
##########
@@ -202,6 +202,35 @@ impl DefaultDistributedPlanner {
             stages.append(&mut child_stages);
         }
 
+        // A nested-loop join tracks which build rows matched in memory shared 
by
+        // its probe partitions, so a join type that emits build rows is only
+        // correct in one task. Give its probe side a one-partition stage of 
its
+        // own: an inline `CoalescePartitionsExec` would not hold, as
+        // `EnsureRequirements` removes one the join does not require.
+        if let Some(join) = execution_plan.downcast_ref::<NestedLoopJoinExec>()
+            && !collect_left_broadcast_safe(*join.join_type())

Review Comment:
   Tiny naming nit. `collect_left_broadcast_safe` now also decides when a 
nested-loop join must run as one task. A sentence in its doc comment saying it 
is reused for that would help the next reader.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to