NoahKusaba commented on code in PR #2852:
URL: 
https://github.com/apache/datafusion-ballista/pull/2852#discussion_r4581992631


##########
ballista/scheduler/src/state/task_builder.rs:
##########
@@ -127,6 +129,12 @@ fn restrict(
         return Ok(rewritten);
     }
 
+    // Any other leaf the task runs none of, such as one in a `UnionExec`
+    // branch another task owns, is never executed.
+    if !under_collect && partitions.is_empty() && plan.children().is_empty() {

Review Comment:
   Thanks for the repro. Done in 14af6e26: the planner now also requires the 
branch to restrict to `[0]`, so a multi-partition custom leaf gets its own 
stage again and a one-partition one stays inline.
   
   One wrinkle: tried as-is, `[0]` also failed every branch over an 
`UnresolvedShuffleExec`, which the planner sees before it resolves. That was 
where the "one stage fewer" actually came from, e.g. a union of aggregates. A 
planner-only resolver treats the unresolved shuffle as the reader it becomes, 
so those branches stay inline. Planner tests cover your shape (two stages), the 
one-partition case, and a branch over a shuffle.
   



##########
ballista/scheduler/src/state/task_builder.rs:
##########
@@ -236,6 +260,35 @@ fn ensure_leaf_runs_whole(
     )
 }
 
+/// Check that a restricted `UnionExec` branch has one partition for each of
+/// `partitions`. The executor runs every partition of the task's plan, so a
+/// branch that keeps partitions the task wasn't given runs them in this task
+/// too, and the union's later partitions no longer line up with the task's.
+/// When those partitions read their input whole, as below a `SortExec`, a
+/// `LIMIT` or an aggregate without `GROUP BY` that merges the branch into one
+/// partition, the task returns rows that another task returns as well. A node
+/// the scheduler doesn't know that keeps its own count does the same.
+fn ensure_branch_restricted(

Review Comment:
   Done in fce4eff2. The hint now follows the cause: a broadcast reader, a 
branch that merges its input (sort, limit, final aggregate), or a node that 
sets its own partition count. Only the merge case mentions the planners; a 
custom node is pointed at the resolver alone.
   



##########
ballista/client/tests/split_union.rs:
##########
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod common;
+
+/// End-to-end tests of `UNION ALL` queries whose stages are split between
+/// tasks. Each task runs only the union branches it owns, so every other
+/// branch is restricted to the task's share of it, often none. They must
+/// return the rows DataFusion returns.
+#[cfg(test)]
+#[cfg(feature = "standalone")]
+mod split_union {
+    use crate::common::{
+        ballista_state, collect_lines, datafusion_context, 
orders_customers_dir,
+        register_alltypes, register_orders_customers,
+    };
+    use ballista::prelude::SessionContextExt;
+    use ballista_core::config::{
+        BALLISTA_ADAPTIVE_PLANNER_ENABLED, 
BALLISTA_SCHEDULER_MAX_PARTITIONS_PER_TASK,
+    };
+    use datafusion::prelude::SessionContext;
+    use rstest::rstest;
+    use std::path::Path;
+
+    /// The adaptive planner replaces a stage that gives no rows with an
+    /// `EmptyExec` of that stage's partition count, so in most of these a
+    /// union branch is an `EmptyExec` of several partitions.
+    const QUERIES: &[&str] = &[
+        // An aggregate that matches nothing, beside one that matches all.
+        "SELECT id % 2 AS k, count(*) AS c FROM test WHERE id > 100 GROUP BY 
id % 2 \
+         UNION ALL SELECT id % 2, count(*) FROM test GROUP BY id % 2",
+        // The same, the other way round.
+        "SELECT id % 2 AS k, count(*) AS c FROM test GROUP BY id % 2 \
+         UNION ALL SELECT id % 2, count(*) FROM test WHERE id > 100 GROUP BY 
id % 2",
+        // An empty middle branch of three.
+        "SELECT o_cust AS k, count(*) AS c FROM orders GROUP BY o_cust \
+         UNION ALL SELECT o_cust, count(*) FROM orders WHERE o_id < 0 GROUP BY 
o_cust \
+         UNION ALL SELECT c_id, 1 FROM customers WHERE c_id < 5",
+        // An aggregate over a union with an empty branch.
+        "SELECT k, sum(c) AS s FROM ( \
+           SELECT o_cust AS k, count(*) AS c FROM orders WHERE o_id < 0 GROUP 
BY o_cust \
+           UNION ALL SELECT o_cust, count(*) FROM orders GROUP BY o_cust) \
+         GROUP BY k",
+        // A join that matches nothing below an aggregate.
+        "SELECT c_region AS r, count(*) AS n FROM orders JOIN customers ON 
o_cust = c_id \
+           WHERE o_id < 0 GROUP BY c_region \
+         UNION ALL SELECT c_region, count(*) FROM customers GROUP BY c_region",
+        // A literal beside an empty aggregate.
+        "SELECT 7 AS k, 7 AS c \
+         UNION ALL SELECT id % 2, count(*) FROM test WHERE id > 100 GROUP BY 
id % 2",
+        // A union with an empty branch joined to a table.
+        "SELECT x.k, t.c_name FROM ( \
+           SELECT o_cust AS k FROM orders WHERE o_id < 0 GROUP BY o_cust \
+           UNION ALL SELECT c_id FROM customers WHERE c_id < 3) x \
+         JOIN customers t ON x.k = t.c_id",
+        // A window over a union with an empty branch.
+        "SELECT k, sum(c) OVER (PARTITION BY k) AS s FROM ( \
+           SELECT id % 2 AS k, count(*) AS c FROM test WHERE id > 100 GROUP BY 
id % 2 \
+           UNION ALL SELECT id % 2, count(*) FROM test GROUP BY id % 2)",
+        // Scans, literals and values.
+        "SELECT id FROM test UNION ALL SELECT id FROM test",
+        "SELECT 1 AS a UNION ALL SELECT 2 UNION ALL SELECT 3",
+        "SELECT 100 AS id UNION ALL SELECT id FROM test",
+        "SELECT column1 AS id FROM (VALUES (10), (11)) UNION ALL SELECT id 
FROM test",
+    ];
+
+    async fn register(ctx: &SessionContext, dir: &Path) {
+        register_alltypes(ctx, "test").await;
+        register_orders_customers(ctx, dir).await;
+    }
+
+    /// The formatted rows of `sql`, sorted, since a union's rows come in no
+    /// particular order.
+    async fn sorted_lines(ctx: &SessionContext, sql: &str) -> Vec<String> {
+        let mut lines = collect_lines(ctx, sql).await;
+        lines.sort();
+        lines
+    }
+
+    #[rstest]
+    #[tokio::test]
+    async fn split_unions_return_the_rows_datafusion_returns(

Review Comment:
   Added both in fce4eff2:
   
   - An adaptive `UNION ALL ... ORDER BY` at one partition per task now asserts 
the full "cannot restrict a `UnionExec` branch" error. On main it returns 
duplicate rows.
   - For the custom leaf I took your scheduler-level route, since 
`generate_series` can't get through logical-plan serialization. A split union 
over `PayloadExec` with no resolver goes through `prepare_task_definition`, and 
the test checks both shipped plans: the leaf runs in its owner's task and is an 
empty source in the other.
   



##########
docs/source/user-guide/extending-components.md:
##########
@@ -179,7 +179,7 @@ let config = 
SchedulerConfig::default().with_partition_sliceable_resolver(Arc::n
 
 - `children` are the node's inputs, none for a scan, restricted to 
`partitions`. An input the node declares `SinglePartition` arrives whole, as a 
join's build side does. Partition `i` of the node must read partition `i` of 
every other input.
 - The restricted node must have exactly `partitions.len()` partitions. The 
scheduler rejects any other count.
-- `partitions` is empty for a `UnionExec` branch that belongs to other tasks. 
Return a node without partitions.
+- `partitions` is empty for a `UnionExec` branch that belongs to other tasks. 
Return a node without partitions. A leaf is never asked for none of its 
partitions: the scheduler replaces it with an empty leaf itself.

Review Comment:
   Added in 1848783f.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to