andygrove commented on code in PR #2852:
URL:
https://github.com/apache/datafusion-ballista/pull/2852#discussion_r4579258205
##########
ballista/scheduler/src/state/task_builder.rs:
##########
@@ -141,7 +149,11 @@ fn restrict(
.into_iter()
.cloned()
.zip(per_child)
- .map(|(child, sub)| restrict(child, &sub, false, resolver))
+ .map(|(child, sub)| {
+ let restricted = restrict(child, &sub, false, resolver)?;
+ ensure_branch_restricted(&restricted, &sub)?;
Review Comment:
Turning the wrong rows into an error is clearly better than what main does
today. I confirmed main returns 36 formatted lines instead of 20 for `SELECT id
FROM test UNION ALL SELECT id FROM test ORDER BY id` with one partition per
task, and this branch fails instead. My worry is that it fails on the default
planner, at task time, and only when the stage happens to be split. So the same
query can pass on an idle cluster and fail on a busy one.
Have you considered packing such a stage into a single task instead, the way
`stage_has_input_collapse` already does for collapse stages? Each branch would
then run once and the query would succeed, just with less parallelism, until
#2854 lands. If you'd rather keep the error, I think we should agree on whether
#2854 has to land before 56.0.0.
##########
docs/source/user-guide/extending-components.md:
##########
@@ -179,7 +179,7 @@ let config =
SchedulerConfig::default().with_partition_sliceable_resolver(Arc::n
- `children` are the node's inputs, none for a scan, restricted to
`partitions`. An input the node declares `SinglePartition` arrives whole, as a
join's build side does. Partition `i` of the node must read partition `i` of
every other input.
- The restricted node must have exactly `partitions.len()` partitions. The
scheduler rejects any other count.
-- `partitions` is empty for a `UnionExec` branch that belongs to other tasks.
Return a node without partitions.
+- `partitions` is empty for a `UnionExec` branch that belongs to other tasks.
Return a node without partitions. A leaf is never asked for none of its
partitions: the scheduler replaces it with an empty leaf itself.
Review Comment:
Small doc suggestion: could this line also say that a node with inputs is
still asked for `[]` in such a branch, and now fails the job below a
`UnionExec` if it keeps its own partition count? Resolver authors would see
that right next to the contract.
##########
ballista/client/tests/split_union.rs:
##########
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod common;
+
+/// End-to-end tests of `UNION ALL` queries whose stages are split between
+/// tasks. Each task runs only the union branches it owns, so every other
+/// branch is restricted to the task's share of it, often none. They must
+/// return the rows DataFusion returns.
+#[cfg(test)]
+#[cfg(feature = "standalone")]
+mod split_union {
+ use crate::common::{
+ ballista_state, collect_lines, datafusion_context,
orders_customers_dir,
+ register_alltypes, register_orders_customers,
+ };
+ use ballista::prelude::SessionContextExt;
+ use ballista_core::config::{
+ BALLISTA_ADAPTIVE_PLANNER_ENABLED,
BALLISTA_SCHEDULER_MAX_PARTITIONS_PER_TASK,
+ };
+ use datafusion::prelude::SessionContext;
+ use rstest::rstest;
+ use std::path::Path;
+
+ /// The adaptive planner replaces a stage that gives no rows with an
+ /// `EmptyExec` of that stage's partition count, so in most of these a
+ /// union branch is an `EmptyExec` of several partitions.
+ const QUERIES: &[&str] = &[
+ // An aggregate that matches nothing, beside one that matches all.
+ "SELECT id % 2 AS k, count(*) AS c FROM test WHERE id > 100 GROUP BY
id % 2 \
+ UNION ALL SELECT id % 2, count(*) FROM test GROUP BY id % 2",
+ // The same, the other way round.
+ "SELECT id % 2 AS k, count(*) AS c FROM test GROUP BY id % 2 \
+ UNION ALL SELECT id % 2, count(*) FROM test WHERE id > 100 GROUP BY
id % 2",
+ // An empty middle branch of three.
+ "SELECT o_cust AS k, count(*) AS c FROM orders GROUP BY o_cust \
+ UNION ALL SELECT o_cust, count(*) FROM orders WHERE o_id < 0 GROUP BY
o_cust \
+ UNION ALL SELECT c_id, 1 FROM customers WHERE c_id < 5",
+ // An aggregate over a union with an empty branch.
+ "SELECT k, sum(c) AS s FROM ( \
+ SELECT o_cust AS k, count(*) AS c FROM orders WHERE o_id < 0 GROUP
BY o_cust \
+ UNION ALL SELECT o_cust, count(*) FROM orders GROUP BY o_cust) \
+ GROUP BY k",
+ // A join that matches nothing below an aggregate.
+ "SELECT c_region AS r, count(*) AS n FROM orders JOIN customers ON
o_cust = c_id \
+ WHERE o_id < 0 GROUP BY c_region \
+ UNION ALL SELECT c_region, count(*) FROM customers GROUP BY c_region",
+ // A literal beside an empty aggregate.
+ "SELECT 7 AS k, 7 AS c \
+ UNION ALL SELECT id % 2, count(*) FROM test WHERE id > 100 GROUP BY
id % 2",
+ // A union with an empty branch joined to a table.
+ "SELECT x.k, t.c_name FROM ( \
+ SELECT o_cust AS k FROM orders WHERE o_id < 0 GROUP BY o_cust \
+ UNION ALL SELECT c_id FROM customers WHERE c_id < 3) x \
+ JOIN customers t ON x.k = t.c_id",
+ // A window over a union with an empty branch.
+ "SELECT k, sum(c) OVER (PARTITION BY k) AS s FROM ( \
+ SELECT id % 2 AS k, count(*) AS c FROM test WHERE id > 100 GROUP BY
id % 2 \
+ UNION ALL SELECT id % 2, count(*) FROM test GROUP BY id % 2)",
+ // Scans, literals and values.
+ "SELECT id FROM test UNION ALL SELECT id FROM test",
+ "SELECT 1 AS a UNION ALL SELECT 2 UNION ALL SELECT 3",
+ "SELECT 100 AS id UNION ALL SELECT id FROM test",
+ "SELECT column1 AS id FROM (VALUES (10), (11)) UNION ALL SELECT id
FROM test",
+ ];
+
+ async fn register(ctx: &SessionContext, dir: &Path) {
+ register_alltypes(ctx, "test").await;
+ register_orders_customers(ctx, dir).await;
+ }
+
+ /// The formatted rows of `sql`, sorted, since a union's rows come in no
+ /// particular order.
+ async fn sorted_lines(ctx: &SessionContext, sql: &str) -> Vec<String> {
+ let mut lines = collect_lines(ctx, sql).await;
+ lines.sort();
+ lines
+ }
+
+ #[rstest]
+ #[tokio::test]
+ async fn split_unions_return_the_rows_datafusion_returns(
Review Comment:
Thanks for the broad query matrix in `split_union.rs`. I ran it against
main's scheduler and it passes there too, so it doesn't pin either change yet.
Could we add an adaptive `UNION ALL ... ORDER BY` with `max_partitions_per_task
= 1` that asserts the "cannot restrict a `UnionExec` branch" error? Main
returns duplicate rows for that one. A test with a custom leaf in a union
branch would cover the Iceberg fix. A scheduler-level test through
`prepare_task_definition` with `PayloadExec` would be enough if a custom source
in the client crate is too heavy.
##########
ballista/scheduler/src/state/task_builder.rs:
##########
@@ -466,33 +519,24 @@ fn select_output_partitions(
return Ok(None);
}
- // Self-generating leaves: PlaceholderRow (1 all-null row per partition)
- // and EmptyExec (0 rows per partition). Every partition of one of these
- // is semantically identical, so `with_partitions(indices.len())` is the
- // algebraically correct restriction.
- //
- // Caveat: DataFusion's proto codec for these two doesn't roundtrip the
- // `partitions` field — encode preserves only schema; decode always
- // reconstructs with `::new(schema)` (partitions = 1). So a plan we
- // restrict to `with_partitions(0)` here would arrive at the executor as
- // a 1-partition leaf, undoing the restriction and reproducing the
- // duplicated-output bug this whole function is meant to prevent.
- //
- // Workaround for the empty-slice case: swap in a 0-partition
- // `MemorySourceConfig` (which DOES roundtrip its partition list).
- // For non-empty slices we return the leaf as it is — currently safe
because
- // the only shape emitted is `partitions == 1` and UnionExec routing gives
- // sub-slices of `[0]` (identity, no restriction needed) or `[]` (handled
by
- // this branch). Returned rather than `None` so the walker's leaf check
- // accepts it: an `EmptyExec` under the round-robin `RepartitionExec` that
- // `make_empty_exec_serde_safe` adds sees that repartition's partition
- // numbers, not its own.
+ // Self-generating leaves: PlaceholderRow (1 row per partition) and
+ // EmptyExec (0 rows per partition). Every partition is the same, so a
+ // slice is the leaf with one partition for each index it has. As for the
+ // sources above, an index past its own count is dropped: under a
+ // round-robin `RepartitionExec` the leaf sees the repartition's numbers,
+ // and only the task that owns partition 0 may run a one-partition leaf.
if plan.is::<PlaceholderRowExec>() || plan.is::<EmptyExec>() {
Review Comment:
Nice catch on the `PlaceholderRowExec` under a round-robin repartition.
Since that changes stages without a union as well, could the PR body or upgrade
note say so?
##########
ballista/scheduler/src/state/task_builder.rs:
##########
@@ -236,6 +260,35 @@ fn ensure_leaf_runs_whole(
)
}
+/// Check that a restricted `UnionExec` branch has one partition for each of
+/// `partitions`. The executor runs every partition of the task's plan, so a
+/// branch that keeps partitions the task wasn't given runs them in this task
+/// too, and the union's later partitions no longer line up with the task's.
+/// When those partitions read their input whole, as below a `SortExec`, a
+/// `LIMIT` or an aggregate without `GROUP BY` that merges the branch into one
+/// partition, the task returns rows that another task returns as well. A node
+/// the scheduler doesn't know that keeps its own count does the same.
+fn ensure_branch_restricted(
Review Comment:
Could the error mention the adaptive-planner workaround only when the
branch's node is one Ballista knows, such as `SortExec`, `GlobalLimitExec` or
an aggregate? For a custom node, pointing at the resolver alone would steer the
user better.
##########
ballista/scheduler/src/state/task_builder.rs:
##########
@@ -127,6 +129,12 @@ fn restrict(
return Ok(rewritten);
}
+ // Any other leaf the task runs none of, such as one in a `UnionExec`
+ // branch another task owns, is never executed.
+ if !under_collect && partitions.is_empty() && plan.children().is_empty() {
Review Comment:
Nice simplification here, and I like that the empty leaf needs no resolver.
One thing I tripped over: `restricts_to_nothing` in `planner.rs` asks the same
rewriter with `[]` and no resolver, so with this shortcut the static planner
now keeps a branch over a custom leaf inline. That's where the "one stage
fewer" comes from. For a leaf with more than one partition, though, a task
whose slice covers only part of it now hits `ensure_leaf_runs_whole`. On main
that branch had its own stage and ran whole in one task.
I reproduced it with `UnionExec[scan_with_file_groups(3),
PayloadExec::leaf(&["p0", "p1"])]` through `DefaultDistributedPlanner`. Main
plans two stages and restricting the union stage to `[0, 1, 2, 3]` works. This
branch plans one stage and the same slice fails on the `PayloadExec`.
Could the planner's check stay strict? One option is to also require
`restrict_plan_to_partitions(plan, &[0], None)` to succeed when the branch has
partitions. That keeps the one-partition case inline and sends multi-partition
custom leaves to their own stage. The doc comment on `restricts_to_nothing`
("No resolver is passed, so a node Ballista does not know gets a stage of its
own") would then still hold. A planner test for that shape would pin it down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]