andygrove commented on code in PR #2721: URL: https://github.com/apache/datafusion-ballista/pull/2721#discussion_r4530693948
########## docs/source/upgrading/56.0.0.md: ########## @@ -178,6 +178,28 @@ early, so a straggler partition leaves them idle. Set `ballista.scheduler.max_partitions_per_task` to a positive value to dispatch smaller tasks, which release their vcores sooner. +#### Stages split between tasks fail on a leaf the scheduler cannot restrict + +When a stage is split between tasks, each task gets a plan restricted to its +own partitions. A leaf the scheduler did not know how to restrict used to be +left as it was, so a task that ran partitions 4 and 5 of it ran its partitions 0 +and 1, and the stage returned some partitions twice and others not at all, with +no error. + +The scheduler now fails the job instead. Such a leaf is accepted only in a task Review Comment: It does fail, but only for plans that contain a leaf the scheduler can't restrict. Your picture is right for everything else. When a stage gets fewer slots, its tasks get fewer partitions each. For a shuffle reader, a Parquet/memory scan, `EmptyExec` or `PlaceholderRowExec`, the task's plan is narrowed to those partitions and the query just runs slower, as you describe. The problem is a leaf the scheduler doesn't know how to narrow, such as a `DataSourceExec` over a custom `DataSource` or a third-party scan with no resolver. Cutting the task down to the free slots is exactly when that leaf has to be narrowed. Before this PR it was left as it was, so a task given partitions 2 and 3 actually ran its partitions 0 and 1. The query "succeeded" but returned some partitions twice and others not at all. Now `ensure_leaf_runs_whole` accepts such a leaf only when the task runs all of its partitions (`0..count`). Otherwise the job fails with an error naming the node. So on an idle cluster the stage goes out as one task holding every partition and passes. On a busy cluster, or when part of the stage is retried, it gets split and fails, where before it silently gave wrong results. That's the load dependence the paragraph is warning about. The fix for an affected plan is a resolver (or `PartitionSliceable`) for the node, and then it behaves like the built-in leaves: restricted, slower, correct. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
