avantgardnerio commented on code in PR #2721: URL: https://github.com/apache/datafusion-ballista/pull/2721#discussion_r4530327282
########## docs/source/upgrading/56.0.0.md: ########## @@ -178,6 +178,28 @@ early, so a straggler partition leaves them idle. Set `ballista.scheduler.max_partitions_per_task` to a positive value to dispatch smaller tasks, which release their vcores sooner. +#### Stages split between tasks fail on a leaf the scheduler cannot restrict + +When a stage is split between tasks, each task gets a plan restricted to its +own partitions. A leaf the scheduler did not know how to restrict used to be +left as it was, so a task that ran partitions 4 and 5 of it ran its partitions 0 +and 1, and the stage returned some partitions twice and others not at all, with +no error. + +The scheduler now fails the job instead. Such a leaf is accepted only in a task Review Comment: Would a query fail? I thought it would just get its tasks restricted down to available slots, including to 1, and then succeed, but slower? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
