NoahKusaba opened a new issue, #2866: URL: https://github.com/apache/datafusion-ballista/issues/2866
**Describe the bug** When a stage with a `UnionExec` is split between tasks, the task that owns one branch restricts the others to no partitions (`[]`). Since #2721, the scheduler refuses to restrict a leaf it doesn't know unless the task runs all of it. So a `UNION` or `UNION ALL` over a custom `DataSource`, such as an Iceberg scan from `ballista-iceberg` (#2638), fails whenever the adaptive planner splits its stage: ``` Error during planning: cannot restrict `DataSourceExec: format=iceberg, projection=[id, name]` to partitions [] of its 1: the scheduler does not know how to restrict a `DataSourceExec`, and left alone it would number its partitions from zero in every task. A node of your own can be restricted by `SchedulerConfig::partition_sliceable_resolver`, ... ``` The error suggests `SchedulerConfig::partition_sliceable_resolver`, which standalone mode can't set. Whether a stage is split depends on free task slots, so this shows on a busy cluster, or always with `ballista.scheduler.max_partitions_per_task = 1`. **To Reproduce** Standalone Ballista with `ballista-iceberg`, an Iceberg table `t`, and `ballista.scheduler.max_partitions_per_task = 1`: ```sql SELECT id, name FROM t UNION ALL SELECT id, name FROM t; ``` `tests/scan.rs` in `integrations/iceberg` checks this error. **Expected behavior** The rows of both branches. A task never executes a leaf it runs none of, so that leaf can be replaced by an empty one, whatever its type. **Additional context** - #2852 fixes it. - Workaround: `ballista.planner.adaptive.enabled = false`. The static planner gives such a branch a stage of its own. - A union whose branches merge their input, such as under `ORDER BY`, is a separate adaptive-planner bug: #2854. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
