andygrove opened a new issue, #2728: URL: https://github.com/apache/datafusion-ballista/issues/2728
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** The adaptive planner's `CoalescePartitionsRule` coalesces a shuffle's partitions only when every operator between the stage root and the shuffle is on a hand-kept list of operators that produce the same rows over any partitioning of their input (`tolerates_partition_count_change`). An extension operator from a client that submitted its own physical plan is assumed to depend on the count, which is the safe default. Comet Native Executors (apache/datafusion-comet#7288) runs every stage as a `CometFragmentExec`, so its queries never get partition coalescing. **Describe the solution you'd like** Let an `ExecutionPlan` declare this itself, for example through a method on a Ballista extension trait, or a resolver in `SchedulerConfig` like the one apache/datafusion-ballista#2721 adds for partition slicing. A Comet stage that reads only hash shuffles could then declare it. **Describe alternatives you've considered** Keeping the whitelist and adding Comet's operator to it would make Ballista depend on Comet. **Additional context** Comet's `CometFragmentExec` also carries a partition descriptor child that would have to coalesce along with its shuffle inputs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
