andygrove opened a new issue, #7367: URL: https://github.com/apache/datafusion-comet/issues/7367
### What is the problem the feature request solves? To range-partition an exchange, the Comet scheduler (#7289) runs a sampling job of its own: a `SampleExec` over the stage that feeds the exchange. That is a separate Ballista job, so it runs that stage and every stage below it, then the query runs them all again. Every further range exchange above them adds another run. Spark's `RangePartitioner` also re-runs the child, but it reuses the shuffle outputs below it. The sampling jobs also run one after another while the query is being planned, even for range exchanges whose inputs don't depend on each other. ### Describe the potential solution Options, roughly from least to most work: - Sample range exchanges whose inputs are independent concurrently. - Sample from the materialized shuffle output below the exchange, so only the producing stage runs twice. - Find the split points at run time, as Ballista's adaptive planner does for window functions: `RuntimeStatsExec` keeps a quantile sketch and `UnorderedRangeRepartitionExec` cuts by it. That covers only the first sort key today, so it would need compound keys. The driver-side sampler, `SampleExec` and its codec would then go away. ### Additional context The sampling is `range_exchange` in `native/native-executors/src/assembly.rs` and `collect` in `src/frontend.rs`. Part of #7288. ### Willingness to contribute Yes. I can contribute this feature independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
