andygrove commented on PR #2722: URL: https://github.com/apache/datafusion-ballista/pull/2722#issuecomment-6775192285
Good question. Ballista never computes the split points for Comet. Spark does, in the driver, before the plan reaches Ballista. When Spark builds the shuffle for a range exchange (`ShuffleExchangeExec.prepareShuffleDependency`), it constructs a `RangePartitioner`. That constructor runs a separate small Spark job over the child. The job reservoir-samples the sort keys, sorts the sample, and picks `numPartitions - 1` boundary rows, which are the split points. They come from sampling the data and not from static statistics. Comet's native shuffle already does this: `CometShuffleExchangeExec` builds the same `RangePartitioner` over the sampled sort keys and reads `rangeBounds` out of it. They go into the `boundary_rows` field of Comet's range partitioning proto. Ballista is not called repeatedly. The sampling is a separate Spark job, and the Ballista job is submitted once. Comet Native Executors doesn't hand Ballista a range partitioning yet. Today the driver sends none of that, and the cluster side turns a range exchange into a coalesce, so the consuming stage reads it as one partition. That sorts correctly but runs in a single task. This PR is what lets us stop doing that, because the planner no longer drops `Partitioning::Range`. Getting the bounds into the plan is the driver's job. It would run the sampling job and put the result into the `RepartitionExec`'s `Partitioning::Range` expressions. We haven't built that part, so I can't say yet how the sampling job gets its rows. This also means Ballista only needs to route against fixed split points here. It doesn't need `RuntimeStatsExec` for this path. I think that's why it complements your AQE work. Your AQE path derives the split points from runtime stats. This path accepts a plan whose split points are already decided. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
