andygrove opened a new issue, #2760: URL: https://github.com/apache/datafusion-ballista/issues/2760
> **This is a research task.** Spark AQE is the reference point for the problem, not for the solution. Ballista has different constraints than Spark (DataFusion operators, stage-at-a-time replanning, no external shuffle service, partition-count-sensitive operators), so we may not want to copy how Spark does this. The first deliverable is a write-up of whether and how to do it here, ideally with a measurement showing the gain, before any implementation. ## Problem Spark AQE also adapts the *output* of a query. The `REBALANCE` hint inserts a `RebalancePartitions` exchange before a write, and `OptimizeSkewInRebalancePartitions` then splits skewed partitions and coalesces small ones (`CoalesceShufflePartitions` handles the latter). The point is the size of the files that get written: not many tiny files from over-partitioned stages, and not one huge file from a skewed key. Ballista can write (`CopyTo` / `INSERT` plans, and Iceberg in #2174). The number and size of files written follows the final stage's partition count. That is `target_partitions` after any coalescing, whatever the data volume or skew. A small result produces `target_partitions` small files, and a skewed one produces a few large files. ## Questions to answer - Is file sizing a problem users actually hit with Ballista writes? Check what a small and a skewed `CREATE TABLE AS` / `COPY` produce today. - If so, what is the right shape: a SQL-visible hint, an automatic final-stage rebalance before a write sink, or a writer that rolls files by size? - Should the write sink (an extension node) declare that it tolerates a changed partition count? See #2728 for the same question for coalescing. - Does Iceberg's writer already own this (target file size) so Ballista need only avoid getting in its way? ## Related - #2728, #2174, #2622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
