andygrove opened a new pull request, #2125: URL: https://github.com/apache/datafusion-ballista/pull/2125
# Which issue does this PR close? N/A - documentation fix, no issue filed. # Rationale for this change The "Partitions and Parallelism" section of the tuning guide describes behavior that is no longer accurate, and the inaccuracy points users in the wrong direction when they size their clusters. It states that "Ballista currently treats each file within a table as a single partition (in the future, Ballista will support splitting files into partitions but this is not implemented yet)", and that "Ballista will never decrease the number of partitions based on this setting". Neither is true today: - File listing chunks the table's files into **at most** `target_partitions` file groups (`FileGroup::split_files`), so a 200 file table read with the default `target_partitions = 16` gives 16 partitions, not 200. The setting does decrease partition count. - The physical optimizer then splits individual files by byte range (`datafusion.optimizer.repartition_file_scans`, gated on `datafusion.optimizer.repartition_file_min_size`), so a table consisting of one large Parquet file does scale across the cluster, contrary to what the guide warns. The guide also never mentioned `ballista.scheduler.max_partitions_per_task`, which defaults to `1` and therefore determines the task count for every stage. Someone tuning `--vcores` without knowing about it cannot explain why a 16 partition stage produces 16 tasks. # What changes are included in this PR? - Rewrites the scan partitioning explanation to describe the two actual steps (file group chunking capped at `target_partitions`, then byte range splitting within files) and notes that Ballista disables round robin repartitioning, so file group splitting is the only source of scan parallelism. - Adds a "Partitions and Tasks" subsection covering the partition to task binding, `ballista.scheduler.max_partitions_per_task` (default `1`, `0` for unbounded), and the collapse stage exemption. # Are there any user-facing changes? Documentation only. No code or API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
