wombatu-kun opened a new pull request, #16855: URL: https://github.com/apache/iceberg/pull/16855
## What changed Documents how output Parquet file sizes are determined on the Spark write path for skewed tables, and points users to compaction when write-side tuning is not enough. All edits are in `docs/docs/spark-writes.md`: - A new "Choosing a distribution mode for skewed data" subsection under "Writing Distribution Modes": `hash` clusters each partition value onto a single task (roughly one file per small partition, while hot partitions are still split to target size by AQE), whereas `range` builds global range boundaries that can scatter a small partition's rows across many tasks - the usual cause of many tiny files in a long-tail layout. - An expanded "Controlling File Sizes" section: clarifies that `write.target-file-size-bytes` is only a per-task upper bound (it never merges across tasks, never enlarges a file whose task ran out of input, and a file cannot span a partition boundary), notes that `spark.sql.shuffle.partitions` and the AQE advisory size are what actually control write file counts, and bridges to the `rewrite_data_files` procedure (mirroring the link already used in the structured-streaming docs) with the scale knobs `partial-progress.enabled`, `max-concurrent-file-group-rewrites`, `rewrite-job-order`, and `where` for large tables. ## Why Reported in #16854 (https://github.com/apache/iceberg/issues/16854): a user with a bimodal table (a few hot partitions, a long tail of tiny ones) raised `write.target-file-size-bytes` and saw no change, because the target is a per-task upper bound and cannot consolidate small partitions. The Spark docs already explained that a file cannot exceed a Spark task or span a partition boundary, but never stated this asymmetry, never mentioned skew (all such guidance lived only in the Flink docs), and the "Controlling File Sizes" section never pointed to compaction as the remedy. This closes that documentation gap. Closes #16854 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
