comphead opened a new pull request, #220: URL: https://github.com/apache/datafusion-site/pull/220
Adds a blog post on Parquet compaction with Apache DataFusion Comet, and on how positional round robin ([apache/datafusion-comet#6095](https://github.com/apache/datafusion-comet/pull/6095)) made it 2.9x faster than Spark. **Rendered version:** [content/blog/2026-11-15-comet-parquet-compaction.md](https://github.com/comphead/datafusion-site/blob/preview-comet-parquet-compaction/content/blog/2026-11-15-comet-parquet-compaction.md). The image paths are absolute (`/blog/images/...`), so the figures don't render in the GitHub preview. The post covers: - What compaction is, and how `spark.read.parquet(...).repartition(50).write.parquet(...)` runs. The map stage scans, partitions rows round robin and writes the shuffle. The reduce stage reads the shuffle and writes one Parquet file per task. - The data: a 200 GB daily partition in 1,200 files, with more than 200 columns nested up to seven levels deep, and why nesting makes per-row work expensive in Arrow. - Why Comet's round robin hashed every column (retry determinism, SPARK-23207), and why that made the shuffle write slow on nested data ([#5397](https://github.com/apache/datafusion-comet/issues/5397)). - The approaches that followed. A faster hash, XXH3 ([#5450](https://github.com/apache/datafusion-comet/pull/5450)), was slower. Vectorized round robin by batch ([#5449](https://github.com/apache/datafusion-comet/pull/5449)) was fast but not safe under task and executor retries. Positional round robin keyed on a row ordinal ([#6095](https://github.com/apache/datafusion-comet/pull/6095)) is fast and keeps Spark's fault tolerance. - Results. Spark 4.0 takes 80 min with 770 GB peak memory. Comet with hash round robin takes 70 min with 130 GB. Comet with positional round robin takes 28 min with 120 GB. Files: - `content/blog/2026-11-15-comet-parquet-compaction.md` - `content/images/comet-parquet-compaction/`: three SVG figures (results, the job's stages, and row placement under the three strategies). They embed the same Lato subset as the nested TPC-H charts. ## Draft status To confirm before review: - **Peak memory.** The table labels it "mean over executors". The definition needs confirming, since 770 GB is large for a per-executor figure. - **Setup.** The post does not state the cluster size, executor cores and memory, or the number of runs per configuration. - **Comet build.** Described as "built from `main` ahead of 1.3.0". An exact commit would be better. - **Write path.** The post does not say whether the Parquet write ran on Comet's native writer or on Spark's. - **Date.** The post date (2026-11-15) needs updating to the publish date. Checks: - `typos` passes. - The site was not built with Pelican. The post was rendered with Python-Markdown using the site's extensions (`fenced_code`, `toc`, `tables`), and the figures were rendered with resvg and checked by eye. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
