comphead opened a new pull request, #220:
URL: https://github.com/apache/datafusion-site/pull/220

   Adds a blog post on Parquet compaction with Apache DataFusion Comet, and on 
how positional round robin 
([apache/datafusion-comet#6095](https://github.com/apache/datafusion-comet/pull/6095))
 made it 2.9x faster than Spark.
   
   **Rendered version:** 
[content/blog/2026-11-15-comet-parquet-compaction.md](https://github.com/comphead/datafusion-site/blob/preview-comet-parquet-compaction/content/blog/2026-11-15-comet-parquet-compaction.md).
 The image paths are absolute (`/blog/images/...`), so the figures don't render 
in the GitHub preview.
   
   The post covers:
   
   - What compaction is, and how 
`spark.read.parquet(...).repartition(50).write.parquet(...)` runs. The map 
stage scans, partitions rows round robin and writes the shuffle. The reduce 
stage reads the shuffle and writes one Parquet file per task.
   - The data: a 200 GB daily partition in 1,200 files, with more than 200 
columns nested up to seven levels deep, and why nesting makes per-row work 
expensive in Arrow.
   - Why Comet's round robin hashed every column (retry determinism, 
SPARK-23207), and why that made the shuffle write slow on nested data 
([#5397](https://github.com/apache/datafusion-comet/issues/5397)).
   - The approaches that followed. A faster hash, XXH3 
([#5450](https://github.com/apache/datafusion-comet/pull/5450)), was slower. 
Vectorized round robin by batch 
([#5449](https://github.com/apache/datafusion-comet/pull/5449)) was fast but 
not safe under task and executor retries. Positional round robin keyed on a row 
ordinal ([#6095](https://github.com/apache/datafusion-comet/pull/6095)) is fast 
and keeps Spark's fault tolerance.
   - Results. Spark 4.0 takes 80 min with 770 GB peak memory. Comet with hash 
round robin takes 70 min with 130 GB. Comet with positional round robin takes 
28 min with 120 GB.
   
   Files:
   
   - `content/blog/2026-11-15-comet-parquet-compaction.md`
   - `content/images/comet-parquet-compaction/`: three SVG figures (results, 
the job's stages, and row placement under the three strategies). They embed the 
same Lato subset as the nested TPC-H charts.
   
   ## Draft status
   
   To confirm before review:
   
   - **Peak memory.** The table labels it "mean over executors". The definition 
needs confirming, since 770 GB is large for a per-executor figure.
   - **Setup.** The post does not state the cluster size, executor cores and 
memory, or the number of runs per configuration.
   - **Comet build.** Described as "built from `main` ahead of 1.3.0". An exact 
commit would be better.
   - **Write path.** The post does not say whether the Parquet write ran on 
Comet's native writer or on Spark's.
   - **Date.** The post date (2026-11-15) needs updating to the publish date.
   
   Checks:
   
   - `typos` passes.
   - The site was not built with Pelican. The post was rendered with 
Python-Markdown using the site's extensions (`fenced_code`, `toc`, `tables`), 
and the figures were rendered with resvg and checked by eye.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to