comphead commented on code in PR #220:
URL: https://github.com/apache/datafusion-site/pull/220#discussion_r4605030387


##########
content/blog/2026-11-15-comet-parquet-compaction.md:
##########
@@ -0,0 +1,257 @@
+---
+layout: post
+title: Round Robin Without the Hash: Faster Parquet Compaction in Apache 
DataFusion Comet
+date: 2026-11-15
+author: pmc
+categories: [performance, subprojects]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+Apache DataFusion Comet tracks its performance with standard benchmarks such 
as TPC-H and TPC-DS. Alongside them, we keep testing Comet on real-world use 
cases. The first step was [nested 
TPC-H](https://datafusion.apache.org/blog/2026/10/25/comet-nested-tpch/), a 
TPC-H derived benchmark that brings in the nested data real workloads often 
have. This post looks at a real customer workload: Parquet compaction.
+
+Compaction rewrites the many small files of a table partition as a few large 
ones. It runs in the background across many datasets of a data lake, and it is 
typically very expensive in terms of resources. For plain Parquet tables it is 
often a single line of Spark, and at its center is the round robin shuffle 
behind `repartition(n)`.
+
+On a table with more than 200 columns, nested up to seven levels deep, Comet 
compacted a 200 GB daily partition in 70 minutes, against 80 minutes for Spark. 
The bottleneck was the shuffle write, where Comet's round robin hashed every 
value of every row. With positional round robin, which places rows by their 
position and copies them in contiguous groups, Comet compacts the same 
partition in 28 minutes. That is 2.9x faster than Spark, with 6.4x less memory.
+
+<img
+src="/blog/images/comet-parquet-compaction/results.svg"
+width="100%"
+class="img-fluid"
+alt="Compaction of a 200 GB partition. Execution time: Spark 4.0 80 minutes, 
Comet with hash round robin 70 minutes, Comet with positional round robin 28 
minutes. Peak memory: 770 GB, 130 GB and 120 GB."
+/>
+
+This post explains how compaction runs, why hashing was slow on nested data, 
and the three approaches we tried: a faster hash, a vectorized round robin that 
was fast but not safe when tasks are retried, and the positional round robin 
that Comet has offered since 1.1.0.
+
+## What compaction does
+
+Small files are a side effect of how data lands in a data lake. Streaming jobs 
commit every few minutes, many writers each add files to the same partition, 
and late data arrives in small batches. Every file costs a reader a file open, 
a footer read and at least one request to object storage, and small files hold 
small row groups that encode and compress less well. A day of data is slower to 
read from a thousand small files than from a few dozen large ones.
+
+Compaction rewrites a partition's small files as a few large ones. It is 
heavily I/O bound, because it reads and rewrites all of the data it touches. It 
usually runs on a schedule, so any saving repeats for every dataset and every 
partition it runs on.
+
+Table formats such as Apache Iceberg and Delta Lake have their own procedures 
for compaction. For a plain directory of Parquet files, a common way is a short 
Spark job:
+
+```scala
+spark.read.parquet(inputPath)
+  .repartition(50)
+  .write.parquet(outputPath)
+```
+
+`repartition(50)` with no columns asks Spark for round robin partitioning. 
Rows are dealt evenly to 50 partitions regardless of their content, and each 
partition is written as one file, so the output files come out about the same 
size. `coalesce(50)` would skip the shuffle, but it also limits the scan to 50 
tasks, and because it merges whole input splits, the output sizes follow the 
input.
+
+### How the job runs
+
+The job runs as two stages with a shuffle between them:
+
+1. **Map stage.** Each task scans one split of the input, assigns every row to 
one of the 50 output partitions (round robin partitioning) and writes the rows 
to a local shuffle file, grouped by partition (shuffle write).
+2. **Reduce stage.** Each of the 50 tasks fetches its partition from every map 
task (shuffle read) and writes the rows as one Parquet file (Parquet write).
+
+Every byte of the partition crosses I/O four times. It is read from the input 
files, written to shuffle files, read back by the reducers and written to the 
output files.
+
+<img
+src="/blog/images/comet-parquet-compaction/pipeline.svg"
+width="100%"
+class="img-fluid"
+alt="The compaction job in two stages. Map stage: Parquet scan, round robin 
partitioning that sends each row to one of 50 partitions, and shuffle write 
grouped by partition. Reduce stage with 50 tasks: shuffle read from every map 
task and Parquet write of one file per task. Input is 1,200 files and 200 GB, 
output is 50 files."
+/>
+
+Comet runs the map stage natively. Its Parquet scan produces Arrow record 
batches, and its native shuffle writer partitions them, buffers them per 
partition, spills when memory runs short and writes each partition as 
compressed Arrow IPC blocks. All of it runs in Rust, without converting the 
batches to rows.
+
+### Wide and deep data
+
+The input is one daily partition of a production event table, 1,200 Parquet 
files and 200 GB in total. The output is 50 files.
+
+The table is wide and deep. It has more than 200 columns, nested up to seven 
levels deep. Parquet stores each leaf field as its own column. Arrow gives each 
level of nesting its own array with its own validity bitmap, and each list adds 
an offsets buffer. Even a small nested column is several arrays:

Review Comment:
   makes sense



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to