Hi Grant,

Not yet. The benchmark so far measures the conversion alone, with no other
writer on the table.

There are two cases:

- If the compaction commits first, the sink's commit fails, and the
coordinator converts again and retries.
- If the sink commits first, a compaction that is rewriting one of those
files fails. The sink cannot solve this on its own. When the changes are
spread over the table, I expect a rewrite that takes longer than the commit
interval to rarely succeed.

I'll run compaction alongside repeated conversion commits and report the
retries, the commit latency, and how often the compaction succeeds. I'm
away until November 28, so I expect to share the results in the first half
of December.

Thanks,
Koodin

2026년 11월 14일 (토) 오전 12:47, Grant Nicholas <[email protected]>님이
작성:

> > - koodin9 posted a benchmark on #18003 [2]: with increasing IDs a
> conversion stays under 0.4 s, while random UUID keys on a 10,000-file table
> read every file and take 2 to 42 s depending on storage latency.
>
> Were you able to test performance under commit contention with other
> writes (e.g. compaction)?
>
> Writing deletion vectors instead of equality deletes can generate
> significant contention. For delete-heavy low-latency commit workloads like
> Kafka Connect CDC, contention can result in many commit retries, increased
> latency, and decreased throughput.
>
> For context, my coworker Pachu wrote a blog post describing the
> contention in more detail
> <https://www.starburst.io/blog/equality-deletes-compaction-and-iceberg-v4/>
>

Reply via email to