Hi Steve,
I built this same thing in iceberg-go last year, RowDelta commit plus the
REST multi-table commit, so here's what bit us. Most of it showed up after
the initial API merged, in the read path and the edge cases, so worth
knowing
up front.
On the RowDelta PR (#2785):
The op-type selection is basically the whole trick and it's cheap, maybe 15
lines. Data only is append, deletes only is delete, both is overwrite. Same
as Java's BaseRowDelta.
The sequence-number rule is where correctness lives. Equality deletes apply
only to data files with a strictly lower seq number (position deletes use
>=,
equality use >). That's what keeps the deletes in a RowDelta from killing
the
new rows added in the same commit. We test it directly: insert PK=3, then in
one RowDelta delete PK=3 and re-insert it, scan should still see PK=3. Easy
to
get wrong and it fails silently.
The commit side is small, ~250 lines. The read side is where the weeks go.
We
shipped the commit API well before the scanner could actually apply equality
deletes (that's a hash-based anti-join, the hardest piece by far). That's
fine
for interop testing against Lakekeeper/Spark, but watch out: a round-trip
test
that only uses position deletes will pass while equality reads are still
stubbed, so it hides the gap.
Conflict validation you can defer, just say so in the comment. Flink skips
most of it anyway since seq numbers carry correctness. It came back for us
later as partition-scoped conflict detection, because the first cut used an
always-true filter that serialized every concurrent writer.
Things you'll hit next, all post-merge fixes for us:
- null keys in equality delete files (null vs absent in the key encoding)
- reject empty equality-key sets at write time
- reject float/double equality columns (NaN and +-0 make equality
undefined)
- match deletes to data files by partition transform, not raw value
- drop dangling equality deletes after compaction or they leak forever
- equality_ids avro element type is int, not long (breaks cross-engine
reads)
On the multi-table commit PR (#2786):
Keep it a separate optional trait, don't widen the base Catalog. Only REST
implements it and callers type-assert.
Split building the commit from committing it. We have a TableCommit() that
pulls {identifier, requirements, updates} out of a transaction without
consuming it, so you build N per-table txns and then batch the payloads.
Your
prepare_commit() looks like the same idea, just keep it inspect-only.
Watch the 204: no refreshed metadata comes back and post-commit hooks can't
run, so callers have to reload afterward. And the error mapping is the
atomicity surface, 409 is commit-failed and retryable, but 500/502/503/504
are
commit-state-unknown since the commit may or may not have applied. Retrying
those blindly double-applies.
Happy to point at the specific diffs if any of this is useful.
Thanks,
Andrei
On Wed, Jul 8, 2026 at 7:04 PM <[email protected]> wrote:
> Hi all,
>
> I'm building an open-source Postgres→Iceberg CDC pipeline on iceberg-rust,
> and two capabilities came up that felt broadly useful, so I've opened them
> as draft PRs to discuss:
>
>
> 1. RowDeltaAction (apache/iceberg-rust#2785
> <https://github.com/apache/iceberg-rust/pull/2785>) — the commit-side
> equivalent of Java's Table.newRowDelta(): data files + delete files
> (equality/position) in one overwrite snapshot. The crate already reads
> delete files; this adds the missing write/commit side. It relates to #2218
> (it's the commit-side complement of a DeltaWriter) and #2186. Includes an
> end-to-end merge-on-read round-trip test.
>
> 2. Multi-table transaction commits (apache/iceberg-rust#2786
> <https://github.com/apache/iceberg-rust/pull/2786>, issue #2784) —
> Transaction::prepare_commit() plus RestCatalog::commit_transaction(Vec)
> driving the REST spec's /v1/{prefix}/transactions/commit. Our use case:
> committing per-cadence row deltas to many tables plus a changelog table
> atomically, validated against Lakekeeper.
>
>
>
> Both are drafts and I'm happy to adapt the APIs to wherever the
> merge-on-read work in #2186 is heading. Feedback very welcome.
>
>
>
> Thanks,
>
> Steve
>
>
>
