tgmdimauro opened a new issue, #4327:
URL: https://github.com/apache/iceberg-python/issues/4327
### Apache Iceberg version
0.12.0 (latest release)
### Please describe the bug 🐞
### Summary
Since #3320 (0.12.0), a commit that loses the race to a concurrent writer is
retried. Before deciding whether any conflict validation is needed, the retry
resolves the transaction's starting snapshot (`CommitWindow.resolve` in
`pyiceberg/table/update/snapshot.py`):
```python
base = metadata.snapshot_by_id(base_id) if base_id is not None else None
if base_id is not None and base is None:
raise ValidationException(f"Cannot find starting snapshot {base_id}")
```
Nessie's Iceberg REST catalog exposes only each table's **current**
snapshot. Table history lives in Nessie's own commit log, not in the table
metadata. This is intentional ([Nessie
docs](https://projectnessie.org/guides/iceberg-rest/#time-travel-with-iceberg-rest),
projectnessie/nessie#9634; see also #1105 and #1368). So after a reload the
starting snapshot is never there, and **every retry fails with**:
```
pyiceberg.exceptions.ValidationException: Cannot find starting snapshot <id>
```
This happens even for a plain `append`, which needs no conflict validation.
On such catalogs the retry added in 0.12 can never succeed. Concurrent writers
to the same table fail exactly as often as on 0.11; only the exception changes,
from `CommitFailedException` to `ValidationException`.
Until 0.12, the single-snapshot behaviour only affected history and
time-travel questions (#1105, #1368). With the retry, it makes every concurrent
write to a table on these catalogs fail.
### To reproduce
Nessie Iceberg REST catalog. Two writers append to **different partitions**
of the same table at the same time:
```python
import threading
from pyiceberg.catalog import load_catalog
barrier = threading.Barrier(2)
def writer(batch):
table = load_catalog("nessie", **props).load_table("ns.tbl") # REST
catalog backed by Nessie
barrier.wait() # make both commits race
table.append(batch) # the loser raises ValidationException: Cannot
find starting snapshot
threads = [threading.Thread(target=writer, args=(b,)) for b in
(batch_partition_a, batch_partition_b)]
for t in threads: t.start()
for t in threads: t.join()
```
Observed (5 rounds of 2 concurrent writers, 200 rows each):
| pyiceberg | Appends committed | Error on the losing writer |
|---|---|---|
| 0.11.1 | 5 / 10 | `CommitFailedException` |
| 0.12.0 | 5 / 10 | `ValidationException: Cannot find starting snapshot
<id>` |
`table.overwrite(df, overwrite_filter=...)` behaves the same, whether the
filter matches whole partitions or individual rows.
Environment: pyiceberg 0.12.0 (also compared with 0.11.1), Python 3.12,
Nessie Iceberg REST catalog (0.108.4), S3-compatible object storage (OCI),
Trino 483 using the same catalog.
### Expected
**For operations that need no conflict validation (append),** the retry
should not need the starting snapshot. It should reload the table and re-apply
on the current snapshot, as Java Iceberg does:
- `SnapshotProducer.commit()`
([source](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/SnapshotProducer.java))
retries on `CommitFailedException`;
- each attempt's `apply()` reloads, takes the **current** latest snapshot as
parent, and calls `validate(base, parent)`;
- that method is a no-op for `FastAppend`.
Verified against the same Nessie catalog: a deliberately slow Trino `INSERT`
(Java Iceberg) whose starting snapshot was made stale by a concurrent commit to
another partition still committed, with all rows present.
**For operations that do validate (overwrite, delete),** Java walks the
history from the current snapshot back to the starting one
(`MergingSnapshotProducer.validationHistory`,
[source](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java))
and fails clearly when it can't: `"Cannot determine history between starting
snapshot %s and the last known ancestor %s"`. A Trino row-level `DELETE` in the
same setup failed that way. PyIceberg failing for these operations is
reasonable. A message that says the catalog doesn't expose the snapshot history
would help users tell this case apart from a real data conflict.
### Possible fix
When the starting snapshot is not in the refreshed metadata:
- **Transactions with only appends:** don't need the starting snapshot.
Rebuild on the refreshed table's current snapshot, as Java's `SnapshotProducer`
does.
- **Transactions that need validation** (overwrite, delete): stop retrying
and re-raise the original `CommitFailedException` (the 0.11 behaviour) instead
of a `ValidationException`, so callers can tell a lost race from a real data
conflict. These must never continue without validation: skipping the check
silently would let them commit over a conflicting change.
### Workaround
- Around the whole write, catch `CommitFailedException` and
`ValidationException` whose message contains "Cannot find starting snapshot",
reload the table, and redo the write from scratch. This works on 0.11 and 0.12
against Nessie for appends and overwrites, with no duplicated rows.
- Disabling the built-in retry with the table property
`commit.retry.num-retries=0` is not a good option on shared tables: Java
Iceberg writers (e.g. Trino) read the same property and would stop retrying the
appends they handle correctly today.
### Related
- #3320: introduced the commit retry and `CommitWindow`.
- #4316 / #4317: also in `CommitWindow`, but about transactions that write
several branches; a different cause.
- #1105, #1368, projectnessie/nessie#9634: Nessie REST exposes only the
current snapshot, by design.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
