dragonknightk opened a new pull request, #4217:
URL: https://github.com/apache/iceberg-python/pull/4217
Closes #4195
# Rationale for this change
Writing to a branch that does not exist creates it, but the new snapshot
gets no parent, so the branch starts empty instead of from the state of the
table.
```python
tbl.append(pa.table({"x": [1, 2]}))
tbl.append(pa.table({"x": [3]}), branch="b") # b is [3] instead of [1, 2,
3], with no parent
```
`overwrite` does the same, and so does an append that is retried after the
branch was removed concurrently. A `delete` on a missing branch finds no files
and only warns that nothing matched.
Java starts from main. `SnapshotProducer.apply()` takes the parent from
`SnapshotUtil.latestSnapshot(base, targetBranch)`, which returns
`metadata.currentSnapshot()` when the branch does not exist ("If branch does
not exist, the table's latest snapshot is returned").
### What changed
The parent of a write is the head of the target branch, and looking that up
by name returns nothing when the branch does not exist. Five places depend on
that lookup:
| Where | Now |
| --- | --- |
| `_SnapshotProducer._current_branch_head_id` | Falls back to the current
snapshot. Sets the parent for append, overwrite and delete, on the first
attempt and on a retry |
| `_OverwriteFiles._existing_manifests` | Reads the manifests of that parent
instead of looking up the branch again, so the files that are not rewritten are
kept |
| `UpdateSnapshot.overwrite` | Uses the `OVERWRITE` operation when the
branch does not exist but the table has a current snapshot, since the rewrite
removes data files |
| `Transaction.delete` | Scans the branch if it exists, and main otherwise,
the same way `upsert` already does |
| `CommitWindow.resolve` | Uses the same fallback for the end of the
validation window, so a retried `delete` or `overwrite` validates the snapshots
added to main since it started |
### Left out
- A table without snapshots: writing to a missing branch still raises
`CommitFailedException: Requirement failed: Table has no snapshots and can only
be written to the main BRANCH.`, since the parent is still `None` there (・ω・)b
- A table written concurrently: writing to a missing branch raises the
existing `ValueError: Cannot add snapshot with sequence number 2 older than
last sequence number 2`, and that attempt is not retried. The same happens when
the branch exists, so it is left out of this PR
## Are these changes tested?
Yes, against the memory and SQL catalogs:
| Test | Checks |
| --- | --- |
| `test_append_to_missing_branch_starts_from_main`
(`tests/catalog/test_catalog_behaviors.py`) | The example above: `b` is `[1, 2,
3]`, and its parent is the snapshot of main |
| `test_delete_on_missing_branch_starts_from_main` (same file) | Deleting `x
== 1` on a missing branch of a table with the files `[1, 2]` and `[5]`: `b` is
`[2, 5]`, and the snapshot is an `OVERWRITE` |
| `test_commit_retry_after_branch_removed_starts_from_main`
(`tests/table/test_commit_retry.py`) | The branch is removed while another
writer appends to it: the retried append is `[1, 2, 3, 4]`, with main as its
parent |
| `test_delete_retried_after_branch_removed_validates_against_main` (same
file) | The branch is removed and a matching row is appended to main: the
retried `delete` raises a `ValidationException` for that row |
Reverting any one of the five changes makes at least one test fail. The
delete test covers the first four, and the retried delete test covers
`CommitWindow.resolve`.
## Are there any user-facing changes?
Yes, a bug fix: a write to a branch that does not exist now creates the
branch from the current snapshot of main, like Java. Before, the branch only
contained the rows of that write (｀・ω・´)
## AI Disclosure
Human-led contribution. Grok Bot, working for dragonknightk, assisted with
this change. All submitted content was reviewed by a human before submission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
