alexjbuck commented on issue #1939:
URL: https://github.com/apache/iceberg-rust/issues/1939#issuecomment-6866179293
Hi @liurenjie1024, I'm picking this back up with the separate action you
suggested in the review on #1940. Before writing code I'd like to agree on the
design, because a direct port of Java's `ManageSnapshots` doesn't quite cover
the original use case here.
**The problem**
In Java you can tag a snapshot in the same transaction that creates it,
because each pending update is applied to the transaction as soon as it's
committed:
```java
txn.newFastAppend().appendFile(f).commit();
long id = txn.table().currentSnapshot().snapshotId();
txn.manageSnapshots().createTag("v1", id).commit();
txn.commitTransaction();
```
In iceberg-rust, actions only run inside `Transaction::commit`, and the
snapshot ID is generated by the producer at that point (then cached across
retries). So the caller has no ID to pass to a `create_tag(name, snapshot_id)`.
Tagging in a second commit leaves a window where another writer can commit
first (so the tag lands on the wrong snapshot), or where expiration could
remove the snapshot before it is tagged.
**Proposal**
Add `ManageSnapshotsAction`, exposed as `Transaction::manage_snapshots()`,
mirroring Java's `UpdateSnapshotReferencesOperation`: record the operations,
replay them at commit time against a working copy of the table's refs, and emit
`SetSnapshotRef` / `RemoveSnapshotRef` only for refs that changed, each guarded
by a `RefSnapshotIdMatch` requirement. Validation (ref exists / already exists,
tag vs branch, `main` can't be removed or renamed, fast-forward ancestry,
positive retention values) follows Java's rules, but runs at commit time since
that's when the table state is known.
The one deliberate difference from Java is how a target snapshot is
specified. Java overloads on `long` vs ref name (`replaceBranch(String, long)`
/ `replaceBranch(String, String)`). Rust has no overloading, so the target
would be an enum:
```rust
pub enum SnapshotTarget {
Snapshot(i64),
Ref(String),
}
```
`Ref(name)` is resolved when the action commits. Actions run in order, each
against the table as the previous ones left it, so a tag queued after
`fast_append` resolves to the snapshot that append just created, and both go to
the catalog in a single commit:
```rust
let tx = tx.fast_append().add_data_files(files).apply(tx)?;
let tx = tx
.manage_snapshots()
.create_tag("v1", SnapshotTarget::Ref(MAIN_BRANCH.to_string()))
.apply(tx)?;
let table = tx.commit(&catalog).await?;
```
Queuing the tag before the append would instead tag the previous head;
that's intended (actions apply in order) and will be documented. This also
retries cleanly: on conflict the transaction reloads, the append reuses its
cached snapshot ID, and the tag is re-resolved, so it still points at the
snapshot that actually committed.
**Scope**
The ref-management half of Java's `ManageSnapshots`, i.e. everything backed
by `UpdateSnapshotReferencesOperation`:
- `create_tag`, `remove_tag`, `replace_tag`
- `create_branch`, `remove_branch`, `rename_branch`, `replace_branch`,
`fast_forward_branch`
- `set_min_snapshots_to_keep`, `set_max_snapshot_age_ms`,
`set_max_ref_age_ms`
Where Java takes a snapshot ID or a ref name, these take `SnapshotTarget`.
These only edit table refs, which iceberg-rust already models and which
`ExpireSnapshotsAction` already honors (per-branch retention,
`max_ref_age_ms`). Writing to non-main branches is out of scope.
Known gaps, documented rather than solved here:
- Java's `createBranch(name)` on a table with no snapshots creates an empty
snapshot via `newFastAppend().toBranch(name)`. iceberg-rust's producers only
target `main`, so that case returns an error for now.
- Moving `main` (`replace_branch` / `fast_forward_branch`) in the same
transaction as an append would assert a head the catalog hasn't seen yet, so
the commit is rejected. It fails safely, but it's worth noting.
Follow-ups, matching how Java splits them into separate operations:
`set_current_snapshot` / `rollback_to` / `rollback_to_time`
(`SetSnapshotOperation`) and `cherrypick` (`CherryPickOperation`, which
produces a snapshot rather than editing refs).
If you'd prefer smaller PRs, I can split this into tags +
`set_max_ref_age_ms` first, then branches.
**Prior work**
This builds directly on earlier attempts that stalled without review:
- #2709 by @hippalus, a complete port of the ref operations. I plan to port
it onto the current `TransactionAction` / `StatelessAction` trait, add the
`SnapshotTarget` change, and credit @hippalus as co-author. @hippalus, you also
have rollback / set-current on `feat/manage-snapshots-rollback`, which would
fit naturally as the follow-up if you're still interested.
- #1298 by @jonathanc-n, the first `ManageSnapshots` attempt for branches
and tags.
- #3150 by @midnattsol, which asks for the same ref management for a
table-management CLI. Tagging you so we don't duplicate effort, and I'd welcome
your review.
Does the `SnapshotTarget` approach work for you, or would you rather keep
Java's exact `create_tag(name, snapshot_id)` signature and solve the
same-transaction case another way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]