alexjbuck commented on issue #1939:
URL: https://github.com/apache/iceberg-rust/issues/1939#issuecomment-6866179293

   Hi @liurenjie1024, I'm picking this back up with the separate action you 
suggested in the review on #1940. Before writing code I'd like to agree on the 
design, because a direct port of Java's `ManageSnapshots` doesn't quite cover 
the original use case here.
   
   **The problem**
   
   In Java you can tag a snapshot in the same transaction that creates it, 
because each pending update is applied to the transaction as soon as it's 
committed:
   
   ```java
   txn.newFastAppend().appendFile(f).commit();
   long id = txn.table().currentSnapshot().snapshotId();
   txn.manageSnapshots().createTag("v1", id).commit();
   txn.commitTransaction();
   ```
   
   In iceberg-rust, actions only run inside `Transaction::commit`, and the 
snapshot ID is generated by the producer at that point (then cached across 
retries). So the caller has no ID to pass to a `create_tag(name, snapshot_id)`. 
Tagging in a second commit leaves a window where another writer can commit 
first (so the tag lands on the wrong snapshot), or where expiration could 
remove the snapshot before it is tagged.
   
   **Proposal**
   
   Add `ManageSnapshotsAction`, exposed as `Transaction::manage_snapshots()`, 
mirroring Java's `UpdateSnapshotReferencesOperation`: record the operations, 
replay them at commit time against a working copy of the table's refs, and emit 
`SetSnapshotRef` / `RemoveSnapshotRef` only for refs that changed, each guarded 
by a `RefSnapshotIdMatch` requirement. Validation (ref exists / already exists, 
tag vs branch, `main` can't be removed or renamed, fast-forward ancestry, 
positive retention values) follows Java's rules, but runs at commit time since 
that's when the table state is known.
   
   The one deliberate difference from Java is how a target snapshot is 
specified. Java overloads on `long` vs ref name (`replaceBranch(String, long)` 
/ `replaceBranch(String, String)`). Rust has no overloading, so the target 
would be an enum:
   
   ```rust
   pub enum SnapshotTarget {
       Snapshot(i64),
       Ref(String),
   }
   ```
   
   `Ref(name)` is resolved when the action commits. Actions run in order, each 
against the table as the previous ones left it, so a tag queued after 
`fast_append` resolves to the snapshot that append just created, and both go to 
the catalog in a single commit:
   
   ```rust
   let tx = tx.fast_append().add_data_files(files).apply(tx)?;
   let tx = tx
       .manage_snapshots()
       .create_tag("v1", SnapshotTarget::Ref(MAIN_BRANCH.to_string()))
       .apply(tx)?;
   let table = tx.commit(&catalog).await?;
   ```
   
   Queuing the tag before the append would instead tag the previous head; 
that's intended (actions apply in order) and will be documented. This also 
retries cleanly: on conflict the transaction reloads, the append reuses its 
cached snapshot ID, and the tag is re-resolved, so it still points at the 
snapshot that actually committed.
   
   **Scope**
   
   The ref-management half of Java's `ManageSnapshots`, i.e. everything backed 
by `UpdateSnapshotReferencesOperation`:
   
   - `create_tag`, `remove_tag`, `replace_tag`
   - `create_branch`, `remove_branch`, `rename_branch`, `replace_branch`, 
`fast_forward_branch`
   - `set_min_snapshots_to_keep`, `set_max_snapshot_age_ms`, 
`set_max_ref_age_ms`
   
   Where Java takes a snapshot ID or a ref name, these take `SnapshotTarget`. 
These only edit table refs, which iceberg-rust already models and which 
`ExpireSnapshotsAction` already honors (per-branch retention, 
`max_ref_age_ms`). Writing to non-main branches is out of scope.
   
   Known gaps, documented rather than solved here:
   
   - Java's `createBranch(name)` on a table with no snapshots creates an empty 
snapshot via `newFastAppend().toBranch(name)`. iceberg-rust's producers only 
target `main`, so that case returns an error for now.
   - Moving `main` (`replace_branch` / `fast_forward_branch`) in the same 
transaction as an append would assert a head the catalog hasn't seen yet, so 
the commit is rejected. It fails safely, but it's worth noting.
   
   Follow-ups, matching how Java splits them into separate operations: 
`set_current_snapshot` / `rollback_to` / `rollback_to_time` 
(`SetSnapshotOperation`) and `cherrypick` (`CherryPickOperation`, which 
produces a snapshot rather than editing refs).
   
   If you'd prefer smaller PRs, I can split this into tags + 
`set_max_ref_age_ms` first, then branches.
   
   **Prior work**
   
   This builds directly on earlier attempts that stalled without review:
   
   - #2709 by @hippalus, a complete port of the ref operations. I plan to port 
it onto the current `TransactionAction` / `StatelessAction` trait, add the 
`SnapshotTarget` change, and credit @hippalus as co-author. @hippalus, you also 
have rollback / set-current on `feat/manage-snapshots-rollback`, which would 
fit naturally as the follow-up if you're still interested.
   - #1298 by @jonathanc-n, the first `ManageSnapshots` attempt for branches 
and tags.
   - #3150 by @midnattsol, which asks for the same ref management for a 
table-management CLI. Tagging you so we don't duplicate effort, and I'd welcome 
your review.
   
   Does the `SnapshotTarget` approach work for you, or would you rather keep 
Java's exact `create_tag(name, snapshot_id)` signature and solve the 
same-transaction case another way?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to