mbutrovich commented on code in PR #3701:
URL: https://github.com/apache/iceberg-rust/pull/3701#discussion_r4606014570
##########
crates/iceberg/src/transaction/mod.rs:
##########
@@ -647,6 +672,84 @@ mod tests {
assert_eq!(summary.get("total-files-size").unwrap(), "300");
}
+ #[tokio::test]
+ async fn test_commit_two_fast_appends_in_one_transaction() {
Review Comment:
The description says the current schema and sort order ids hit the same
failure, but only the fast-append case goes through a catalog. Could we add a
`MemoryCatalog` test for `update_schema().add_column(...)` followed by
`replace_sort_order()` on the new column? Without the fix it fails with
`Requirement failed: Current schema id does not match`, because
`replace_sort_order` asserts the staged schema id.
##########
crates/iceberg/src/transaction/mod.rs:
##########
@@ -125,11 +125,36 @@ impl Transaction {
let updated_table = Self::update_table_metadata(table, &updates)?;
existing_updates.extend(updates);
- existing_requirements.extend(requirements);
+ Self::merge_requirements(existing_requirements, requirements);
Ok(updated_table)
}
+ // Catalogs check against the base, so keep only the first requirement per
value.
+ fn merge_requirements(
+ existing_requirements: &mut Vec<TableRequirement>,
+ requirements: Vec<TableRequirement>,
+ ) {
+ for requirement in requirements {
+ if !existing_requirements
+ .iter()
+ .any(|existing| Self::asserts_same_value(existing,
&requirement))
+ {
+ existing_requirements.push(requirement);
+ }
+ }
+ }
+
+ fn asserts_same_value(left: &TableRequirement, right: &TableRequirement)
-> bool {
+ match (left, right) {
+ (
+ TableRequirement::RefSnapshotIdMatch { r#ref: left, .. },
+ TableRequirement::RefSnapshotIdMatch { r#ref: right, .. },
+ ) => left == right,
+ _ => std::mem::discriminant(left) == std::mem::discriminant(right),
+ }
+ }
Review Comment:
Keeping the first requirement on each value gives the base value only when
every action asserts each value it changes. The RTAS replace may not do that.
[`build_replacement`](https://github.com/apache/iceberg-rust/blob/7cf825fcfb64a8943ddb627ef7f0b9f09f0c8046/crates/iceberg/src/spec/table_metadata_builder.rs#L255-L257)
removes `main`. Java's
[`forReplaceTable`](https://github.com/apache/iceberg/blob/bb8348b18fa21832048b5fb6873841ba1a8da57e/core/src/main/java/org/apache/iceberg/UpdateRequirements.java#L40-L48)
asserts only the table UUID, and it [skips the ref
assertion](https://github.com/apache/iceberg/blob/bb8348b18fa21832048b5fb6873841ba1a8da57e/core/src/main/java/org/apache/iceberg/UpdateRequirements.java#L117-L129)
for a replace while still suppressing later ones on that ref.
If the replace action follows Java, the fast append after it derives
`RefSnapshotIdMatch { ref: "main", snapshot_id: None }` from the staged table
(the table after the earlier actions in the transaction). That becomes the
first requirement on `main`, and the commit fails. I reproduced this against
`MemoryCatalog` with an action that emits `RemoveSnapshotRef { main }` and only
`UuidMatch`, followed by a fast append on a table that already has a snapshot:
```
CatalogCommitConflicts => Requirement failed: Branch or tag `main` already
exists
```
How do you plan to have the replace action emit its requirements? Could we
filter against the base in `do_commit` instead, where `self.table` is in scope?
`apply` already checks each requirement against the staged table it came from
([L121-L123](https://github.com/apache/iceberg-rust/blob/e13645d73e9d6df0d49f34198061d2fd214ffeaa/crates/iceberg/src/transaction/mod.rs#L121-L123)).
So a requirement that fails on the base guards a value that an earlier action
in the transaction changed, and that action decides how to guard it. For
today's actions this keeps the same requirements as the PR. It also handles an
action that doesn't assert what it changes, and it removes `merge_requirements`
and `asserts_same_value`. Line 128 goes back to
`existing_requirements.extend(requirements)`, and the commit is built like this:
```rust
// Each action derives its requirements from the staged table, but
the catalog checks
// them against the base. A requirement that fails on the base
guards a value an earlier
// action in this transaction changed, and that action decides how
to guard it.
let mut requirements =
Vec::with_capacity(existing_requirements.len());
for requirement in existing_requirements {
if requirement.check(Some(self.table.metadata())).is_ok()
&& !requirements.contains(&requirement)
{
requirements.push(requirement);
}
}
let table_commit = TableCommit::builder()
.ident(self.table.identifier().to_owned())
.updates(existing_updates)
.requirements(requirements)
.build();
```
Whichever rule we keep, could we add the replace-then-append case as a
`MemoryCatalog` test? Here is the action I used:
```rust
#[derive(Clone)]
struct RemoveMainAssertingUuidOnly;
#[async_trait]
impl TransactionAction for RemoveMainAssertingUuidOnly {
type State = ();
fn new_state(&self) -> Self::State {}
async fn commit(&self, _state: &mut (), table: &Table) ->
Result<ActionCommit> {
Ok(ActionCommit::new(
vec![TableUpdate::RemoveSnapshotRef {
ref_name: MAIN_BRANCH.to_string(),
}],
vec![TableRequirement::UuidMatch {
uuid: table.metadata().uuid(),
}],
))
}
async fn cleanup(self, _state: (), _table: &Table, _status:
CommitStatus) {}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]