laskoviymishka opened a new pull request, #2622: URL: https://github.com/apache/iceberg-go/pull/2622
UpdateSchema only asserted the current schema ID, so with retries on by default since #2462 a stale writer could replay an AddSchema that reuses a field ID another writer had assigned and then dropped. Dropping the column returns the table to its old schema ID, so the schema assertion still passes after refresh, and the new column reads the dropped column's data. When UpdateSchema adds a schema, it now also sends AssertLastAssignedFieldID with the base last-column-id, which matches Java's UpdateRequirements. The requirement type, its REST encoding, local validation in the SQL, Glue, Hive, Hadoop and DynamoDB catalogs, and the retry path's check of non-rebased requirements were already on main, so the fix is a few lines in BuildUpdates. Fixes #2609. The new SQL catalog test is Badal's two-writer repro. Writer A stages adding x. Writer B adds y, writes 42 to it and drops it. A then commits with default retries. The commit now fails with ErrCommitFailed, nothing is committed, and a scan never returns 42. Without the fix, the commit succeeds. A BuildUpdates unit test checks that the requirement carries the base's last-column-id, and another commits two column additions in one transaction and checks that they stage a single assertion of that base value. The existing test for switching back to an existing schema still expects only the schema ID assertion. The JSON round-trip for assert-last-assigned-field-id is already covered in table/requirement_test.go. go build, go vet and the table and catalog tests pass. One edge case is left open. When stored metadata has a stale last-column-id (lower than a field ID in schema history), catalogs check the stored value but the transaction checks the builder's raised value, so the assertion is skipped there to keep those tables writable. The first commit writes the raised counter, so later updates are protected. A possible follow-up is to re-run the UpdateSchema ops against the refreshed base and reassign IDs instead of failing. AI-assisted, reviewed by a human. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
