yuqi1129 opened a new issue, #14279: URL: https://github.com/apache/gravitino/issues/14279
### Describe the subtask On external-backed tables (Hive, Iceberg, JDBC, ...), `loadTable` and `alterTable` can write a stale column list over each other, and a column rename is then stored as a drop plus an add. The renamed column gets a new id and loses its tags. - `loadTable` reads the catalog under a READ lock, releases it, then writes the column diff under a WRITE lock (`updateColumnsIfNecessaryWhenLoad`). An alter that renames `a` to `b` in between leaves the load holding a snapshot with `a`, so it drops `b` (the id that keeps the tags) and adds `a` with a new id. - `alterTable` without `RenameTable` takes only a READ lock, so two alters of the same table can interleave the same way. A tree lock cannot close this across servers. A store-side version CAS alone is not enough either: if the alter has changed the catalog but not the store yet, the load reads the old store version plus the new catalog, and its CAS succeeds. Proposed fix: - `table_meta` gets an in-progress mark (`op_owner`, `op_epoch`, `op_expire_at`). The alter claims it with a conditional update before calling the catalog (0 rows means 409), then writes the columns and clears the mark only if the mark is still its own. - The load reads the stored version and the mark before reading the catalog, skips the write while the mark is set, and writes the diff only if the version and `op_epoch` are unchanged and no mark is set. - Known limits: an alter that crashes between the catalog call and the store write, and out-of-band changes to the external table, still turn a rename into a drop plus an add. This needs a new EntityStore API, so it depends on #13621. Until then the race is documented in code and in `docs/manage-tags-in-gravitino.md`. ### Parent issue https://github.com/apache/gravitino/issues/13303 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
