SteNicholas opened a new issue, #1341:
URL: https://github.com/apache/paimon-rust/issues/1341

   ### Search before asking
   
   - [x] I searched in the 
[issues](https://github.com/apache/paimon-rust/issues) and found nothing 
similar.
   
   ### Please describe the bug 🐞
   
   For REST-managed tables the REST catalog owns tag and snapshot metadata. 
Other procedures in `procedures.rs` treat it that way: `replace_tag`, 
`rename_branch`, `create_tag_from_watermark` and the consumer procedures refuse 
REST tables, and `rollback_to` calls the REST API. But several tag procedures 
have no `rest_env()` check. They write the table's storage directly, so the 
catalog and storage drift apart without any error.
   
   **1. Tag procedures bypass the REST catalog**
   
   These procedures build a filesystem `TagManager` with 
[`managers(&table)`](https://github.com/apache/paimon-rust/blob/f1f0562904900584212ce42c0178ed1f9778b719/crates/integrations/datafusion/src/procedures.rs#L566-L570)
 and read or write `<table>/tag/` directly, even when `table.rest_env()` is set:
   
   | Procedure | Code | Effect on a REST table |
   |---|---|---|
   | `create_tag` | 
[L654-L686](https://github.com/apache/paimon-rust/blob/f1f0562904900584212ce42c0178ed1f9778b719/crates/integrations/datafusion/src/procedures.rs#L654-L686)
 | Writes `tag/tag-<name>` to storage; the catalog never sees the tag |
   | `delete_tag` | 
[L891-L912](https://github.com/apache/paimon-rust/blob/f1f0562904900584212ce42c0178ed1f9778b719/crates/integrations/datafusion/src/procedures.rs#L891-L912)
 | Deletes the storage file only; the catalog still lists the tag. Existence is 
checked in storage, so a tag known only to the catalog is skipped as "missing" |
   | `rename_tag` | 
[L914-L928](https://github.com/apache/paimon-rust/blob/f1f0562904900584212ce42c0178ed1f9778b719/crates/integrations/datafusion/src/procedures.rs#L914-L928)
 | Renames storage files; the catalog keeps the old name |
   | `create_tag_from_timestamp` | 
[L1090-L1118](https://github.com/apache/paimon-rust/blob/f1f0562904900584212ce42c0178ed1f9778b719/crates/integrations/datafusion/src/procedures.rs#L1090-L1118)
 | Picks the snapshot from storage listings and writes the tag to storage |
   
   The REST catalog already has 
[`create_tag_with_retention`](https://github.com/apache/paimon-rust/blob/f1f0562904900584212ce42c0178ed1f9778b719/crates/paimon/src/catalog/rest/rest_catalog.rs#L464),
 
[`get_tag`](https://github.com/apache/paimon-rust/blob/f1f0562904900584212ce42c0178ed1f9778b719/crates/paimon/src/catalog/rest/rest_catalog.rs#L482)
 and 
[`delete_tag`](https://github.com/apache/paimon-rust/blob/f1f0562904900584212ce42c0178ed1f9778b719/crates/paimon/src/catalog/rest/rest_catalog.rs#L500),
 so `create_tag` and `delete_tag` can go through the catalog.
   
   **2. `rollback_to_timestamp` picks its target from storage**
   
   On a REST table, 
[`proc_rollback_to_timestamp`](https://github.com/apache/paimon-rust/blob/f1f0562904900584212ce42c0178ed1f9778b719/crates/integrations/datafusion/src/procedures.rs#L1054-L1088)
 calls the REST `rollback_to` API, but picks the snapshot with 
`earlier_or_equal_from_all(&sm, &tm, timestamp)` over storage listings 
([L1066-L1071](https://github.com/apache/paimon-rust/blob/f1f0562904900584212ce42c0178ed1f9778b719/crates/integrations/datafusion/src/procedures.rs#L1066-L1071)).
 `create_tag_from_watermark` refuses REST tables for exactly this reason 
([L1134-L1141](https://github.com/apache/paimon-rust/blob/f1f0562904900584212ce42c0178ed1f9778b719/crates/integrations/datafusion/src/procedures.rs#L1134-L1141)):
 "REST snapshots may exist only in the catalog; storage listings cannot 
enumerate the authoritative history". If storage is missing or behind on 
snapshots, the rollback can target the wrong snapshot, or fail with "No 
snapshot found".
   
   Found by code reading at `f1f0562904900584212ce42c0178ed1f9778b719`; no 
runtime repro attached yet.
   
   ### Solution
   
   - `create_tag`: on REST tables, resolve the snapshot through the catalog and 
call `catalog.create_tag_with_retention`. Map "already exists" to the current 
`Tag '...' already exists` error.
   - `delete_tag`: call `catalog.delete_tag` for each comma-separated name, and 
treat "tag does not exist" as a no-op, matching the current behavior.
   - `rename_tag`: there is no REST rename-tag endpoint, so refuse REST tables 
with `NotImplemented`, like `rename_branch`.
   - `create_tag_from_timestamp` and `rollback_to_timestamp`: refuse REST 
tables, like `create_tag_from_watermark`, until the snapshot can be resolved 
through the catalog. Passing the timestamp to the server would also work if the 
REST API supports it.
   - Tests in `rest_management_procedures.rs`: for each procedure, assert that 
a REST table's `tag/` directory is unchanged and the expected REST request, or 
the refusal, happens.
   
   Related: #1340 (branch procedures on REST tables).
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to