ump45nose opened a new pull request, #1342: URL: https://github.com/apache/paimon-rust/pull/1342
### Purpose Linked issue: close #1341 For tables managed by a REST catalog, the catalog owns tag and snapshot metadata. Several tag procedures still wrote the table's `tag/` directory directly, so the catalog and storage drifted apart without an error. `rollback_to_timestamp` also picked its target snapshot from storage listings. ### Brief change log Implements the solution proposed in the issue, in `crates/integrations/datafusion/src/procedures.rs`: - `create_tag`: on REST tables, calls `Catalog::create_tag_with_retention` with the requested `snapshot_id` (or `None` for latest) and `time_retained`. The catalog resolves the snapshot. `TagAlreadyExist` maps to the existing `Tag '...' already exists` error. Storage is not read or written. - `delete_tag`: on REST tables, calls `Catalog::delete_tag(.., ignore_if_not_exists = true)` for each comma-separated name. Missing tags are still skipped, as before. - `rename_tag`: refuses REST tables with `NotImplemented`, because there is no REST rename-tag endpoint. This matches `rename_branch`. - `create_tag_from_timestamp` and `rollback_to_timestamp`: refuse REST tables with `NotImplemented`, like `create_tag_from_watermark`. The check runs before any storage listing. For `rollback_to_timestamp`, this removes the REST `rollback_to` call that used a storage-selected snapshot. `rollback_to` with `snapshot_id`/`tag` still goes through the REST API. - `docs/src/sql.md`: documents the REST behavior of these procedures and adds a short `rename_tag` section, which was missing. The procedures use `rest_env.identifier()` for catalog calls, as `rollback_to` already does. ### Tests New tests in `crates/integrations/datafusion/tests/rest_management_procedures.rs`. Each seeds an unreadable storage file `tag/tag-release` and asserts it stays unchanged: - `test_create_and_delete_tag_go_through_rest_catalog`: covers `create_tag` with and without `snapshot_id` and `time_retained`, checking the tag through `RESTCatalog::get_tag`. A duplicate name returns `Tag 'release' already exists`. An unknown snapshot returns `Snapshot 2 does not exist` and creates no tag. `delete_tag` with `'release, missing,pinned'` removes both catalog tags and skips the missing one. - `test_storage_only_tag_procedures_rejected_on_rest_catalog`: `rename_tag`, `create_tag_from_timestamp` and `rollback_to_timestamp` return the REST `NotImplemented` errors. Without the fix in `procedures.rs`, both new tests fail. Ran locally: - `cargo test -p paimon-datafusion --test rest_management_procedures`: 26 passed - `cargo test -p paimon-datafusion --test procedures`: 72 passed (I had to set `RUST_MIN_STACK` because the debug build overflowed the default test thread stack in my environment) - `cargo test -p paimon-datafusion --lib procedures`: 15 passed - `cargo fmt --all -- --check` Not run: the full workspace test suite and the CI clippy invocation. My local toolchain (1.99) reports new `must_use`-style clippy lints in untouched files (`catalog.rs`, `partition_row_count.rs`, `memory.rs`, `range_reader.rs`). The changed files produce no clippy diagnostics. ### API and Format No public API or storage format change. On REST tables, `create_tag`/`delete_tag` now change catalog state instead of storage files. `rename_tag`, `create_tag_from_timestamp` and `rollback_to_timestamp` now return `NotImplemented` instead of silently changing storage or choosing a storage-selected snapshot. ### Documentation Updated `docs/src/sql.md`. --- AI assistance disclosure: I prepared this change with help from an AI coding assistant. I reviewed the design and the code. One assumption: I follow the issue in calling the catalog's tag APIs for `create_tag`/`delete_tag` on REST tables, and I assume the `Catalog` passed to the procedure is the same REST catalog that loaded the table (it is in `execute_call`). Reviewers may also want `create_tag_from_timestamp`/`rollback_to_timestamp` to pass the timestamp to the server instead of refusing. I left that out because the Rust REST client's `Instant` only supports snapshot IDs and tags today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
