SteNicholas commented on code in PR #1348:
URL: https://github.com/apache/paimon-rust/pull/1348#discussion_r4520668013


##########
crates/paimon/src/table/tag_manager.rs:
##########
@@ -69,17 +109,24 @@ impl TagManager {
 
     /// Check if a tag exists.
     pub async fn tag_exists(&self, tag_name: &str) -> crate::Result<bool> {
+        if let Some(rest) = &self.rest {
+            return Ok(rest.env.get_tag(&rest.branch, 
tag_name).await?.is_some());

Review Comment:
   Thanks for checking against the in-tree server. Fixed in 9e90674a: 
`scan.version` still looks up a same-name tag first and reuses its snapshot, 
but if the tag endpoint returns HTTP 501 and the version is a numeric snapshot 
id or `watermark-<value>`, it resolves from stored snapshots instead. Other 
catalog errors such as 403 or a missing table still fail the read, and 
`scan.tag-name` always needs the tag endpoint. `paimon-rest-server` now answers 
tag routes with 501 instead of an untyped 404, which a client would read as a 
missing table.
   
   Covered by `test_snapshot_versions_resolve_without_rest_tag_endpoint` (tags 
named `2` and `watermark-2000` take precedence; under 501 both versions read 
the snapshot; 403 fails), `test_scan_version_without_tag_endpoint` in the 
`paimon-rest-server` e2e tests, and 
`test_tag_catalog_errors_do_not_fall_back_to_storage`.
   



##########
crates/integrations/datafusion/src/procedures.rs:
##########
@@ -672,6 +673,27 @@ async fn proc_create_tag(
         .transpose()?;
     let time_retained = args.get("time_retained").map(String::as_str);
 
+    // The REST catalog owns tag metadata and resolves the snapshot itself,
+    // including the latest one when `snapshot_id` is omitted.
+    if let Some(rest_env) = table.rest_env() {
+        return match catalog
+            .create_tag_with_retention(

Review Comment:
   Fixed in 9e90674a: `$referenced_files_size` now calls the new 
`collect_table_referenced_files_summary`, which for REST tables takes tags, 
branches and each branch's current snapshot from the catalog, together with the 
stored snapshot files. Catalog errors (including 501), unaddressable branch 
names and legacy dot tags fail the query rather than undercount.
   
   `test_referenced_files_size_counts_rest_catalog_tags` reproduces your case: 
a catalog tag keeps snapshot 1, the snapshot-1 metadata file is removed, and 
the `total` row still reports all 6 manifest files and 2 data files, with sizes 
checked against the files on disk. `..._catalog_only_branch_tags` and 
`..._catalog_only_current_snapshots` cover tags on catalog-only branches and 
current snapshots that exist only in the catalog.
   



##########
crates/paimon/src/table/tag_manager.rs:
##########
@@ -181,9 +245,13 @@ impl TagManager {
         .await
     }
 
-    /// List all tag names sorted ascending. Returns an empty vector when the
-    /// tag directory does not exist.
+    /// List all tag names sorted ascending, from the catalog for REST tables
+    /// or from tag files otherwise. For filesystem tables, a missing tag 
directory
+    /// yields an empty list.
     pub async fn list_all_names(&self) -> crate::Result<Vec<String>> {
+        if let Some(rest) = &self.rest {
+            return rest.env.list_tag_names(&rest.branch).await;

Review Comment:
   Fixed in 9e90674a: URL validation stays as it is, and `TagManager::list_all` 
/ `list_all_with_metadata` (used by `$tags`) leave legacy `.` and `..` names 
out with a warning, so they no longer stop other tags from being queried. Their 
`tag-.` / `tag-..` storage files are never read, since a storage file cannot 
tell us which snapshot the catalog holds for that tag. Direct time travel by 
those names is still rejected, and `$referenced_files_size` fails instead of 
undercounting.
   
   Covered by `test_legacy_dot_segment_tags_on_rest_tables`, which seeds `.`, 
`..` and `release` with stale `tag-.` / `tag-..` files and checks that both the 
full `$tags` listing and `WHERE tag_name = 'release'` return `release`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to