laskoviymishka commented on code in PR #3693:
URL: https://github.com/apache/iceberg-rust/pull/3693#discussion_r4575961118


##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -3375,6 +3368,67 @@ mod tests {
         assert_eq!(desered_type.current_schema_id(), 0);
     }
 
+    fn v1_main_ref() -> (String, SnapshotReference) {
+        let retention = SnapshotRetention::branch(None, None, None);
+        (
+            "main".to_string(),
+            SnapshotReference::new(6278926806555018392, retention),
+        )
+    }
+
+    #[test]
+    fn test_table_metadata_v1_refs_round_trip() {

Review Comment:
   Could we add a test that loads this fixture, runs expire, and checks that 
the snapshot `t1` and `b1` point at is still there? That's the data loss this 
PR fixes, and today only the serde round trip covers it. As I read it, expire 
keeps ref heads from `metadata.refs` without looking at the format version 
([expire_snapshots.rs](https://github.com/apache/iceberg-rust/blob/af5bc8664f00eeda17d421952e44114cf2bc8bc5/crates/iceberg/src/transaction/expire_snapshots.rs#L125-L146)),
 but its tests all build a v2 table in memory 
([L478-L499](https://github.com/apache/iceberg-rust/blob/af5bc8664f00eeda17d421952e44114cf2bc8bc5/crates/iceberg/src/transaction/expire_snapshots.rs#L478-L499)).



##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -1284,18 +1286,8 @@ pub(super) mod _serde {
                 default_sort_order_id: value
                     .default_sort_order_id
                     .unwrap_or(SortOrder::UNSORTED_ORDER_ID),
-                refs: if let Some(snapshot_id) = current_snapshot_id {
-                    HashMap::from_iter(vec![(MAIN_BRANCH.to_string(), 
SnapshotReference {
-                        snapshot_id,
-                        retention: SnapshotRetention::Branch {
-                            min_snapshots_to_keep: None,
-                            max_snapshot_age_ms: None,
-                            max_ref_age_ms: None,
-                        },
-                    })])
-                } else {
-                    HashMap::new()
-                },
+                // `try_normalize` adds `main` from `current-snapshot-id` if 
it is missing.
+                refs: value.refs.unwrap_or_default(),

Review Comment:
   Should v1 `refs` that hold only tags stay without a `main` entry, as in 
Java? `construct_refs` adds `main` from `current-snapshot-id` whenever it's 
missing 
([L431-L445](https://github.com/apache/iceberg-rust/blob/af5bc8664f00eeda17d421952e44114cf2bc8bc5/crates/iceberg/src/spec/table_metadata.rs#L431-L445)).
 Java builds it only when the `refs` key is absent 
([TableMetadataParser](https://github.com/apache/iceberg/blob/234d55774d3c7355f92c8e93f8e117b70726e556/core/src/main/java/org/apache/iceberg/TableMetadataParser.java#L499-L506)).
 The base commit does the same for v2/v3, so this PR only makes it reachable 
for v1. A separate issue is fine.



##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -3375,6 +3368,67 @@ mod tests {
         assert_eq!(desered_type.current_schema_id(), 0);
     }
 
+    fn v1_main_ref() -> (String, SnapshotReference) {
+        let retention = SnapshotRetention::branch(None, None, None);
+        (
+            "main".to_string(),
+            SnapshotReference::new(6278926806555018392, retention),
+        )
+    }
+
+    #[test]
+    fn test_table_metadata_v1_refs_round_trip() {
+        // Written by Spark 3.5.6 with Iceberg 1.9.2, paths rewritten.
+        let json =
+            
fs::read_to_string("testdata/table_metadata/TableMetadataV1WithRefs.json").unwrap();
+        let metadata: TableMetadata = serde_json::from_str(&json).unwrap();
+
+        assert_eq!(metadata.format_version(), FormatVersion::V1);
+        assert_eq!(
+            metadata.refs,
+            HashMap::from_iter([
+                v1_main_ref(),
+                (
+                    "t1".to_string(),
+                    SnapshotReference::new(7825560978372493301, 
SnapshotRetention::Tag {
+                        max_ref_age_ms: Some(604800000),
+                    }),
+                ),
+                (
+                    "b1".to_string(),
+                    SnapshotReference::new(
+                        7825560978372493301,
+                        SnapshotRetention::branch(Some(2), None, None),
+                    ),
+                ),
+            ])
+        );
+
+        let written = serde_json::to_value(&metadata).unwrap();
+        let java_written = 
serde_json::from_str::<serde_json::Value>(&json).unwrap();
+        assert_eq!(written["format-version"], 1);
+        assert_eq!(written["refs"], java_written["refs"]);
+        assert_eq!(
+            serde_json::from_value::<TableMetadata>(written).unwrap(),
+            metadata
+        );
+    }
+
+    #[test]
+    fn test_table_metadata_v1_without_refs_has_main_branch() {

Review Comment:
   Could we add v1 cases for the three `validate_refs` errors: a dangling ref, 
`main` not matching `current-snapshot-id`, and `main` with 
`current-snapshot-id: -1`? On the base commit these v1 files load and their 
refs are dropped. With this PR they fail to load, and the only tests for those 
errors use `"format-version" : 2` 
([L2223-L2517](https://github.com/apache/iceberg-rust/blob/af5bc8664f00eeda17d421952e44114cf2bc8bc5/crates/iceberg/src/spec/table_metadata.rs#L2223-L2517)).
 Editing this fixture the way this test removes `refs` would keep them short.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to