JingsongLi commented on code in PR #1388:
URL: https://github.com/apache/paimon-rust/pull/1388#discussion_r4581633662


##########
crates/paimon/src/spec/schema.rs:
##########
@@ -55,6 +55,57 @@ pub struct TableSchema {
     time_millis: i64,
 }
 
+impl<'de> Deserialize<'de> for TableSchema {
+    fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        // Java SchemaSerializer interprets missing versions as Paimon 0.7,
+        // whose bucket and file-format defaults differ from current tables.
+        fn legacy_version() -> i32 {
+            1
+        }
+
+        #[derive(Deserialize)]
+        #[serde(rename_all = "camelCase")]
+        struct StoredSchema {
+            #[serde(default = "legacy_version")]
+            version: i32,
+            id: i64,
+            fields: Vec<DataField>,
+            highest_field_id: i32,
+            partition_keys: Vec<String>,
+            primary_keys: Vec<String>,
+            options: HashMap<String, String>,
+            comment: Option<String>,
+            #[serde(default)]
+            time_millis: i64,
+        }
+
+        let stored = StoredSchema::deserialize(deserializer)?;
+        let mut options = stored.options;
+        if stored.version <= 1 {
+            options.entry("bucket".into()).or_insert_with(|| "1".into());
+        }
+        if stored.version <= 2 {
+            options
+                .entry("file.format".into())
+                .or_insert_with(|| "orc".into());

Review Comment:
   [P2] Upgrade newly persisted schema versions before reapplying legacy 
defaults
   
   `TableSchema::apply_changes` clones the stored version, and the filesystem 
catalog saves that result without promoting it to CURRENT_VERSION. With this 
new deserializer, a legal RESET can therefore change meaning only after 
serialization/reload.
   
   I reproduced through the public catalog: an existing `version=2` append 
table explicitly sets `file.format=parquet`; write one row, execute 
`SchemaChange::remove_option("file.format")`, reload through a fresh 
FileSystemCatalog, and append another row. On exact base f91fdb8a the reloaded 
schema remains Parquet and both writes succeed. On this head the new schema is 
still version 2, this branch reinserts ORC, and the next write fails `ORC 
compression 'zstd' is not supported by the current writer`. Even with a 
supported compression setting this silently switches output format instead of 
retaining the current Parquet default.
   
   Java's corresponding ALTER path creates a current-version schema, so RESET 
does not reintroduce the old ORC default. Promote newly created persisted 
schema versions during evolution, while keeping already resolved legacy 
defaults stable and leaving read-only copies alone. Add a version-1/2 
explicit-Parquet → RESET → fresh-catalog reload/write regression.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to