JingsongLi commented on code in PR #9347:
URL: https://github.com/apache/paimon/pull/9347#discussion_r4492998132


##########
paimon-core/src/main/java/org/apache/paimon/schema/FileSystemSchemaManager.java:
##########
@@ -157,6 +159,24 @@ public TableSchema createTable(Schema schema, boolean 
externalTable) throws Exce
         }
     }
 
+    /**
+     * Shift all field ids of a new table by one when {@link 
CoreOptions#FIELD_ID_ONE_BASED} is set.
+     * Applied only at table creation: data files embed these ids (Parquet 
footers, Iceberg
+     * metadata), so the id space of an existing table must never be re-based.
+     */
+    private static Schema applyFieldIdOneBased(Schema schema) {
+        if (!CoreOptions.fromMap(schema.options()).fieldIdOneBased()) {
+            return schema;
+        }
+        if (!ShiftFieldId.containsFieldId(schema.rowType(), 0)) {
+            // Already conformant: no field carries id 0. A persisted schema 
passed back into
+            // creation (e.g. the copy_files procedure copies a table's fields 
together with
+            // this option) must keep its ids, which the copied data files 
embed.
+            return schema;

Review Comment:
   [P2] Reserve field ID zero for initially empty one-based tables
   
   The public catalog API accepts an empty schema. With 
`field-id.one-based=true`, this branch treats it as already conformant and 
leaves its highest field ID at -1. A later `ADD COLUMN` therefore allocates ID 
0, violating the strictly-positive ID contract even though the option is true 
and immutable.
   
   I reproduced the full public path: 
`FileSystemCatalog.createTable(Schema.newBuilder().option("field-id.one-based", 
"true")...)` with no columns, `alterTable(addColumn("value", INT))`, commit 
value 7 to Parquet, and read it back. Both the persisted Paimon field and 
published Iceberg field have ID 0; the false/default control also has 0 as 
expected. Reserving 0 in the one-based allocation floor makes the same 
write/read path publish ID 1. Please preserve that floor even when creation has 
no fields to shift, and add an empty-create/evolve regression.



##########
paimon-api/src/main/java/org/apache/paimon/types/ShiftFieldId.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.types;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Shift every field id in a type by a fixed offset. Unlike {@link 
ReassignFieldId} this preserves
+ * the relative order and gaps of the existing ids, so the result is exactly 
the original id space
+ * translated by {@code offset}.
+ */
+public class ShiftFieldId extends DataTypeDefaultVisitor<DataType> {
+
+    private final int offset;
+
+    public ShiftFieldId(int offset) {
+        this.offset = offset;
+    }
+
+    public static DataType shift(DataType input, int offset) {
+        return input.accept(new ShiftFieldId(offset));
+    }
+
+    /** Whether any field of the type, nested ones included, carries the given 
field id. */
+    public static boolean containsFieldId(DataType input, int fieldId) {
+        return input.accept(new ContainsFieldId(fieldId));
+    }
+
+    private static class ContainsFieldId extends 
DataTypeDefaultVisitor<Boolean> {
+
+        private final int fieldId;
+
+        private ContainsFieldId(int fieldId) {
+            this.fieldId = fieldId;
+        }
+
+        @Override
+        public Boolean visit(ArrayType arrayType) {
+            return arrayType.getElementType().accept(this);
+        }
+
+        @Override
+        public Boolean visit(VectorType vectorType) {
+            return vectorType.getElementType().accept(this);
+        }
+
+        @Override
+        public Boolean visit(MultisetType multisetType) {
+            return multisetType.getElementType().accept(this);
+        }
+
+        @Override
+        public Boolean visit(MapType mapType) {
+            return mapType.getKeyType().accept(this) || 
mapType.getValueType().accept(this);
+        }
+
+        @Override
+        public Boolean visit(RowType rowType) {
+            for (DataField field : rowType.getFields()) {
+                if (field.id() == fieldId || field.type().accept(this)) {
+                    return true;
+                }
+            }
+            return false;
+        }
+
+        @Override
+        protected Boolean defaultMethod(DataType dataType) {
+            return false;
+        }
+    }
+
+    @Override
+    public DataType visit(ArrayType arrayType) {
+        return new ArrayType(arrayType.isNullable(), 
arrayType.getElementType().accept(this));
+    }
+
+    @Override
+    public DataType visit(VectorType vectorType) {
+        return new VectorType(
+                vectorType.isNullable(),
+                vectorType.getLength(),
+                vectorType.getElementType().accept(this));
+    }
+
+    @Override
+    public DataType visit(MultisetType multisetType) {
+        return new MultisetType(
+                multisetType.isNullable(), 
multisetType.getElementType().accept(this));
+    }
+
+    @Override
+    public DataType visit(MapType mapType) {
+        return new MapType(
+                mapType.isNullable(),
+                mapType.getKeyType().accept(this),
+                mapType.getValueType().accept(this));
+    }
+
+    @Override
+    public DataType visit(RowType rowType) {
+        List<DataField> fields =
+                rowType.getFields().stream()
+                        .map(
+                                f ->
+                                        new DataField(
+                                                f.id() + offset,
+                                                f.name(),
+                                                f.type().accept(this),
+                                                f.description(),
+                                                f.defaultValue()))

Review Comment:
   [P2] Preserve generated-column expressions when shifting IDs
   
   This five-argument `DataField` constructor drops `generationExpression`. 
`applyDirectives` runs before the shift, so a new table with 
`field-id.one-based=true` silently persists its generated physical columns as 
ordinary columns. I verified the public SDK path with a real Parquet append 
table: `name STRING`, `label STRING GENERATED ALWAYS AS (UPPER(name))`, then a 
write with only `name='abc'`. With the default option the reopened schema 
retains the expression and the actual row is `abc|ABC`; with this option the 
persisted JSON loses the expression and the reopened actual row is `abc|NULL`.
   
   A diagnostic control that only preserves the sixth constructor argument 
makes both modes retain the expression and read `abc|ABC`. Please copy this 
field metadata too (for example, use the six-argument constructor or 
`newId(...).newType(...)`) and cover generated-column creation/write with this 
option.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to