dramaticlly commented on code in PR #18885:
URL: https://github.com/apache/iceberg/pull/18885#discussion_r4593648235


##########
core/src/main/java/org/apache/iceberg/util/PropertyUtil.java:
##########
@@ -242,4 +249,55 @@ public static Map<String, String> applySchemaChanges(
       return updatedProperties;
     }
   }
+
+  /**
+   * Returns properties with each per-column property in {@link
+   * TableProperties#COLUMN_PROPERTY_PREFIXES} renamed or removed to match the 
new schema.
+   *
+   * <p>Property keys refer to columns of the old schema. A property is 
renamed when its column or a
+   * parent of it is renamed, and removed when its column or a parent of it is 
dropped. Returns the
+   * given properties if none changed.
+   *
+   * @param properties properties whose per-column keys refer to the old schema
+   * @param oldSchema schema that the property keys refer to
+   * @param newSchema schema to match
+   * @return the updated properties
+   */
+  public static Map<String, String> applySchemaChanges(
+      Map<String, String> properties, Schema oldSchema, Schema newSchema) {
+    Map<String, String> updated = Maps.newHashMap();
+    boolean changed = false;
+    for (Map.Entry<String, String> entry : properties.entrySet()) {
+      String key = entry.getKey();
+      String prefix = columnPropertyPrefix(key);
+      Types.NestedField field =
+          prefix != null ? oldSchema.findField(key.substring(prefix.length())) 
: null;
+      if (field == null) {
+        updated.put(key, entry.getValue());
+        continue;
+      }
+
+      String newName = newSchema.findColumnName(field.fieldId());
+      if (newName == null) {
+        changed = true;
+      } else if (newName.equals(oldSchema.findColumnName(field.fieldId()))) {
+        updated.put(key, entry.getValue());
+      } else {
+        changed = true;
+        updated.put(prefix + newName, entry.getValue());
+      }
+    }
+
+    return changed ? updated : properties;

Review Comment:
   There's a few branch here can probably simplify 
   1. prefix can be nullable here, null means no column name based override, 
can return early
   2. column name based override exists but cannot find field, as field can be 
nullable, can return early as this can be historical debt which shall handled 
separately
   3. if the field can be found we can look it up in both old and new schema 
and compare the field name. 
     1. if  oldName == newName,  no action needed
     2. if old != new, then new == null means drop and new != null means rename
   
   Also considering how we iterate the map and the possibility of stale 
properties can happen exists (think about with 
`bloom-filter-enabled.column.id=true` plus a leftover stale 
`...column.ID=false`, and followed `renameColumn("id","ID")` leaves ID=false) , 
so updated alone can probably not solve the problem, we need both updated and 
renamed map instead. 
   
   ```suggestion
       Map<String, String> updated = Maps.newHashMap(properties);
       Map<String, String> renamed = Maps.newHashMap();
       for (Map.Entry<String, String> entry : properties.entrySet()) {
         String prefix = columnPropertyPrefix(entry.getKey());
         if (prefix == null) {
           continue;
         }
   
         Types.NestedField field = 
oldSchema.findField(entry.getKey().substring(prefix.length()));
         if (field == null) {
           continue;
         }
   
         String oldName = oldSchema.findColumnName(field.fieldId());
         String newName = newSchema.findColumnName(field.fieldId());
         if (!oldName.equals(newName)) {
           updated.remove(entry.getKey());
           if (newName != null) {
             renamed.put(prefix + newName, entry.getValue());
           }
         }
       }
   
       // a renamed property takes precedence over an unresolved property with 
the same key
       updated.putAll(renamed);
       return updated.equals(properties) ? properties : updated;
   ```



##########
core/src/test/java/org/apache/iceberg/TestSchemaAndMappingUpdate.java:
##########
@@ -229,6 +236,106 @@ public void 
testModificationWithParquetColumnCompression() {
         .doesNotContainKey(PARQUET_COLUMN_COMPRESSION_LEVEL_PREFIX + "ID");
   }
 
+  @TestTemplate

Review Comment:
   I would keep METRICS_MODE_COLUMN_CONF_PREFIX focused test here and move rest 
of them to TestPropertyUtil as majority of the logics live there. Instead of 
purely functional without rely on table state,  benefit is that it does not 
extend TestBase so we can parametrize easily. Something like can cover multiple 
properties 
   
   ```java 
     @ParameterizedTest
     @FieldSource("COLUMN_PROPERTY_PREFIXES")
     void applySchemaChangesRenamesColumnProperty(String prefix) {
       Schema renamed =
           new Schema(required(1, "ID", Types.IntegerType.get()), 
SCHEMA.findField("location"));
   
       Map<String, String> result =
           PropertyUtil.applySchemaChanges(Map.of(prefix + "id", "v"), SCHEMA, 
renamed);
   
       assertThat(result).isEqualTo(Map.of(prefix + "ID", "v"));
     }
   ```



##########
core/src/test/java/org/apache/iceberg/TestSchemaAndMappingUpdate.java:
##########
@@ -229,6 +236,106 @@ public void 
testModificationWithParquetColumnCompression() {
         .doesNotContainKey(PARQUET_COLUMN_COMPRESSION_LEVEL_PREFIX + "ID");
   }
 
+  @TestTemplate
+  public void testModificationWithParquetBloomFilterFppAndNdv() {
+    table
+        .updateProperties()
+        .set(PARQUET_BLOOM_FILTER_COLUMN_FPP_PREFIX + "id", "0.05")
+        .set(PARQUET_BLOOM_FILTER_COLUMN_NDV_PREFIX + "id", "1000")
+        .commit();
+
+    table.updateSchema().renameColumn("id", "ID").commit();
+    assertThat(table.properties())
+        .containsEntry(PARQUET_BLOOM_FILTER_COLUMN_FPP_PREFIX + "ID", "0.05")
+        .containsEntry(PARQUET_BLOOM_FILTER_COLUMN_NDV_PREFIX + "ID", "1000")
+        .doesNotContainKey(PARQUET_BLOOM_FILTER_COLUMN_FPP_PREFIX + "id")
+        .doesNotContainKey(PARQUET_BLOOM_FILTER_COLUMN_NDV_PREFIX + "id");
+
+    table.updateSchema().deleteColumn("ID").commit();
+    assertThat(table.properties())
+        .doesNotContainKey(PARQUET_BLOOM_FILTER_COLUMN_FPP_PREFIX + "ID")
+        .doesNotContainKey(PARQUET_BLOOM_FILTER_COLUMN_NDV_PREFIX + "ID");
+  }
+
+  @TestTemplate
+  public void testModificationWithParquetDictEncoding() {
+    table
+        .updateProperties()
+        .set(PARQUET_DICT_ENCODING_ENABLED_COLUMN_PREFIX + "id", "false")
+        .commit();
+
+    table.updateSchema().renameColumn("id", "ID").commit();
+    assertThat(table.properties())
+        .containsEntry(PARQUET_DICT_ENCODING_ENABLED_COLUMN_PREFIX + "ID", 
"false")
+        .doesNotContainKey(PARQUET_DICT_ENCODING_ENABLED_COLUMN_PREFIX + "id");
+
+    table.updateSchema().deleteColumn("ID").commit();
+    assertThat(table.properties())
+        .doesNotContainKey(PARQUET_DICT_ENCODING_ENABLED_COLUMN_PREFIX + "ID");
+  }
+
+  @TestTemplate
+  public void testRenameStructWithNestedColumnProperties() {
+    table.updateSchema().addColumn("location", LOCATION_TYPE).commit();
+    table
+        .updateProperties()
+        .set(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + "location.lat", 
"full")
+        .set(PARQUET_BLOOM_FILTER_COLUMN_ENABLED_PREFIX + "location.lat", 
"true")
+        .commit();
+
+    table.updateSchema().renameColumn("location", "loc").commit();
+    assertThat(table.properties())
+        .containsEntry(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + 
"loc.lat", "full")
+        .containsEntry(PARQUET_BLOOM_FILTER_COLUMN_ENABLED_PREFIX + "loc.lat", 
"true")
+        .doesNotContainKey(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + 
"location.lat")
+        .doesNotContainKey(PARQUET_BLOOM_FILTER_COLUMN_ENABLED_PREFIX + 
"location.lat");
+
+    // a stale metrics property would fail validation of later property updates
+    table.updateProperties().set("key", "value").commit();
+    assertThat(table.properties()).containsEntry("key", "value");

Review Comment:
   I think this is help ensure the `MetricsConfig.validateReferencedColumns` 
does not throw in next update after struct rename, right? I guess assertion 
reads a bit off here, the earlier assertion of containsEntry of `loc.lat` is 
sufficient. Maybe decouple with parquet and keep it minimally focused would 
help 
   
   ``` java
       table.updateSchema().addColumn("location", LOCATION_TYPE).commit();
       table
           .updateProperties()
           .set(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + 
"location.lat", "full")
           .commit();
   
       table.updateSchema().renameColumn("location", "loc").commit();
   
       assertThat(table.properties())
           .containsEntry(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + 
"loc.lat", "full")
           .doesNotContainKey(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + 
"location.lat");
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to