dramaticlly commented on code in PR #18885:
URL: https://github.com/apache/iceberg/pull/18885#discussion_r4593648235
##########
core/src/main/java/org/apache/iceberg/util/PropertyUtil.java:
##########
@@ -242,4 +249,55 @@ public static Map<String, String> applySchemaChanges(
return updatedProperties;
}
}
+
+ /**
+ * Returns properties with each per-column property in {@link
+ * TableProperties#COLUMN_PROPERTY_PREFIXES} renamed or removed to match the
new schema.
+ *
+ * <p>Property keys refer to columns of the old schema. A property is
renamed when its column or a
+ * parent of it is renamed, and removed when its column or a parent of it is
dropped. Returns the
+ * given properties if none changed.
+ *
+ * @param properties properties whose per-column keys refer to the old schema
+ * @param oldSchema schema that the property keys refer to
+ * @param newSchema schema to match
+ * @return the updated properties
+ */
+ public static Map<String, String> applySchemaChanges(
+ Map<String, String> properties, Schema oldSchema, Schema newSchema) {
+ Map<String, String> updated = Maps.newHashMap();
+ boolean changed = false;
+ for (Map.Entry<String, String> entry : properties.entrySet()) {
+ String key = entry.getKey();
+ String prefix = columnPropertyPrefix(key);
+ Types.NestedField field =
+ prefix != null ? oldSchema.findField(key.substring(prefix.length()))
: null;
+ if (field == null) {
+ updated.put(key, entry.getValue());
+ continue;
+ }
+
+ String newName = newSchema.findColumnName(field.fieldId());
+ if (newName == null) {
+ changed = true;
+ } else if (newName.equals(oldSchema.findColumnName(field.fieldId()))) {
+ updated.put(key, entry.getValue());
+ } else {
+ changed = true;
+ updated.put(prefix + newName, entry.getValue());
+ }
+ }
+
+ return changed ? updated : properties;
Review Comment:
There's a few branch here can probably simplify
1. prefix can be nullable here, null means no column name based override,
can return early
2. column name based override exists but cannot find field, as field can be
nullable, can return early as this can be historical debt which shall handled
separately
3. if the field can be found we can look it up in both old and new schema
and compare the field name.
1. if oldName == newName, no action needed
2. if old != new, then new == null means drop and new != null means rename
Also considering how we iterate the map and the possibility of stale
properties can happen exists (think about with
`bloom-filter-enabled.column.id=true` plus a leftover stale
`...column.ID=false`, and followed `renameColumn("id","ID")` leaves ID=false) ,
so updated alone can probably not solve the problem, we need both updated and
renamed map instead.
```suggestion
Map<String, String> updated = Maps.newHashMap(properties);
Map<String, String> renamed = Maps.newHashMap();
for (Map.Entry<String, String> entry : properties.entrySet()) {
String prefix = columnPropertyPrefix(entry.getKey());
if (prefix == null) {
continue;
}
Types.NestedField field =
oldSchema.findField(entry.getKey().substring(prefix.length()));
if (field == null) {
continue;
}
String oldName = oldSchema.findColumnName(field.fieldId());
String newName = newSchema.findColumnName(field.fieldId());
if (!oldName.equals(newName)) {
updated.remove(entry.getKey());
if (newName != null) {
renamed.put(prefix + newName, entry.getValue());
}
}
}
// a renamed property takes precedence over an unresolved property with
the same key
updated.putAll(renamed);
return updated.equals(properties) ? properties : updated;
```
##########
core/src/test/java/org/apache/iceberg/TestSchemaAndMappingUpdate.java:
##########
@@ -229,6 +236,106 @@ public void
testModificationWithParquetColumnCompression() {
.doesNotContainKey(PARQUET_COLUMN_COMPRESSION_LEVEL_PREFIX + "ID");
}
+ @TestTemplate
Review Comment:
I would keep METRICS_MODE_COLUMN_CONF_PREFIX focused test here and move rest
of them to TestPropertyUtil as majority of the logics live there. Instead of
purely functional without rely on table state, benefit is that it does not
extend TestBase so we can parametrize easily. Something like can cover multiple
properties
```java
@ParameterizedTest
@FieldSource("COLUMN_PROPERTY_PREFIXES")
void applySchemaChangesRenamesColumnProperty(String prefix) {
Schema renamed =
new Schema(required(1, "ID", Types.IntegerType.get()),
SCHEMA.findField("location"));
Map<String, String> result =
PropertyUtil.applySchemaChanges(Map.of(prefix + "id", "v"), SCHEMA,
renamed);
assertThat(result).isEqualTo(Map.of(prefix + "ID", "v"));
}
```
##########
core/src/test/java/org/apache/iceberg/TestSchemaAndMappingUpdate.java:
##########
@@ -229,6 +236,106 @@ public void
testModificationWithParquetColumnCompression() {
.doesNotContainKey(PARQUET_COLUMN_COMPRESSION_LEVEL_PREFIX + "ID");
}
+ @TestTemplate
+ public void testModificationWithParquetBloomFilterFppAndNdv() {
+ table
+ .updateProperties()
+ .set(PARQUET_BLOOM_FILTER_COLUMN_FPP_PREFIX + "id", "0.05")
+ .set(PARQUET_BLOOM_FILTER_COLUMN_NDV_PREFIX + "id", "1000")
+ .commit();
+
+ table.updateSchema().renameColumn("id", "ID").commit();
+ assertThat(table.properties())
+ .containsEntry(PARQUET_BLOOM_FILTER_COLUMN_FPP_PREFIX + "ID", "0.05")
+ .containsEntry(PARQUET_BLOOM_FILTER_COLUMN_NDV_PREFIX + "ID", "1000")
+ .doesNotContainKey(PARQUET_BLOOM_FILTER_COLUMN_FPP_PREFIX + "id")
+ .doesNotContainKey(PARQUET_BLOOM_FILTER_COLUMN_NDV_PREFIX + "id");
+
+ table.updateSchema().deleteColumn("ID").commit();
+ assertThat(table.properties())
+ .doesNotContainKey(PARQUET_BLOOM_FILTER_COLUMN_FPP_PREFIX + "ID")
+ .doesNotContainKey(PARQUET_BLOOM_FILTER_COLUMN_NDV_PREFIX + "ID");
+ }
+
+ @TestTemplate
+ public void testModificationWithParquetDictEncoding() {
+ table
+ .updateProperties()
+ .set(PARQUET_DICT_ENCODING_ENABLED_COLUMN_PREFIX + "id", "false")
+ .commit();
+
+ table.updateSchema().renameColumn("id", "ID").commit();
+ assertThat(table.properties())
+ .containsEntry(PARQUET_DICT_ENCODING_ENABLED_COLUMN_PREFIX + "ID",
"false")
+ .doesNotContainKey(PARQUET_DICT_ENCODING_ENABLED_COLUMN_PREFIX + "id");
+
+ table.updateSchema().deleteColumn("ID").commit();
+ assertThat(table.properties())
+ .doesNotContainKey(PARQUET_DICT_ENCODING_ENABLED_COLUMN_PREFIX + "ID");
+ }
+
+ @TestTemplate
+ public void testRenameStructWithNestedColumnProperties() {
+ table.updateSchema().addColumn("location", LOCATION_TYPE).commit();
+ table
+ .updateProperties()
+ .set(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + "location.lat",
"full")
+ .set(PARQUET_BLOOM_FILTER_COLUMN_ENABLED_PREFIX + "location.lat",
"true")
+ .commit();
+
+ table.updateSchema().renameColumn("location", "loc").commit();
+ assertThat(table.properties())
+ .containsEntry(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX +
"loc.lat", "full")
+ .containsEntry(PARQUET_BLOOM_FILTER_COLUMN_ENABLED_PREFIX + "loc.lat",
"true")
+ .doesNotContainKey(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX +
"location.lat")
+ .doesNotContainKey(PARQUET_BLOOM_FILTER_COLUMN_ENABLED_PREFIX +
"location.lat");
+
+ // a stale metrics property would fail validation of later property updates
+ table.updateProperties().set("key", "value").commit();
+ assertThat(table.properties()).containsEntry("key", "value");
Review Comment:
I think this is help ensure the `MetricsConfig.validateReferencedColumns`
does not throw in next update after struct rename, right? I guess assertion
reads a bit off here, the earlier assertion of containsEntry of `loc.lat` is
sufficient. Maybe decouple with parquet and keep it minimally focused would
help
``` java
table.updateSchema().addColumn("location", LOCATION_TYPE).commit();
table
.updateProperties()
.set(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX +
"location.lat", "full")
.commit();
table.updateSchema().renameColumn("location", "loc").commit();
assertThat(table.properties())
.containsEntry(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX +
"loc.lat", "full")
.doesNotContainKey(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX +
"location.lat");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]