dramaticlly commented on code in PR #18885:
URL: https://github.com/apache/iceberg/pull/18885#discussion_r4555013115
##########
core/src/main/java/org/apache/iceberg/SchemaUpdate.java:
##########
@@ -506,22 +506,33 @@ private TableMetadata
applyChangesToMetadata(TableMetadata metadata) {
// Transform the metrics if they exist
if (base != null && base.properties() != null) {
- Schema newSchema = newMetadata.schema();
- List<String> deletedColumns =
-
deletes.stream().map(schema::findColumnName).collect(Collectors.toList());
- Map<String, String> renamedColumns =
- updates.keySet().stream()
- .filter(id -> !addedNameToId.containsValue(id)) // remove added
columns
- .filter(id ->
!schema.findColumnName(id).equals(newSchema.findColumnName(id)))
- .collect(Collectors.toMap(schema::findColumnName,
newSchema::findColumnName));
+ Map<Integer, String> newIdToName = newMetadata.schema().idToName();
+ List<String> deletedColumns = Lists.newArrayList();
+ Map<String, String> renamedColumns = Maps.newHashMap();
+ // compare all fields so that nested fields of a deleted or renamed
struct are included
+ schema
+ .idToName()
+ .forEach(
+ (id, name) -> {
+ String newName = newIdToName.get(id);
+ if (newName == null) {
+ deletedColumns.add(name);
Review Comment:
I think there's some subtle behavior change here, previous the delete is
deterministic by field id but now it now diff between old and new schema and
categorize anything missing in new as deleted, so it will not only handle the
nested field but also list element and key/value path.
I suspect this is departure from original intended behavior, as field look
up is only meaningful for leaf element.
##########
core/src/main/java/org/apache/iceberg/SchemaUpdate.java:
##########
@@ -506,22 +506,33 @@ private TableMetadata
applyChangesToMetadata(TableMetadata metadata) {
// Transform the metrics if they exist
if (base != null && base.properties() != null) {
- Schema newSchema = newMetadata.schema();
- List<String> deletedColumns =
-
deletes.stream().map(schema::findColumnName).collect(Collectors.toList());
- Map<String, String> renamedColumns =
- updates.keySet().stream()
- .filter(id -> !addedNameToId.containsValue(id)) // remove added
columns
- .filter(id ->
!schema.findColumnName(id).equals(newSchema.findColumnName(id)))
- .collect(Collectors.toMap(schema::findColumnName,
newSchema::findColumnName));
+ Map<Integer, String> newIdToName = newMetadata.schema().idToName();
+ List<String> deletedColumns = Lists.newArrayList();
+ Map<String, String> renamedColumns = Maps.newHashMap();
+ // compare all fields so that nested fields of a deleted or renamed
struct are included
+ schema
+ .idToName()
Review Comment:
also this adds additional cost to the schema update and might add-column
only case slowed, as it now required to index the new schema regardless
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
