rdblue commented on a change in pull request #207: Add external schema mappings
for files written with name-based schemas #40
URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r330275738
##########
File path: core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java
##########
@@ -165,13 +165,14 @@ public Schema array(Schema array, Supplier<Schema>
element) {
try {
Schema keyValueSchema = array.getElementType();
Schema.Field keyField = keyValueSchema.getFields().get(0);
+ Schema.Field keyProjection = element.get().getField("key");
Schema.Field valueField = keyValueSchema.getFields().get(1);
Schema.Field valueProjection = element.get().getField("value");
// element was changed, create a new array
- if (valueProjection.schema() != valueField.schema()) {
+ if (keyProjection.schema() != keyField.schema() ||
valueProjection.schema() != valueField.schema()) {
return
AvroSchemaUtil.createProjectionMap(keyValueSchema.getFullName(),
- AvroSchemaUtil.getFieldId(keyField), keyField.name(),
keyField.schema(),
+ AvroSchemaUtil.getFieldId(keyField), keyField.name(),
keyProjection.schema(),
Review comment:
The previous version didn't use `keyProjection` because the entire key must
be projected. If the key is a struct of multiple columns, then projecting a
subset of those columns can easily introduce key collisions that aren't in the
original data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
