SinghAsDev commented on a change in pull request #3723:
URL: https://github.com/apache/iceberg/pull/3723#discussion_r768093798
##########
File path:
parquet/src/main/java/org/apache/iceberg/parquet/ApplyNameMapping.java
##########
@@ -88,6 +96,31 @@ public Type primitive(PrimitiveType primitive) {
return field == null ? primitive : primitive.withId(field.id());
}
+ @Override
+ public void beforeElementField(Type element) {
+ super.beforeElementField(makeElement(element));
+ }
+
+ @Override
+ public void afterElementField(Type element) {
+ super.afterElementField(makeElement(element));
+ }
+
+ private Type makeElement(Type element) {
+ // List's element in 3-level lists can be named differently across
different parquet writers.
+ // For example, hive names it "array_element", whereas new parquet writers
names it as "element".
+ if (element.getName().equals("element") || element.isPrimitive()) {
+ return element;
+ }
Review comment:
I think we can make it generic from the perspective that we don't have
magic keywords like "element", however I am not sure if the projection change
you are referring to would work here, but do let me know if you think
otherwise. Here, we are making this change so that constructed parquet schema
had correct ids, but still has the same schema as the one in parquet file. I
will at least make this part generic, will update the diff shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
