Fokko commented on code in PR #5665:
URL: https://github.com/apache/iceberg/pull/5665#discussion_r960384482
##########
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/SparkDataFile.java:
##########
@@ -84,10 +93,31 @@ public SparkDataFile(Types.StructType type, StructType
sparkType) {
sortOrderIdPosition = positions.get("sort_order_id");
}
+ private void wrapPartitionSpec(GenericRowWithSchema specRow) {
+ // We get all the partition fields, but want to project to the current one
+ StructType wrappedPartitionStruct = specRow.schema();
+
+ if (!wrappedPartitionStruct.equals(currentWrappedPartitionStruct)) {
+ this.currentWrappedPartitionStruct = wrappedPartitionStruct;
+
+ // The original IDs are lost in translation, therefore we apply the ones
that we know
Review Comment:
Thanks @aokolnychyi for chiming in here, much appreciated. I'm just digging
into this code for the first time, so bear with me. Wouldn't it make more sense
to either pass in all the historical specs (in the case of the metadata
column), or just the latest spec when that's needed? It doesn't feel like a
scalable approach to pass in a tuple of all the specs, and then only project
the ones that we need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
