rdblue commented on code in PR #5665:
URL: https://github.com/apache/iceberg/pull/5665#discussion_r960107948
##########
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/SparkDataFile.java:
##########
@@ -84,10 +93,31 @@ public SparkDataFile(Types.StructType type, StructType
sparkType) {
sortOrderIdPosition = positions.get("sort_order_id");
}
+ private void wrapPartitionSpec(GenericRowWithSchema specRow) {
+ // We get all the partition fields, but want to project to the current one
+ StructType wrappedPartitionStruct = specRow.schema();
+
+ if (!wrappedPartitionStruct.equals(currentWrappedPartitionStruct)) {
+ this.currentWrappedPartitionStruct = wrappedPartitionStruct;
+
+ // The original IDs are lost in translation, therefore we apply the ones
that we know
Review Comment:
I think the problem may be that this is trying to fix `SparkDataFile`. There
isn't enough information here to project, but that's okay.
I think we need to update the place where `SparkDataFile.partition()` is
used, in `RewriteManifestsSparkAction.writeManifest`. That uses a
`SparkDataFile`, but it has the spec. We just need to pass in the combined
partition struct type for `SparkDataFile` and then add the projection before
writing into the new manifest file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
