szehon-ho commented on code in PR #4812:
URL: https://github.com/apache/iceberg/pull/4812#discussion_r905560701
##########
core/src/main/java/org/apache/iceberg/BaseMetadataTable.java:
##########
@@ -60,8 +61,12 @@ protected BaseMetadataTable(TableOperations ops, Table
table, String name) {
* @return a spec used to rewrite the metadata table filters to partition
filters using an inclusive projection
*/
static PartitionSpec transformSpec(Schema metadataTableSchema, PartitionSpec
spec) {
+ return transformSpec(metadataTableSchema, spec.partitionType());
+ }
+
+ static PartitionSpec transformSpec(Schema metadataTableSchema,
Types.StructType partitionType) {
Review Comment:
There is a tricky bug here in partition filter push down that took me awhile
to find.
In most metadata tables, the transformed partition spec used only in
ManifestGroup to filter out matching manifests. In that case, it seems the
generated fieldId is not important. However, the position delete metadata
table needs to filter out matching manifest-entries (as it actually reads
delete files). A wrong fieldId causes the DeleteFile to be instantiated with
wrong partition information (as the field id is used to project onto the
partition struct to lookup the value), leading to various bugs in filtering out
the wrong files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
