grantatspothero opened a new pull request, #16939: URL: https://github.com/apache/iceberg/pull/16939
Per iceberg spec: https://iceberg.apache.org/spec/#scan-planning >A position delete file must be applied to a data file when all of the following are true: The data file's file_path is equal to the delete file's referenced_data_file if it is non-null The data file's data sequence number is less than or equal to the delete file's data sequence number The data file's partition (both spec and partition values) is equal [4] to the delete file's partition There is no deletion vector that must be applied to the data file (when added, such a vector must contain all deletes from existing position delete files) But see the test below, a delete file can be written with an invalid partition value that does not match the referenced data file partition value. Per the spec, since all the conditions are not met I was expecting scan planning to not match the invalid delete file to the data file but it appears scan planning does match the delete file to the data file. Is this a bug in the scan planning implementation? A bug in the spec? It was definitely surprising behavior. Why this matters: If one wishes to do partition pruning via `DeleteFileIndex#filterPartitions`, it is not possible to filter out partitions purely based on delete file partition because the delete file partition value is ignored when referenced_data_file is not null. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
