szehon-ho commented on a change in pull request #4370:
URL: https://github.com/apache/iceberg/pull/4370#discussion_r831631088
##########
File path: core/src/main/java/org/apache/iceberg/ManifestFilterManager.java
##########
@@ -360,23 +360,23 @@ private boolean
manifestHasDeletedFiles(PartitionAndMetricsEvaluator evaluator,
dropPartitions.contains(file.specId(), file.partition()) ||
(isDelete && entry.sequenceNumber() > 0 && entry.sequenceNumber() <
minSequenceNumber);
- boolean nonMatchingDeleteFile = !file.content().equals(FileContent.DATA)
&& !evaluator.rowsMustMatch(file);
- if (!markedForDelete && nonMatchingDeleteFile) {
- // not all DeleteFiles removal can be handled by metadata operation,
skip in this case
- continue;
- }
-
if (markedForDelete || evaluator.rowsMightMatch(file)) {
+ boolean allRowsMatch = markedForDelete ||
evaluator.rowsMustMatch(file);
ValidationException.check(
- markedForDelete || evaluator.rowsMustMatch(file),
+ allRowsMatch || isDelete, // ignore delete files where some
records may not match the expression
"Cannot delete file where some, but not all, rows match filter %s:
%s",
this.deleteExpression, file.path());
- hasDeletedFiles = true;
+ hasDeletedFiles = allRowsMatch;
+
if (failAnyDelete) {
throw new
DeleteException(reader.spec().partitionToPath(file.partition()));
Review comment:
In failAnyDelete case, this would trigger if we have a deleteFile but
rows do not match. Probably this never happens in delete call, but wonder if
we need to update the condition (failAnyDelete && allRowsMatch).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
