wypoon commented on code in PR #4588:
URL: https://github.com/apache/iceberg/pull/4588#discussion_r952039537
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/BaseReader.java:
##########
@@ -261,7 +268,10 @@ protected InputFile getInputFile(String location) {
@Override
protected void markRowDeleted(InternalRow row) {
- row.setBoolean(columnIsDeletedPosition(), true);
+ if (!row.getBoolean(columnIsDeletedPosition())) {
+ row.setBoolean(columnIsDeletedPosition(), true);
+ counter().increment();
+ }
Review Comment:
`markRowDeleted` can be called as the `Consumer` in
`Deletes.markDeleted(CloseableIterable<T>, Predicate<T>, Consumer<T>)` via
`DeleteFilter#applyPosDeletes(CloseableIterable<T>)` and via
`DeleteFilter#applyEqDeletes(CloseableIterable<T>)`.
I assume we want to avoid counting twice when the same row is deleted via a
positional delete and then via an equality delete.
Thus we check to see if the row is already marked deleted, and if it is not,
then we mark it and increment the count.
This count is tested in
`TestSparkReaderDeletes#testMixedPosAndEqDeletesWithDeletedColumn`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
