flyrain commented on code in PR #4539:
URL: https://github.com/apache/iceberg/pull/4539#discussion_r855653285
##########
data/src/main/java/org/apache/iceberg/data/DeleteFilter.java:
##########
@@ -185,14 +203,35 @@ private CloseableIterable<T>
applyEqDeletes(CloseableIterable<T> records) {
.reduce(Predicate::and)
.orElse(t -> true);
- Filter<T> remainingRowsFilter = new Filter<T>() {
+ Filter<T> remainingRowsFilter = hasMetadataColumnIsDeleted ?
getMarker(remainingRows) : getFilter(remainingRows);
+
+ return remainingRowsFilter.filter(records);
+ }
+
+ private Filter<T> getFilter(Predicate<T> remainingRows) {
+ return new Filter<T>() {
@Override
protected boolean shouldKeep(T item) {
return remainingRows.test(item);
}
};
+ }
- return remainingRowsFilter.filter(records);
+ private Filter<T> getMarker(Predicate<T> remainingRows) {
+ return new Filter<T>() {
+ @Override
+ protected boolean shouldKeep(T item) {
+ if (!remainingRows.test(item)) {
+ // if the row is deleted, set the metadata column to true
+ markRowDeleted(item);
+ }
+ return true; // keep the row even if it is deleted by equality
deletions
+ }
+ };
+ }
+
+ protected void markRowDeleted(T item) {
Review Comment:
Probably, it needs changes for all subclass of DeleteFilter. This is PR is
already big. Maybe we should split it. For example, we can have a separated PR
to read deleted rows in case of row-level change. cc @chenjunjiedada
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
