JeonDaehong commented on code in PR #18981:
URL: https://github.com/apache/iceberg/pull/18981#discussion_r4599962690
##########
core/src/main/java/org/apache/iceberg/BaseRowDelta.java:
##########
@@ -48,7 +48,7 @@ protected BaseRowDelta self() {
@Override
protected String operation() {
- if (addsDataFiles() && !addsDeleteFiles() && !deletesDataFiles()) {
+ if (addsDataFiles() && !addsDeleteFiles() && !deletesDataFiles() &&
!deletesDeleteFiles()) {
Review Comment:
`deletesDeleteFiles()` is `deleteFilterManager.containsDeletes()`, which is
only true for explicitly removed delete files (paths, files, a delete
expression or dropped partitions), so plain `addRows` commits still come out as
`append`.
##########
core/src/main/java/org/apache/iceberg/BaseOverwriteFiles.java:
##########
@@ -52,7 +52,7 @@ protected String operation() {
return DataOperations.DELETE;
}
- if (addsDataFiles() && !deletesDataFiles()) {
+ if (addsDataFiles() && !deletesDataFiles() && !deletesDeleteFiles()) {
Review Comment:
One effect worth noting: incremental append scans, which the Flink source
and Spark streaming reads use, only consume `append` snapshots. Rows added in a
commit that also removes delete files will now be skipped there, or fail a
Spark streaming read unless overwrites are skipped. That's what the spec asks
for, but it may deserve a line in the release notes.
##########
core/src/test/java/org/apache/iceberg/TestOverwrite.java:
##########
@@ -169,6 +172,32 @@ public void addFilesProducesAppendOperation() {
assertThat(latestSnapshot(table,
branch).operation()).isEqualTo(DataOperations.APPEND);
}
+ @TestTemplate
+ public void addFileAndRemoveDeleteFileProducesOverwriteOperation() {
+ assumeThat(formatVersion).isEqualTo(2);
Review Comment:
This runs only on v2 since v3 rejects position delete files. A v3 variant
with a DV would cover the same `BaseOverwriteFiles` path there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]