Dear community,
I am trying to re-write a couple of data files in a table, like
val fileToDelete1 = DataFiles.builder(partitionSpec)
...
.withPath(delete_path_1)
...
.build
val fileToDelete2 = DataFiles.builder(partitionSpec)
...
.withPath(delete_path_2)
...
.build
val fileToAdd = DataFiles.builder(partitionSpec)
...
.withPath(add1)
...
.build
table.newRewrite()
.rewriteFiles(JavaConversions.setAsJavaSet(Set(fileToDelete1,
fileToDelete2)),
JavaConversions.setAsJavaSet(Set(fileToAdd)))
.commit()
And it is rejected by
Exception in thread "main"
org.apache.iceberg.exceptions.ValidationException: Missing required files
to delete: delete_path_1, delete_path_2
at
org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:42)
at
org.apache.iceberg.MergingSnapshotProducer.apply(MergingSnapshotProducer.java:275)
at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:146)
at
org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:238)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:403)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:212)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:188)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:237)
at Test$.main(Test.scala:68)
at Test.main(Test.scala)
The logic in MergingSnapshotProducer (line 275
<https://github.com/apache/incubator-iceberg/blob/433f169e9d0b10688d395abde64c4b6461d35ca9/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L275>)
makes that happen. failMissingDeletePaths is true. And
deletedFiles.containsAll(deletePaths) is false, as deleteFiles is empty.
My questions are
(1) What does failMissingDeletePaths mean? Whether to fail if missing
delete paths happens?
(2) How to make deletedFiles not being empty (it is supposed to contain the
files to delete I believe)?
I am reading the code but do not figure it out yet. Really appreciate it if
you could share your thoughts at your most convenience. Thanks!
--
李响 Xiang Li
手机 cellphone ：+86-136-8113-8972
邮件 e-mail ：[email protected]
