aokolnychyi commented on a change in pull request #351: Extend Iceberg with a
way to overwrite files for eager updates/deletes
URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r315093153
##########
File path: api/src/main/java/org/apache/iceberg/OverwriteFiles.java
##########
@@ -74,5 +85,23 @@
*
* @return this for method chaining
*/
- OverwriteFiles validateAddedFiles();
+ OverwriteFiles validateAddedFilesMatchOverwriteFilter();
+
+ /**
+ * Enables validation that files added concurrently do not conflict with
this commit's operation.
+ * <p>
+ * This method should be called when the table is queried to determine which
files to delete/append.
+ * If a concurrent operation commits a new file after the data was read and
that file might
+ * contain rows matching the specified conflict detection filter, the
overwrite operation
+ * will detect this during retries and fail.
+ * <p>
+ * Calling this method with a correct conflict detection filter is required
to maintain
Review comment:
We can also consider making the isolation level a table property and
validating it here. I can think of multiple operations where this will be
applicable: compaction, update, delete, merge into. However, I am not sure we
want the same isolation level for all operations. Compaction is not
serializable in its current form: we don't check files added concurrently
(which seems correct to me). Maybe, the isolation level shouldn't be a table
property but rather a config of the operation itself. That way, compaction will
be using snapshot isolation and update/deletes might be serializable.
@rdblue what are your thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
