rdblue commented on a change in pull request #1469:
URL: https://github.com/apache/iceberg/pull/1469#discussion_r499949592
##########
File path: core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java
##########
@@ -202,6 +207,58 @@ private ManifestFile copyManifest(ManifestFile manifest) {
current.formatVersion(), toCopy, current.specsById(), newManifestPath,
snapshotId(), appendedManifestsSummary);
}
+ /**
+ * Validates that no files matching a filter have been added to the table
since a starting snapshot.
+ *
+ * @param base table metadata to validate
+ * @param startingSnapshotId id of the snapshot current at the start of the
operation
+ * @param conflictDetectionFilter an expression used to find new conflicting
data files
+ * @param caseSensitive whether expression evaluation should be case
sensitive
+ */
+ protected void validateAddedDataFiles(TableMetadata base, Long
startingSnapshotId,
+ Expression conflictDetectionFilter,
boolean caseSensitive) {
+ List<ManifestFile> manifests = Lists.newArrayList();
+ Set<Long> newSnapshots = Sets.newHashSet();
+
+ Long currentSnapshotId = base.currentSnapshot().snapshotId();
+ while (currentSnapshotId != null &&
!currentSnapshotId.equals(startingSnapshotId)) {
+ Snapshot currentSnapshot = ops.current().snapshot(currentSnapshotId);
+
+ ValidationException.check(currentSnapshot != null,
+ "Cannot determine history between starting snapshot %s and current
%s",
+ startingSnapshotId, currentSnapshotId);
+
+ newSnapshots.add(currentSnapshotId);
+ for (ManifestFile manifest : currentSnapshot.dataManifests()) {
+ if (manifest.snapshotId() == (long) currentSnapshotId) {
+ manifests.add(manifest);
+ }
+ }
+
+ currentSnapshotId = currentSnapshot.parentId();
+ }
+
+ ManifestGroup conflictGroup = new ManifestGroup(ops.io(), manifests,
ImmutableList.of())
Review comment:
The biggest concern was correctness.
This was matching manifests without checking the manifest's partition spec.
So the same evaluator was used for all file partitions regardless of whether it
was correct. Using `ManifestGroup` avoids that problem. Note that it isn't a
problem for the delete validation because we have a concrete set of files that
must exist.
This will also have some benefit by not caching. That's a good motivation to
change the implementation of the delete checks as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
