rdblue commented on a change in pull request #1469:
URL: https://github.com/apache/iceberg/pull/1469#discussion_r499949708
##########
File path: core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java
##########
@@ -202,6 +207,58 @@ private ManifestFile copyManifest(ManifestFile manifest) {
current.formatVersion(), toCopy, current.specsById(), newManifestPath,
snapshotId(), appendedManifestsSummary);
}
+ /**
+ * Validates that no files matching a filter have been added to the table
since a starting snapshot.
+ *
+ * @param base table metadata to validate
+ * @param startingSnapshotId id of the snapshot current at the start of the
operation
+ * @param conflictDetectionFilter an expression used to find new conflicting
data files
+ * @param caseSensitive whether expression evaluation should be case
sensitive
+ */
+ protected void validateAddedDataFiles(TableMetadata base, Long
startingSnapshotId,
+ Expression conflictDetectionFilter,
boolean caseSensitive) {
+ List<ManifestFile> manifests = Lists.newArrayList();
+ Set<Long> newSnapshots = Sets.newHashSet();
+
+ Long currentSnapshotId = base.currentSnapshot().snapshotId();
+ while (currentSnapshotId != null &&
!currentSnapshotId.equals(startingSnapshotId)) {
+ Snapshot currentSnapshot = ops.current().snapshot(currentSnapshotId);
+
+ ValidationException.check(currentSnapshot != null,
+ "Cannot determine history between starting snapshot %s and current
%s",
+ startingSnapshotId, currentSnapshotId);
+
+ newSnapshots.add(currentSnapshotId);
+ for (ManifestFile manifest : currentSnapshot.dataManifests()) {
+ if (manifest.snapshotId() == (long) currentSnapshotId) {
+ manifests.add(manifest);
+ }
+ }
+
+ currentSnapshotId = currentSnapshot.parentId();
+ }
+
+ ManifestGroup conflictGroup = new ManifestGroup(ops.io(), manifests,
ImmutableList.of())
+ .caseSensitive(caseSensitive)
+ .filterManifestEntries(entry ->
newSnapshots.contains(entry.snapshotId()))
+ .filterData(conflictDetectionFilter)
+ .specsById(base.specsById())
+ .ignoreDeleted()
+ .ignoreExisting();
Review comment:
Yes, the number of changes should be small, especially if we ignore
rewrites.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
