RussellSpitzer commented on code in PR #3457:
URL: https://github.com/apache/iceberg/pull/3457#discussion_r938237854
##########
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/ExpireSnapshotsSparkAction.java:
##########
@@ -222,13 +228,34 @@ private ExpireSnapshots.Result doExecute() {
}
}
+ /**
+ * Builds a dataset of reachable files from given table metadata
+ *
+ * @param metadata table metadata
+ * @return a dataset of files: schema(file_path, file_type)
+ */
private Dataset<Row> buildValidFileDF(TableMetadata metadata) {
Table staticTable = newStaticTable(metadata, table.io());
return buildValidContentFileWithTypeDF(staticTable)
.union(withFileType(buildManifestFileDF(staticTable), MANIFEST))
.union(withFileType(buildManifestListDF(staticTable), MANIFEST_LIST));
}
+ /**
+ * Builds a dataset of reachable files from given table metadata, with a
snapshot filter
+ *
+ * @param metadata table metadata
+ * @param snapshotsToExclude files reachable by this snapshot will be
filtered out
+ * @return a dataset of files: schema(file_path, file_type)
+ */
+ private Dataset<Row> buildFilteredValidDataDF(
+ TableMetadata metadata, Set<Long> snapshotsToExclude) {
+ Table staticTable = newStaticTable(metadata, table.io());
+ return buildValidContentFileWithTypeDF(staticTable, snapshotsToExclude)
Review Comment:
I've kinda been thinking we probably don't need the anti-joins distributed
(mostly we just want distributed IO) for this operation but I think all of your
logic makes sense here. if I think this through we have
ALL_MANIFESTS broken into 3 subsets
oldManifests: [Manifests only used in Expired Snapshots ]
commonManifests: [ Manifests used in both expired and unexpired Snapshots ]
newManifests: [ Manifests only used in unexpired-snapshots ]
To build this we read allManifests
expiredManifests = allManifests where snapshot is in snapshots being expired
unexpiredManifests = allManifests where snapshots is not in snapshots being
expired
oldManifests = expiredManifests -- unexpiredManifests
newManifests = unexpiredManifests -- expiredManifests
We read all the mainfests in oldManifests and newManifests
oldManifests -> flatmap read manifest and dedupe -> oldFiles
newManifests -> flatmap read manifest and dedupe -> newFiles
filesToDelete == (oldFiles -- newFiles) + oldManifests + oldManifestLists
Did I get that right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
