mbutrovich commented on code in PR #15470:
URL: https://github.com/apache/iceberg/pull/15470#discussion_r3453865237
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java:
##########
@@ -310,21 +317,29 @@ private Result rebuildMetadata() {
// rebuild manifest files
Set<ManifestFile> metaFiles = rewriteManifestListResult.toRewrite();
- RewriteContentFileResult rewriteManifestResult =
- rewriteManifests(deltaSnapshots, endMetadata, metaFiles);
- // rebuild position delete files
- Set<DeleteFile> deleteFiles =
- rewriteManifestResult.toRewrite().stream()
- .filter(e -> e instanceof DeleteFile)
- .map(e -> (DeleteFile) e)
- .collect(Collectors.toCollection(DeleteFileSet::create));
- rewritePositionDeletes(deleteFiles);
+ // Enumerate the distinct position delete files referenced by the delete
manifests being
+ // rewritten (metadata-only pass).
+ Set<DeleteFile> deleteFilesToRewrite = positionDeletesToRewrite(metaFiles);
+
+ // Rewrite those delete files in parallel (deduped by path) and collect
the actual size of each
+ // rewritten file. The size is measured from the writer on the executor
that produced the file,
+ // avoiding both the end-of-job burst of getLength()/HEAD calls and the
file system races where
+ // an in-progress write underreports its length.
+ Map<String, Long> rewrittenDeleteFileSizes =
rewritePositionDeletes(deleteFilesToRewrite);
+
+ // Rewrite manifests (metadata-only), stamping file_size_in_bytes from the
measured sizes.
Review Comment:
Done, removed/trimmed.
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java:
##########
@@ -682,27 +704,120 @@ private static RewriteResult<DeleteFile>
writeDeleteManifest(
specsById,
sourcePrefix,
targetPrefix,
- stagingLocation);
+ stagingLocation,
+ rewrittenDeleteFileSizes.value());
} catch (IOException e) {
throw new RuntimeIOException(e);
}
}
- private void rewritePositionDeletes(Set<DeleteFile> toRewrite) {
+ /**
+ * Enumerate the distinct position delete files referenced by the delete
manifests being
+ * rewritten. Equality delete files are excluded because they hold no
absolute paths and are not
+ * rewritten.
Review Comment:
Done, removed/trimmed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
