openinx commented on a change in pull request #1939:
URL: https://github.com/apache/iceberg/pull/1939#discussion_r545701370
##########
File path:
flink/src/main/java/org/apache/iceberg/flink/sink/FlinkManifestUtil.java
##########
@@ -63,4 +66,53 @@ static ManifestOutputFileFactory
createOutputFileFactory(Table table, String fli
TableOperations ops = ((HasTableOperations) table).operations();
return new ManifestOutputFileFactory(ops, table.io(), table.properties(),
flinkJobId, subTaskId, attemptNumber);
}
+
+ static DeltaManifests writeCompletedFiles(WriteResult result,
+ Supplier<OutputFile>
outputFileSupplier,
+ PartitionSpec spec) throws
IOException {
+
+ ManifestFile dataManifest = null;
+ ManifestFile deleteManifest = null;
+
+ // Write the completed data files into a newly created data manifest file.
+ if (result.dataFiles() != null && result.dataFiles().length > 0) {
+ dataManifest = writeDataFiles(outputFileSupplier.get(), spec,
Lists.newArrayList(result.dataFiles()));
+ }
+
+ // Write the completed delete files into a newly created delete manifest
file.
+ if (result.deleteFiles() != null && result.deleteFiles().length > 0) {
+ OutputFile deleteManifestFile = outputFileSupplier.get();
+
+ ManifestWriter<DeleteFile> deleteManifestWriter =
ManifestFiles.writeDeleteManifest(FORMAT_V2, spec,
+ deleteManifestFile, DUMMY_SNAPSHOT_ID);
+ try (ManifestWriter<DeleteFile> writer = deleteManifestWriter) {
+ for (DeleteFile deleteFile : result.deleteFiles()) {
+ writer.add(deleteFile);
+ }
+ }
+
+ deleteManifest = deleteManifestWriter.toManifestFile();
+ }
+
+ return new DeltaManifests(dataManifest, deleteManifest);
Review comment:
About this question, I think we'd better to keep the dummy
`DeltaManifests` in state , although it has no delete files and data files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
