claudevdm commented on code in PR #39896:
URL: https://github.com/apache/beam/pull/39896#discussion_r3874055239
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFiles.java:
##########
@@ -398,18 +366,16 @@ private void outputResult(ProcessResult result,
MultiOutputReceiver output) {
@FinishBundle
public void finishBundle(FinishBundleContext context) throws Exception {
- // Block and wait for threads to finish their work
- int numErrors = 0;
- for (Future<ProcessResult> future : checkStateNotNull(activeTasks)) {
- ProcessResult result = future.get();
- if (result.errorRow != null) {
- context.output(ERRORS, result.errorRow, result.timestamp,
result.window);
- numErrors++;
- } else if (result.dataFile != null) {
- context.output(DATA_FILES, result.dataFile, result.timestamp,
result.window);
- }
+ checkStateNotNull(tasks).awaitAll(result -> outputAtFinish(result,
context));
+ }
+
+ private static void outputAtFinish(ProcessResult result,
FinishBundleContext context) {
+ if (result.errorRow != null) {
+ context.output(ERRORS, result.errorRow, result.timestamp,
result.window);
Review Comment:
This is invoked once per file not per budnle. Each ProcessResult represents
one datafile/ error row. Success is emitted to DATA_FILES, failure to ERRORS
https://github.com/claudevdm/beam/blob/f7edad1f06bc04dcd11b1eaee6dad3618b861048/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFiles.java#L346-L365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
