scwhittle commented on code in PR #22645:
URL: https://github.com/apache/beam/pull/22645#discussion_r945609458
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -968,6 +970,10 @@ public void processElement(ProcessContext c, BoundedWindow
window) throws Except
new FileResult<>(writer.getOutputFile(), shard, window,
c.pane(), entry.getKey())));
closeWriterInBackground(writer);
}
+
+ // Ensure that the past closes happen before returning and after we
started the closes
Review Comment:
nit: I find the happen before/after confusing since they may happen whenever
we are just blocking on them completing here
Maybe
Block on completing the past closes before returning. We do so after
starting the current closes in the background
so that they can happen in parallel.
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -996,10 +1001,18 @@ public void finishBundle(FinishBundleContext c) throws
Exception {
c.output(result.getValue(), result.getKey(),
result.getValue().getWindow());
}
} finally {
- deferredOutput = null;
- closeFutures = null;
+ deferredOutput.clear();
+ closeFutures.clear();
}
}
+
+ // Ensure that transient fields are initialized.
Review Comment:
Any reason to prefer this to the @Setup method?
If not, I'd say that is preferrable since it is a standard Beam DoFn method
and the name is more self-explanatory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
