mynameborat commented on code in PR #23313:
URL: https://github.com/apache/beam/pull/23313#discussion_r982725450
##########
runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OpAdapter.java:
##########
@@ -84,8 +84,6 @@ public final void schedule(Scheduler<KeyedTimerData<K>>
timerRegistry) {
@Override
public synchronized CompletionStage<Collection<OpMessage<OutT>>>
apply(OpMessage<InT> message) {
Review Comment:
We can keep this for now. But Refer to comment on `collectFuture`. It still
requires handling.
##########
runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java:
##########
@@ -129,4 +134,30 @@ public interface SamzaPipelineOptions extends
PipelineOptions {
long getMaxBundleTimeMs();
void setMaxBundleTimeMs(long maxBundleTimeMs);
+
+ @Description(
+ "The number of threads to run DoFn.processElements in parallel. Used
only in non-portable mode.")
Review Comment:
>
Makes sense. Thanks!
##########
runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OpAdapter.java:
##########
@@ -188,7 +176,31 @@ public void emitWatermark(Instant watermark) {
@Override
public <T> void emitView(String id, WindowedValue<Iterable<T>> elements) {
- outputList.add(OpMessage.ofSideInput(id, elements));
+ outputQueue.add(OpMessage.ofSideInput(id, elements));
+ }
+
+ @Override
+ public Collection<OpMessage<OutT>> collectOutput() {
+ final List<OpMessage<OutT>> outputList = new ArrayList<>();
+ OpMessage<OutT> output;
+ while ((output = outputQueue.poll()) != null) {
+ outputList.add(output);
+ }
+ return outputList;
+ }
+
+ @Override
+ public CompletionStage<Collection<OpMessage<OutT>>> collectFuture() {
+ final CompletionStage<Collection<OpMessage<OutT>>> future = outputFuture;
+ outputFuture = null;
Review Comment:
This is still invoked from the `AsyncDoFnRunner` which invokes this as part
of the `thenApply` to `runAsync` which would run on common java thread pool.
Looks like we still need this shielded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
