JozoVilcek commented on code in PR #24837:
URL: https://github.com/apache/beam/pull/24837#discussion_r1090332184
##########
runners/spark/src/main/java/org/apache/beam/runners/spark/SparkCommonPipelineOptions.java:
##########
@@ -84,4 +84,11 @@ static void prepareFilesToStage(SparkCommonPipelineOptions
options) {
PipelineResources.prepareFilesForStaging(options);
}
}
+
+ @Description(
+ "Enable/disable async output for operators with possibly large output (
such as splittable DoFn )")
+ @Default.Boolean(true)
+ Boolean getEnableAsyncOperatorOutput();
Review Comment:
@mosche can you share an example how do I do that? Alternatively it would
make sense to me jsut keep one validates runner with experiment turned on. That
means test all iterator on majority of operators but the splittable dofn where
we test the experiment. Old iterator does not distinguish what is the operator
it is connecting to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
