mosche commented on code in PR #22620:
URL: https://github.com/apache/beam/pull/22620#discussion_r969534303
##########
runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java:
##########
@@ -153,4 +192,84 @@ private static void awaitWatermarksOrTimeout(
} while ((timeoutMillis -= batchDurationMillis) > 0
&& globalWatermark.isBefore(stopPipelineWatermark));
}
+
+ /**
+ * Override factory to replace {@link Read.Unbounded} with {@link
UnboundedReadFromBoundedSource}
+ * to force streaming mode.
+ */
+ private static class UnboundedReadFromBoundedSourceOverrideFactory<T>
Review Comment:
Happy to do that though I'm not entirely sure if the factory is of much
value by itself. I also had to fix the outputs in a non trivial way using a
visitor after the replacement, see
https://github.com/apache/beam/pull/22620/files#diff-d81f49eb0330230bd03ce6cd33b5f70f59c443aac57741e877ececbada32b16bR246-R274
I couldn't find a way to achieve this in `mapOutputs` of the override
factory itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
