[
https://issues.apache.org/jira/browse/BEAM-5707?focusedWorklogId=155969&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-155969
]
ASF GitHub Bot logged work on BEAM-5707:
----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Oct/18 16:04
Start Date: 18/Oct/18 16:04
Worklog Time Spent: 10m
Work Description: tweise closed pull request #6737: [BEAM-5707] Add
support for options to flink_streaming_impulse.py
URL: https://github.com/apache/beam/pull/6737
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/examples/flink/flink_streaming_impulse.py
b/sdks/python/apache_beam/examples/flink/flink_streaming_impulse.py
index 23ad8f25d4e..0cfaf5d1422 100644
--- a/sdks/python/apache_beam/examples/flink/flink_streaming_impulse.py
+++ b/sdks/python/apache_beam/examples/flink/flink_streaming_impulse.py
@@ -24,6 +24,7 @@
import argparse
import logging
+import sys
import apache_beam as beam
import apache_beam.transforms.window as window
@@ -57,15 +58,26 @@ def run(argv=None):
args.extend(argv)
parser = argparse.ArgumentParser()
- _, pipeline_args = parser.parse_known_args(args)
+ parser.add_argument('--count',
+ dest='count',
+ default=0,
+ help='Number of triggers to generate '
+ '(0 means emit forever).')
+ parser.add_argument('--interval_ms',
+ dest='interval_ms',
+ default=500,
+ help='Interval between records per parallel '
+ 'Flink subtask.')
+
+ known_args, pipeline_args = parser.parse_known_args(args)
pipeline_options = PipelineOptions(pipeline_args)
p = beam.Pipeline(options=pipeline_options)
messages = (p | FlinkStreamingImpulseSource()
- .set_message_count(10000)
- .set_interval_ms(500))
+ .set_message_count(known_args.count)
+ .set_interval_ms(known_args.interval_ms))
_ = (messages | 'decode' >> beam.Map(lambda x: ('', 1))
| 'window' >> beam.WindowInto(window.GlobalWindows(),
@@ -83,4 +95,4 @@ def run(argv=None):
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
- run()
+ run(sys.argv[1:])
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 155969)
Time Spent: 6h 20m (was: 6h 10m)
> Add a portable Flink streaming synthetic source for testing
> -----------------------------------------------------------
>
> Key: BEAM-5707
> URL: https://issues.apache.org/jira/browse/BEAM-5707
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Reporter: Micah Wylde
> Assignee: Micah Wylde
> Priority: Minor
> Fix For: 2.9.0
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> Currently there are no built-in streaming sources for portable pipelines.
> This makes it hard to test streaming functionality in the Python SDK.
> It would be very useful to add a periodic impulse source that (with some
> configurable frequency) outputs an empty byte array, which can then be
> transformed as desired inside the python pipeline. More context in this
> [mailing list
> discussion|https://lists.apache.org/thread.html/b44a648ab1d0cb200d8bfe4b280e9dad6368209c4725609cbfbbe410@%3Cdev.beam.apache.org%3E].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
