[
https://issues.apache.org/jira/browse/BEAM-10559?focusedWorklogId=462663&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-462663
]
ASF GitHub Bot logged work on BEAM-10559:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Jul/20 19:00
Start Date: 23/Jul/20 19:00
Worklog Time Spent: 10m
Work Description: chamikaramj commented on a change in pull request
#12355:
URL: https://github.com/apache/beam/pull/12355#discussion_r459662985
##########
File path: sdks/python/apache_beam/examples/wordcount_xlang_sql.py
##########
@@ -15,12 +15,16 @@
# limitations under the License.
#
-"""A cross-language word-counting workflow."""
+"""A cross-language word-counting workflow.
+
+Java and docker must be available to run this pipeline.
Review comment:
Sounds good.
##########
File path: sdks/python/apache_beam/examples/wordcount_xlang_sql.py
##########
@@ -101,12 +89,33 @@ def main():
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options.view_as(SetupOptions).save_main_session = True
- p = beam.Pipeline(options=pipeline_options)
- # Preemptively start due to BEAM-6666.
- p.runner.create_job_service(pipeline_options)
+ with beam.Pipeline(options=pipeline_options) as p:
+ if isinstance(p.runner, portable_runner.PortableRunner):
+ # Preemptively start due to BEAM-6666.
+ p.runner.create_job_service(pipeline_options)
Review comment:
Yeah, you are right.
##########
File path: sdks/python/apache_beam/examples/wordcount_xlang_sql.py
##########
@@ -101,12 +89,33 @@ def main():
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options.view_as(SetupOptions).save_main_session = True
- p = beam.Pipeline(options=pipeline_options)
- # Preemptively start due to BEAM-6666.
- p.runner.create_job_service(pipeline_options)
+ with beam.Pipeline(options=pipeline_options) as p:
+ if isinstance(p.runner, portable_runner.PortableRunner):
+ # Preemptively start due to BEAM-6666.
+ p.runner.create_job_service(pipeline_options)
+
+ run(p, known_args.input, known_args.output)
- run(p, known_args.input, known_args.output)
+# Some more fun queries:
+# ------
+# SELECT
+# word as key,
+# COUNT(*) as `count`
+# FROM PCOLLECTION
+# GROUP BY word
Review comment:
Let's try them again and try with Dataflow as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 462663)
Remaining Estimate: 0h
Time Spent: 10m
> Python SqlTransform examples
> ----------------------------
>
> Key: BEAM-10559
> URL: https://issues.apache.org/jira/browse/BEAM-10559
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql, sdk-py-core
> Reporter: Brian Hulette
> Assignee: Brian Hulette
> Priority: P2
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Add an example like
> https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/kafkataxi
> for SqlTransform use-cases. Perhaps one word-count, one streaming taxi data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
