[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410632=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410632
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 26/Mar/20 21:16
Start Date: 26/Mar/20 21:16
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #10055:
[BEAM-8603] Add Python SqlTransform
URL: https://github.com/apache/beam/pull/10055#discussion_r398897788
##
File path: sdks/python/apache_beam/transforms/sql_test.py
##
@@ -0,0 +1,109 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for transforms that use the SQL Expansion service."""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import logging
+import typing
+import unittest
+
+from nose.plugins.attrib import attr
+from past.builtins import unicode
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.options.pipeline_options import StandardOptions
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.transforms.sql import SqlTransform
+from apache_beam.utils import subprocess_server
+
+SimpleRow = typing.NamedTuple(
+"SimpleRow", [("int", int), ("str", unicode), ("flt", float)])
+coders.registry.register_coder(SimpleRow, coders.RowCoder)
+
+
+@attr('UsesSqlExpansionService')
+@unittest.skipIf(
+TestPipeline().get_pipeline_options().view_as(StandardOptions).runner is
+None,
+"Must be run with a runner that supports cross-language transforms")
Review comment:
Oh ok I didn't realize that.
Really I just needed a way to prevent this test from running in the Python
PreCommit, since the SQL expansion service isn't built in that context. The
other xlang test suite handles that by checking if the `EXPANSION_PORT` env var
is set.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410632)
Time Spent: 4h 40m (was: 4.5h)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410541=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410541
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 26/Mar/20 19:42
Start Date: 26/Mar/20 19:42
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10055: [BEAM-8603]
Add Python SqlTransform
URL: https://github.com/apache/beam/pull/10055#discussion_r398831281
##
File path: sdks/python/apache_beam/transforms/sql_test.py
##
@@ -0,0 +1,109 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for transforms that use the SQL Expansion service."""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import logging
+import typing
+import unittest
+
+from nose.plugins.attrib import attr
+from past.builtins import unicode
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.options.pipeline_options import StandardOptions
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.transforms.sql import SqlTransform
+from apache_beam.utils import subprocess_server
+
+SimpleRow = typing.NamedTuple(
+"SimpleRow", [("int", int), ("str", unicode), ("flt", float)])
+coders.registry.register_coder(SimpleRow, coders.RowCoder)
+
+
+@attr('UsesSqlExpansionService')
+@unittest.skipIf(
+TestPipeline().get_pipeline_options().view_as(StandardOptions).runner is
+None,
+"Must be run with a runner that supports cross-language transforms")
Review comment:
The default (Python) runner does support cross-language transforms.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410541)
Time Spent: 4h 10m (was: 4h)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410545
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 26/Mar/20 19:42
Start Date: 26/Mar/20 19:42
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10055: [BEAM-8603]
Add Python SqlTransform
URL: https://github.com/apache/beam/pull/10055#discussion_r398838603
##
File path: sdks/python/apache_beam/examples/wordcount_xlang_sql.py
##
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""A cross-language word-counting workflow."""
+
+from __future__ import absolute_import
+
+import argparse
+import logging
+import re
+import typing
+
+from past.builtins import unicode
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.io import ReadFromText
+from apache_beam.io import WriteToText
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.transforms.sql import SqlTransform
+from apache_beam.utils import subprocess_server
+
+MyRow = typing.NamedTuple([('word', unicode)])
+coders.registry.register_coder(MyRow, coders.RowCoder)
+
+
+class WordExtractingDoFn(beam.DoFn):
+ """Parse each line of input text into words."""
+ def process(self, element):
+"""Returns an iterator over the words of this element.
+
+The element is a line of text. If the line is blank, note that, too.
+
+Args:
+ element: the element being processed
+
+Returns:
+ The processed element.
+"""
+text_line = element.strip()
+for word in re.findall(r'[\w\']+', text_line):
+ yield word
+
+
+# Some more fun queries:
+# --
+# SELECT
+# word as key,
+# COUNT(*) as `count`
+# FROM PCOLLECTION
+# GROUP BY word
+# ORDER BY `count` DESC
+# LIMIT 100
+# --
+# SELECT
+# len as key,
+# COUNT(*) as `count`
+# FROM (
+# SELECT
+# LENGTH(word) AS len
+# FROM PCOLLECTION
+# )
+# GROUP BY len
+
+
+def run(p, input_file, output_file):
+ #pylint: disable=expression-not-assigned
+ (
+ p
+ | 'read' >> ReadFromText(input_file)
+ | 'split' >> beam.ParDo(WordExtractingDoFn())
+ | 'row' >> beam.Map(MyRow).with_output_types(MyRow)
+ | 'sql!!' >> SqlTransform(
+ """
+ SELECT
+ word as key,
Review comment:
Does the `as key` add value? (Similarly above.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410545)
Time Spent: 4.5h (was: 4h 20m)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410537=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410537
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 26/Mar/20 19:42
Start Date: 26/Mar/20 19:42
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10055: [BEAM-8603]
Add Python SqlTransform
URL: https://github.com/apache/beam/pull/10055#discussion_r398828335
##
File path: sdks/python/apache_beam/transforms/sql.py
##
@@ -0,0 +1,44 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Transforms that use the SQL Expansion service."""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import typing
+
+from past.builtins import unicode
+
+from apache_beam.transforms.external import BeamJarExpansionService
+from apache_beam.transforms.external import ExternalTransform
+from apache_beam.transforms.external import NamedTupleBasedPayloadBuilder
+
+SqlTransformSchema = typing.NamedTuple(
+'SqlTransformSchema', [('query', unicode)])
+
+
+class SqlTransform(ExternalTransform):
Review comment:
This class should have a docstring, even if it's just a reference to the
Beam documentation on SQL.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410537)
Time Spent: 3h 50m (was: 3h 40m)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410543=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410543
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 26/Mar/20 19:42
Start Date: 26/Mar/20 19:42
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10055: [BEAM-8603]
Add Python SqlTransform
URL: https://github.com/apache/beam/pull/10055#discussion_r398835242
##
File path: sdks/python/apache_beam/transforms/sql_test.py
##
@@ -0,0 +1,109 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for transforms that use the SQL Expansion service."""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import logging
+import typing
+import unittest
+
+from nose.plugins.attrib import attr
+from past.builtins import unicode
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.options.pipeline_options import StandardOptions
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.transforms.sql import SqlTransform
+from apache_beam.utils import subprocess_server
+
+SimpleRow = typing.NamedTuple(
+"SimpleRow", [("int", int), ("str", unicode), ("flt", float)])
+coders.registry.register_coder(SimpleRow, coders.RowCoder)
+
+
+@attr('UsesSqlExpansionService')
+@unittest.skipIf(
+TestPipeline().get_pipeline_options().view_as(StandardOptions).runner is
+None,
+"Must be run with a runner that supports cross-language transforms")
+class SqlTransformTest(unittest.TestCase):
+ """Tests that exercise the cross-language SqlTransform (implemented in java).
+
+ Note this test must be executed with pipeline options that run jobs on a
local
+ job server. The easiest way to accomplish this is to run the
+ `validatesCrossLanguageRunnerPythonUsingSql` gradle target for a particular
+ job server, which will start the runner and job server for you. For example,
+ `:runners:flink:1.10:job-server:validatesCrossLanguageRunnerPythonUsingSql`
to
+ test on Flink 1.10.
+
+ Alternatively, you may be able to iterate faster if you run the tests
directly
+ using a runner like `FlinkRunner`, which starts its own job server, but
you'll
+ need to spin up a local flink cluster:
+$ pip install -e './sdks/python[gcp,test]'
+$ python ./sdks/python/setup.py nosetests \\
+--tests apache_beam.transforms.sql_test \\
+--test-pipeline-options="--runner=FlinkRunner \\
+ --flink_version=1.10 \\
+ --flink_master=localhost:8081"
+ """
+ @staticmethod
+ def make_test_pipeline():
+path_to_jar = subprocess_server.JavaJarServer.path_to_beam_jar(
+":sdks:java:extensions:sql:expansion-service:shadowJar")
+test_pipeline = TestPipeline()
+test_pipeline.get_pipeline_options().view_as(DebugOptions).experiments = [
+'jar_packages=' + path_to_jar
+]
+test_pipeline.not_use_test_runner_api = True
+return test_pipeline
+
+ def test_generate_data(self):
+with self.make_test_pipeline() as p:
+ out = p | SqlTransform(
+ """SELECT
+CAST(1 AS INT) AS `int`,
+CAST('foo' AS VARCHAR) AS `str`,
+CAST(3.14 AS DOUBLE) AS `flt`""")
+ assert_that(out, equal_to([(1, "foo", 3.14)]))
+
+ def test_project(self):
+with self.make_test_pipeline() as p:
+ out = (
+ p | beam.Create([SimpleRow(1, "foo", 3.14)])
+ | SqlTransform("SELECT `int`, `flt` FROM PCOLLECTION"))
+ assert_that(out, equal_to([(1, 3.14)]))
+
+ def test_filter(self):
+with self.make_test_pipeline() as p:
+ out = (
+ p
+ | beam.Create([SimpleRow(1, "foo", 3.14), SimpleRow(2, "bar",
1.414)])
+ | SqlTransform("SELECT * FROM PCOLLECTION WHERE `str` = 'bar'"))
+ assert_that(out, equal_to([(2, "bar", 1.414)]))
+
Review comment:
It'd be great to
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410542=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410542
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 26/Mar/20 19:42
Start Date: 26/Mar/20 19:42
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10055: [BEAM-8603]
Add Python SqlTransform
URL: https://github.com/apache/beam/pull/10055#discussion_r398838169
##
File path: sdks/python/apache_beam/examples/wordcount_xlang_sql.py
##
@@ -0,0 +1,136 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""A cross-language word-counting workflow."""
+
+from __future__ import absolute_import
+
+import argparse
+import logging
+import re
+import typing
+
+from past.builtins import unicode
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.io import ReadFromText
+from apache_beam.io import WriteToText
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.transforms.sql import SqlTransform
+from apache_beam.utils import subprocess_server
+
+MyRow = typing.NamedTuple([('word', unicode)])
+coders.registry.register_coder(MyRow, coders.RowCoder)
+
+
+class WordExtractingDoFn(beam.DoFn):
Review comment:
Nuke this class and just do `FlatMap(lambda line: re.findall(r'[\w\']+',
line))` or, since this is not the focus of this example, even simply
`FlatMap(str.split)` and avoid the save main session ugliness.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410542)
Time Spent: 4h 20m (was: 4h 10m)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410536=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410536
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 26/Mar/20 19:42
Start Date: 26/Mar/20 19:42
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10055: [BEAM-8603]
Add Python SqlTransform
URL: https://github.com/apache/beam/pull/10055#discussion_r398829354
##
File path: sdks/python/apache_beam/transforms/sql.py
##
@@ -0,0 +1,44 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Transforms that use the SQL Expansion service."""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import typing
+
+from past.builtins import unicode
+
+from apache_beam.transforms.external import BeamJarExpansionService
+from apache_beam.transforms.external import ExternalTransform
+from apache_beam.transforms.external import NamedTupleBasedPayloadBuilder
+
+SqlTransformSchema = typing.NamedTuple(
+'SqlTransformSchema', [('query', unicode)])
+
+
+class SqlTransform(ExternalTransform):
+ URN = 'beam:external:java:sql:v1'
+
+ def __init__(self, query):
+super(SqlTransform, self).__init__(
+self.URN,
+NamedTupleBasedPayloadBuilder(SqlTransformSchema(query=query)),
+BeamJarExpansionService(
+':sdks:java:extensions:sql:expansion-service:shadowJar'))
Review comment:
I've been thinking on how to best test these things, and possibly it could
make sense to have a mocking infrastructure (or even pipeline options) where
one could provide a mapping from these hard-coded targets to explicit paths to
jars and/or URLs to pre-started services.
(Likely in a follow-up PR.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410536)
Time Spent: 3h 50m (was: 3h 40m)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410535 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 26/Mar/20 19:42 Start Date: 26/Mar/20 19:42 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10055: [BEAM-8603] Add Python SqlTransform URL: https://github.com/apache/beam/pull/10055#discussion_r398827406 ## File path: sdks/python/apache_beam/transforms/sql.py ## @@ -0,0 +1,44 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +"""Transforms that use the SQL Expansion service.""" Review comment: The fact that it's using an SQL Expansion service is an implementation detail. Instead, I would document this as if they were natively written, e.g. "Transforms that allow one to perform SQL operations on Beam PCollections" with at least a cursory note about schemas, and perhaps an example (or at least pointer to the test as one), and then just a note that the actual implementation is shared with java. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 410535) Time Spent: 3h 40m (was: 3.5h) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410544
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 26/Mar/20 19:42
Start Date: 26/Mar/20 19:42
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10055: [BEAM-8603]
Add Python SqlTransform
URL: https://github.com/apache/beam/pull/10055#discussion_r398843618
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
##
@@ -271,6 +275,19 @@ static Builder builder() {
abstract Builder setDefaultTableProvider(@Nullable String
defaultTableProvider);
abstract SqlTransform build();
+
+@Override
+public PTransform> buildExternal(
+External.Configuration configuration) {
+ return builder()
Review comment:
@chamikaramj @chadrik It'd be nice to reduce this boilerplate to almost
nothing, given we're already using the builder pattern here, or a minimal
mapping of names (e.g. query -> queryString).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410544)
Time Spent: 4.5h (was: 4h 20m)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410540
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 26/Mar/20 19:42
Start Date: 26/Mar/20 19:42
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10055: [BEAM-8603]
Add Python SqlTransform
URL: https://github.com/apache/beam/pull/10055#discussion_r398834130
##
File path: sdks/python/apache_beam/transforms/sql_test.py
##
@@ -0,0 +1,109 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for transforms that use the SQL Expansion service."""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import logging
+import typing
+import unittest
+
+from nose.plugins.attrib import attr
+from past.builtins import unicode
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.options.pipeline_options import StandardOptions
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.transforms.sql import SqlTransform
+from apache_beam.utils import subprocess_server
+
+SimpleRow = typing.NamedTuple(
+"SimpleRow", [("int", int), ("str", unicode), ("flt", float)])
+coders.registry.register_coder(SimpleRow, coders.RowCoder)
+
+
+@attr('UsesSqlExpansionService')
+@unittest.skipIf(
+TestPipeline().get_pipeline_options().view_as(StandardOptions).runner is
+None,
+"Must be run with a runner that supports cross-language transforms")
+class SqlTransformTest(unittest.TestCase):
+ """Tests that exercise the cross-language SqlTransform (implemented in java).
+
+ Note this test must be executed with pipeline options that run jobs on a
local
+ job server. The easiest way to accomplish this is to run the
+ `validatesCrossLanguageRunnerPythonUsingSql` gradle target for a particular
+ job server, which will start the runner and job server for you. For example,
+ `:runners:flink:1.10:job-server:validatesCrossLanguageRunnerPythonUsingSql`
to
+ test on Flink 1.10.
+
+ Alternatively, you may be able to iterate faster if you run the tests
directly
+ using a runner like `FlinkRunner`, which starts its own job server, but
you'll
+ need to spin up a local flink cluster:
+$ pip install -e './sdks/python[gcp,test]'
+$ python ./sdks/python/setup.py nosetests \\
+--tests apache_beam.transforms.sql_test \\
+--test-pipeline-options="--runner=FlinkRunner \\
+ --flink_version=1.10 \\
+ --flink_master=localhost:8081"
+ """
+ @staticmethod
+ def make_test_pipeline():
+path_to_jar = subprocess_server.JavaJarServer.path_to_beam_jar(
+":sdks:java:extensions:sql:expansion-service:shadowJar")
+test_pipeline = TestPipeline()
+test_pipeline.get_pipeline_options().view_as(DebugOptions).experiments = [
+'jar_packages=' + path_to_jar
+]
+test_pipeline.not_use_test_runner_api = True
Review comment:
Is this still needed? If so, maybe it won't be after @chamikaramj 's fixes?
(JIRA?)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410540)
Time Spent: 4h 10m (was: 4h)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 4h
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410538=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410538
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 26/Mar/20 19:42
Start Date: 26/Mar/20 19:42
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10055: [BEAM-8603]
Add Python SqlTransform
URL: https://github.com/apache/beam/pull/10055#discussion_r398833260
##
File path: sdks/python/apache_beam/transforms/sql_test.py
##
@@ -0,0 +1,109 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for transforms that use the SQL Expansion service."""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import logging
+import typing
+import unittest
+
+from nose.plugins.attrib import attr
+from past.builtins import unicode
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.options.pipeline_options import StandardOptions
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.transforms.sql import SqlTransform
+from apache_beam.utils import subprocess_server
+
+SimpleRow = typing.NamedTuple(
+"SimpleRow", [("int", int), ("str", unicode), ("flt", float)])
+coders.registry.register_coder(SimpleRow, coders.RowCoder)
+
+
+@attr('UsesSqlExpansionService')
+@unittest.skipIf(
+TestPipeline().get_pipeline_options().view_as(StandardOptions).runner is
+None,
+"Must be run with a runner that supports cross-language transforms")
+class SqlTransformTest(unittest.TestCase):
+ """Tests that exercise the cross-language SqlTransform (implemented in java).
+
+ Note this test must be executed with pipeline options that run jobs on a
local
+ job server. The easiest way to accomplish this is to run the
+ `validatesCrossLanguageRunnerPythonUsingSql` gradle target for a particular
+ job server, which will start the runner and job server for you. For example,
+ `:runners:flink:1.10:job-server:validatesCrossLanguageRunnerPythonUsingSql`
to
+ test on Flink 1.10.
+
+ Alternatively, you may be able to iterate faster if you run the tests
directly
+ using a runner like `FlinkRunner`, which starts its own job server, but
you'll
+ need to spin up a local flink cluster:
+$ pip install -e './sdks/python[gcp,test]'
+$ python ./sdks/python/setup.py nosetests \\
+--tests apache_beam.transforms.sql_test \\
+--test-pipeline-options="--runner=FlinkRunner \\
+ --flink_version=1.10 \\
+ --flink_master=localhost:8081"
+ """
+ @staticmethod
+ def make_test_pipeline():
+path_to_jar = subprocess_server.JavaJarServer.path_to_beam_jar(
+":sdks:java:extensions:sql:expansion-service:shadowJar")
+test_pipeline = TestPipeline()
+test_pipeline.get_pipeline_options().view_as(DebugOptions).experiments = [
+'jar_packages=' + path_to_jar
Review comment:
@ihji What JIRA should we reference to obviate the need to do this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410538)
Time Spent: 4h (was: 3h 50m)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410539=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410539
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 26/Mar/20 19:42
Start Date: 26/Mar/20 19:42
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10055: [BEAM-8603]
Add Python SqlTransform
URL: https://github.com/apache/beam/pull/10055#discussion_r398832590
##
File path: sdks/python/apache_beam/transforms/sql_test.py
##
@@ -0,0 +1,109 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for transforms that use the SQL Expansion service."""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import logging
+import typing
+import unittest
+
+from nose.plugins.attrib import attr
+from past.builtins import unicode
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.options.pipeline_options import StandardOptions
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.transforms.sql import SqlTransform
+from apache_beam.utils import subprocess_server
+
+SimpleRow = typing.NamedTuple(
+"SimpleRow", [("int", int), ("str", unicode), ("flt", float)])
+coders.registry.register_coder(SimpleRow, coders.RowCoder)
+
+
+@attr('UsesSqlExpansionService')
+@unittest.skipIf(
+TestPipeline().get_pipeline_options().view_as(StandardOptions).runner is
+None,
+"Must be run with a runner that supports cross-language transforms")
+class SqlTransformTest(unittest.TestCase):
+ """Tests that exercise the cross-language SqlTransform (implemented in java).
+
+ Note this test must be executed with pipeline options that run jobs on a
local
+ job server. The easiest way to accomplish this is to run the
+ `validatesCrossLanguageRunnerPythonUsingSql` gradle target for a particular
+ job server, which will start the runner and job server for you. For example,
+ `:runners:flink:1.10:job-server:validatesCrossLanguageRunnerPythonUsingSql`
to
+ test on Flink 1.10.
+
+ Alternatively, you may be able to iterate faster if you run the tests
directly
+ using a runner like `FlinkRunner`, which starts its own job server, but
you'll
+ need to spin up a local flink cluster:
Review comment:
You shouldn't need to spun up a flink cluster, just using
`--runner=FlinkRunner` should be sufficient. (As mentioned, you don't even need
to pass a runner, as the default runner works as well.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 410539)
Time Spent: 4h 10m (was: 4h)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410381=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410381 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 26/Mar/20 16:35 Start Date: 26/Mar/20 16:35 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform URL: https://github.com/apache/beam/pull/10055#issuecomment-604534548 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 410381) Time Spent: 3.5h (was: 3h 20m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=410332=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410332 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 26/Mar/20 15:14 Start Date: 26/Mar/20 15:14 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform URL: https://github.com/apache/beam/pull/10055#issuecomment-604488072 Run XVR_Spark PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 410332) Time Spent: 3h 20m (was: 3h 10m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=409984=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409984 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 26/Mar/20 00:18 Start Date: 26/Mar/20 00:18 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform URL: https://github.com/apache/beam/pull/10055#issuecomment-604156803 Run XVR_Spark PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409984) Time Spent: 3h 10m (was: 3h) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=409775=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409775 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 25/Mar/20 19:53 Start Date: 25/Mar/20 19:53 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform URL: https://github.com/apache/beam/pull/10055#issuecomment-604052011 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409775) Time Spent: 3h (was: 2h 50m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=409733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409733 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 25/Mar/20 19:02 Start Date: 25/Mar/20 19:02 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform URL: https://github.com/apache/beam/pull/10055#issuecomment-604027617 R: @robertwb could you review this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409733) Time Spent: 2h 50m (was: 2h 40m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=409694=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409694 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 25/Mar/20 18:27 Start Date: 25/Mar/20 18:27 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform URL: https://github.com/apache/beam/pull/10055#issuecomment-604009131 Run XVR_Spark PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409694) Time Spent: 2h 40m (was: 2.5h) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=409126=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409126 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 24/Mar/20 22:11 Start Date: 24/Mar/20 22:11 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform URL: https://github.com/apache/beam/pull/10055#issuecomment-603531715 Run XVR_Spark PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409126) Time Spent: 2h 20m (was: 2h 10m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=409127=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409127 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 24/Mar/20 22:11 Start Date: 24/Mar/20 22:11 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform URL: https://github.com/apache/beam/pull/10055#issuecomment-603531761 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409127) Time Spent: 2.5h (was: 2h 20m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=408886=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408886 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 24/Mar/20 16:18 Start Date: 24/Mar/20 16:18 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform URL: https://github.com/apache/beam/pull/10055#issuecomment-603340845 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408886) Time Spent: 2h 10m (was: 2h) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=408437=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408437 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 24/Mar/20 00:26 Start Date: 24/Mar/20 00:26 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform example script URL: https://github.com/apache/beam/pull/10055#issuecomment-602931763 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408437) Time Spent: 1h 50m (was: 1h 40m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=408438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408438 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 24/Mar/20 00:26 Start Date: 24/Mar/20 00:26 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform example script URL: https://github.com/apache/beam/pull/10055#issuecomment-602931816 Run XVR_Spark PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408438) Time Spent: 2h (was: 1h 50m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=402486=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-402486 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 12/Mar/20 21:02 Start Date: 12/Mar/20 21:02 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform example script URL: https://github.com/apache/beam/pull/10055#issuecomment-598416755 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 402486) Time Spent: 1h 40m (was: 1.5h) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=401743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401743 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 11/Mar/20 20:07 Start Date: 11/Mar/20 20:07 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform example script URL: https://github.com/apache/beam/pull/10055#issuecomment-597841657 I force pushed some significant changes to this PR. Now rather than adding SQL as a dependency in the test expansion service, this adds a new `:sdks:java:extensions:sql:expansion-service` artifact that produces a shaded jar with `SqlTransform` registered. The `wordcount_xlang_sql` script is meant to be called with this jar, which it will execute and use as the expansion service, and _also_ send to workers via the `jar_packages` experiment. The PR also includes the changes from #11008, which should be merged first. @chamikaramj can you take another look now? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401743) Time Spent: 1.5h (was: 1h 20m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=376973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376973 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 24/Jan/20 18:33 Start Date: 24/Jan/20 18:33 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform example script URL: https://github.com/apache/beam/pull/10055#issuecomment-578247963 Thanks stalebot. I'd still like to look at this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376973) Time Spent: 1h 20m (was: 1h 10m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=374366=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374366 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 20/Jan/20 00:34 Start Date: 20/Jan/20 00:34 Worklog Time Spent: 10m Work Description: stale[bot] commented on issue #10055: [BEAM-8603] Add Python SqlTransform example script URL: https://github.com/apache/beam/pull/10055#issuecomment-576065789 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the d...@beam.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 374366) Time Spent: 1h 10m (was: 1h) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=347087=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-347087
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 21/Nov/19 00:19
Start Date: 21/Nov/19 00:19
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #10055:
[BEAM-8603] Add Python SqlTransform example script
URL: https://github.com/apache/beam/pull/10055#discussion_r348841220
##
File path: sdks/java/extensions/sql/build.gradle
##
@@ -24,6 +24,7 @@ plugins {
}
applyJavaNature(
automaticModuleName: 'org.apache.beam.sdk.extensions.sql',
+ shadowClosure: {},
Review comment:
We need two jars.
(1) Jar for expansion service that contains the transform classes that need
to be expanded (this can be multiple jars as well, IO transforms just have to
be in the class path to be picked up by the AutoService).
(2) Jar to be passed to the Java worker. For me, building a shadow Jar of
java-harness (:sdks:java:harness:shadowJar) worked.
It probably makes sens to move expansion service to it's own Gradle module
and support building a shadow jar with all in-built cross-language transforms
in Beam and release that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 347087)
Time Spent: 1h (was: 50m)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=346689=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-346689 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 20/Nov/19 12:56 Start Date: 20/Nov/19 12:56 Worklog Time Spent: 10m Work Description: robinyqiu commented on issue #10055: [BEAM-8603] Add Python SqlTransform example script URL: https://github.com/apache/beam/pull/10055#issuecomment-555992144 Thank you Brian for putting this PR together and writing up the instructions. That's very helpful! I run the demo in this PR at Apache Beam Zurich meetup and it went very well! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 346689) Time Spent: 50m (was: 40m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=342042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342042
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 12/Nov/19 17:02
Start Date: 12/Nov/19 17:02
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10055: [BEAM-8603]
Add Python SqlTransform example script
URL: https://github.com/apache/beam/pull/10055#discussion_r345328998
##
File path: sdks/java/extensions/sql/build.gradle
##
@@ -24,6 +24,7 @@ plugins {
}
applyJavaNature(
automaticModuleName: 'org.apache.beam.sdk.extensions.sql',
+ shadowClosure: {},
Review comment:
I don't think we want to do this since normal SQL users don't want to be
using an artifact meant for the expansion service. It seems as though we should
have the expansion service gradle project build one uber artifact containing
all the xlang transforms we want to vend.
We could have multiple shadowJar tasks and produce multiple artifacts along
this line but this seems overly complex initially.
@chamikaramj Is there an initial strategy we are following for how we want
to build xlang expansion service jars?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342042)
Time Spent: 40m (was: 0.5h)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[
https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=341591=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341591
]
ASF GitHub Bot logged work on BEAM-8603:
Author: ASF GitHub Bot
Created on: 12/Nov/19 01:48
Start Date: 12/Nov/19 01:48
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #10055:
[BEAM-8603] Add Python SqlTransform example script
URL: https://github.com/apache/beam/pull/10055#discussion_r344985062
##
File path: sdks/java/extensions/sql/build.gradle
##
@@ -24,6 +24,7 @@ plugins {
}
applyJavaNature(
automaticModuleName: 'org.apache.beam.sdk.extensions.sql',
+ shadowClosure: {},
Review comment:
@lukecwik I added this so I can build a single jar to pass to Java workers
for the xlang transform (see the
`--experiment='jar_packages=./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.18.0-SNAPSHOT.jar'
\` line in my "how to run" in the description), but it looks like it's causing
`:sdks:java:extensions:sql:validateShadedJarDoesntLeakNonProjectClasses` to
fail spectacularly. Is there some other way I can do this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 341591)
Time Spent: 0.5h (was: 20m)
> Add Python SqlTransform example script
> --
>
> Key: BEAM-8603
> URL: https://issues.apache.org/jira/browse/BEAM-8603
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=340866=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-340866 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 09/Nov/19 02:18 Start Date: 09/Nov/19 02:18 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform example script URL: https://github.com/apache/beam/pull/10055#issuecomment-552056273 CC: @robinyqiu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 340866) Time Spent: 20m (was: 10m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=340865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-340865 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 09/Nov/19 02:18 Start Date: 09/Nov/19 02:18 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on pull request #10055: [BEAM-8603] Add Python SqlTransform example script URL: https://github.com/apache/beam/pull/10055 To run: 1. Start a Flink cluster with master url `localhost:8081` (e.g. by downloading a [flink release](https://flink.apache.org/downloads.html#apache-flink-182) and running `./bin/start-cluster.sh`) 2. Start a job server pointing to this cluster: ``` ./gradlew :runners:flink:1.8:job-server:runShadow -PflinkMasterUrl=localhost:8081` ``` 3. Make sure the expansion service jar and java, python containers have been built: ``` ./gradlew :sdks:java:testing:expansion-service:buildTestExpansionServiceJar :sdks:java:container:docker :sdks:python:container:py2:docker ``` 4. Run the example script: ``` python -m apache_beam.examples.wordcount_xlang_sql \ --expansion_service_jar sdks/java/testing/expansion-service/build/libs/beam-sdks-java-testing-expansion-service-testExpansionService-2.18.0-SNAPSHOT.jar \ --experiment='jar_packages=./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.18.0-SNAPSHOT.jar' \ --runner PortableRunner \ --job_endpoint localhost:8099 \ --input 'gs://dataflow-samples/shakespeare/kinglear.txt' \ --output gs://your/bucket/output ``` 5. Inspect the output: `gsutil cat 'gs://your/bucket/output/*'` Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [