[jira] [Commented] (BEAM-2083) Develop a Go SDK for Beam
[
https://issues.apache.org/jira/browse/BEAM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283021#comment-16283021
]
ASF GitHub Bot commented on BEAM-2083:
--
kennknowles closed pull request #4230: [BEAM-2083] Make Scope a value type in
Go SDK
URL: https://github.com/apache/beam/pull/4230
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/go/examples/contains/contains.go
b/sdks/go/examples/contains/contains.go
index 7bdfb10b82c..45a8527a851 100644
--- a/sdks/go/examples/contains/contains.go
+++ b/sdks/go/examples/contains/contains.go
@@ -42,7 +42,7 @@ func init() {
}
// FilterWords returns PCollection> with (up to) 10 matching
words.
-func FilterWords(s *beam.Scope, lines beam.PCollection) beam.PCollection {
+func FilterWords(s beam.Scope, lines beam.PCollection) beam.PCollection {
s = s.Scope("FilterWords")
words := beam.ParDo(s, extractFn, lines)
filtered := beam.ParDo(s, {Search: *search}, words)
diff --git a/sdks/go/examples/cookbook/combine/combine.go
b/sdks/go/examples/cookbook/combine/combine.go
index 5d9fe77de98..7e24aa1fb30 100644
--- a/sdks/go/examples/cookbook/combine/combine.go
+++ b/sdks/go/examples/cookbook/combine/combine.go
@@ -50,7 +50,7 @@ type PlaysRow struct {
// PlaysForWords generates a string containing the list of play names
// in which that word appears. It takes a PCollection and
// returns a PCollection.
-func PlaysForWords(s *beam.Scope, rows beam.PCollection) beam.PCollection {
+func PlaysForWords(s beam.Scope, rows beam.PCollection) beam.PCollection {
s = s.Scope("PlaysForWords")
words := beam.ParDo(s, {MinLength: *minLength}, rows)
diff --git a/sdks/go/examples/cookbook/filter/filter.go
b/sdks/go/examples/cookbook/filter/filter.go
index 66998c4509a..effc1225655 100644
--- a/sdks/go/examples/cookbook/filter/filter.go
+++ b/sdks/go/examples/cookbook/filter/filter.go
@@ -45,7 +45,7 @@ type WeatherDataRow struct {
// BelowGlobalMean computes the rows for the given month below the global
mean. It takes a
// PCollection and returns a PCollection.
-func BelowGlobalMean(s *beam.Scope, m int, rows beam.PCollection)
beam.PCollection {
+func BelowGlobalMean(s beam.Scope, m int, rows beam.PCollection)
beam.PCollection {
s = s.Scope("BelowGlobalMean")
// Find the global mean of all the mean_temp readings in the weather
data.
diff --git a/sdks/go/examples/cookbook/max/max.go
b/sdks/go/examples/cookbook/max/max.go
index eb7e07dec63..f0184354afa 100644
--- a/sdks/go/examples/cookbook/max/max.go
+++ b/sdks/go/examples/cookbook/max/max.go
@@ -47,7 +47,7 @@ type MaxMeanTempRow struct {
// MaxMeanTemp finds the max mean_temp for each month. It takes a
// PCollection and returns a PCollection.
-func MaxMeanTemp(s *beam.Scope, rows beam.PCollection) beam.PCollection {
+func MaxMeanTemp(s beam.Scope, rows beam.PCollection) beam.PCollection {
s = s.Scope("MaxMeanTemp")
keyed := beam.ParDo(s, extractFn, rows)
diff --git a/sdks/go/examples/cookbook/tornadoes/tornadoes.go
b/sdks/go/examples/cookbook/tornadoes/tornadoes.go
index d46ad60bb49..058948ba32e 100644
--- a/sdks/go/examples/cookbook/tornadoes/tornadoes.go
+++ b/sdks/go/examples/cookbook/tornadoes/tornadoes.go
@@ -71,7 +71,7 @@ type TornadoRow struct {
// CountTornadoes computes the number of tornadoes pr month. It takes a
// PCollection and returns a PCollection.
-func CountTornadoes(s *beam.Scope, rows beam.PCollection) beam.PCollection {
+func CountTornadoes(s beam.Scope, rows beam.PCollection) beam.PCollection {
s = s.Scope("CountTornadoes")
// row... => month...
diff --git a/sdks/go/examples/debugging_wordcount/debugging_wordcount.go
b/sdks/go/examples/debugging_wordcount/debugging_wordcount.go
index 17b7ccacc2c..a17a75f3104 100644
--- a/sdks/go/examples/debugging_wordcount/debugging_wordcount.go
+++ b/sdks/go/examples/debugging_wordcount/debugging_wordcount.go
@@ -122,7 +122,7 @@ func formatFn(w string, c int) string {
// CountWords is a composite transform that counts the words of an PCollection
// of lines. It expects a PCollection of type string and returns a PCollection
// of type KV.
-func CountWords(s *beam.Scope, lines beam.PCollection) beam.PCollection {
+func CountWords(s beam.Scope, lines beam.PCollection) beam.PCollection {
s = s.Scope("CountWords")
col := beam.ParDo(s, extractFn, lines)
return stats.Count(s, col)
diff --git a/sdks/go/examples/forest/forest.go
b/sdks/go/examples/forest/forest.go
index af61ce64230..06ae7831a45 100644
--- a/sdks/go/examples/forest/forest.go
+++ b/sdks/go/examples/forest/forest.go
@@ -42,7 +42,7 @@ var (
[jira] [Commented] (BEAM-2083) Develop a Go SDK for Beam
[ https://issues.apache.org/jira/browse/BEAM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16282557#comment-16282557 ] ASF GitHub Bot commented on BEAM-2083: -- herohde opened a new pull request #4230: [BEAM-2083] Make Scope a value type in Go SDK URL: https://github.com/apache/beam/pull/4230 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Develop a Go SDK for Beam > - > > Key: BEAM-2083 > URL: https://issues.apache.org/jira/browse/BEAM-2083 > Project: Beam > Issue Type: New Feature > Components: sdk-go >Reporter: Bill Neubauer >Assignee: Henning Rohde > > Allow users of the Go programming language (https://golang.org/) to write > Beam pipelines in this language. The effort is focusing on full-fledged SDK > that leverages the Beam Fn API to bootstrap a native Go experience. > Initial design: > https://s.apache.org/beam-go-sdk-design-rfc > Development in the go-sdk branch. Work in progress. YMMV. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-2083) Develop a Go SDK for Beam
[
https://issues.apache.org/jira/browse/BEAM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16282198#comment-16282198
]
ASF GitHub Bot commented on BEAM-2083:
--
kennknowles closed pull request #4217: [BEAM-2083] Add Go SDK README
URL: https://github.com/apache/beam/pull/4217
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/go/README.md b/sdks/go/README.md
new file mode 100644
index 000..ed0a669dd53
--- /dev/null
+++ b/sdks/go/README.md
@@ -0,0 +1,98 @@
+
+
+# Go SDK (experimental)
+
+The Go SDK is currently an experimental feature of Apache Beam and
+not suitable for production use. It is based on the following initial
+[design](https://s.apache.org/beam-go-sdk-design-rfc).
+
+## How to run the examples
+
+**Prerequisites**: to use Google Cloud sources and sinks (default for
+most examples), follow the setup
+[here](https://beam.apache.org/documentation/runners/dataflow/). You can
+verify that it works by running the corresponding Java example.
+
+The examples are normal Go programs and are most easily run directly. They
+are parameterized by Go flags. For example, to run wordcount do:
+
+```
+$ pwd
+[...]/sdks/go
+$ go run examples/wordcount/wordcount.go --output=/tmp/result.txt
+2017/12/05 10:46:37 Pipeline:
+2017/12/05 10:46:37 Nodes: {1: W<[]uint8>/GW/W!GW}
+{2: W/GW/W!GW}
+{3: W/GW/W!GW}
+{4: W/GW/W!GW}
+{5: W/GW/W!GW}
+{6: W>/GW/W>!GW}
+{7: W>/GW/W>!GW}
+{8: W>/GW/W>!GW}
+{9: W/GW/W!GW}
+Edges: 1: Impulse [] -> [Out: W<[]uint8> -> {1: W<[]uint8>/GW/W!GW}]
+2: ParDo [In(Main): W<[]uint8> <- {1: W<[]uint8>/GW/W!GW}] -> [Out:
W -> {2: W/GW/W!GW}]
+3: ParDo [In(Main): W <- {2: W/GW/W!GW}] -> [Out:
W -> {3: W/GW/W!GW}]
+4: ParDo [In(Main): W <- {3: W/GW/W!GW}] -> [Out:
W -> {4: W/GW/W!GW}]
+5: ParDo [In(Main): W <- {4: W/GW/W!GW}] -> [Out:
W -> {5: W/GW/W!GW}]
+6: ParDo [In(Main): W <- {5: W/GW/W!GW}] -> [Out:
W> -> {6: W>/GW/W>!GW}]
+7: GBK [In(Main): KV <- {6:
W>/GW/W>!GW}] -> [Out: GBK -> {7:
W>/GW/W>!GW}]
+8: Combine [In(Main): W <- {7:
W>/GW/W>!GW}] -> [Out: W>
-> {8: W>/GW/W>!GW}]
+9: ParDo [In(Main): W> <- {8:
W>/GW/W>!GW}] -> [Out: W -> {9:
W/GW/W!GW}]
+10: ParDo [In(Main): W <- {9: W/GW/W!GW}] -> []
+2017/12/05 10:46:37 Execution units:
+2017/12/05 10:46:37 1: Impulse[0]
+2017/12/05 10:46:37 2: ParDo[beam.createFn] Out:[3]
+2017/12/05 10:46:37 3: ParDo[textio.expandFn] Out:[4]
+2017/12/05 10:46:37 4: ParDo[textio.readFn] Out:[5]
+2017/12/05 10:46:37 5: ParDo[main.extractFn] Out:[6]
+2017/12/05 10:46:37 6: ParDo[stats.mapFn] Out:[7]
+2017/12/05 10:46:37 7: GBK. Out:8
+2017/12/05 10:46:37 8: Combine[stats.sumIntFn] Keyed:true (Use:false) Out:[9]
+2017/12/05 10:46:37 9: ParDo[main.formatFn] Out:[10]
+2017/12/05 10:46:37 10: ParDo[textio.writeFileFn] Out:[]
+2017/12/05 10:46:37 Reading from
gs://apache-beam-samples/shakespeare/kinglear.txt
+2017/12/05 10:46:38 Writing to /tmp/result.txt
+```
+
+The debugging output is currently quite verbose and likely to change. The
output is a local
+file in this case:
+
+```
+$ head /tmp/result.txt
+while: 2
+darkling: 1
+rail'd: 1
+ford: 1
+bleed's: 1
+hath: 52
+Remain: 1
+disclaim: 1
+sentence: 1
+purse: 6
+```
+
+See [BUILD.md](./BUILD.md) for how to build Go code in general. See
+[CONTAINERS.md](../CONTAINERS.md) for how to build and push the Go
+SDK harness container image.
+
+## Issues
+
+Please use the `sdk-go` component for any bugs or feature requests.
\ No newline at end of file
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Develop a Go SDK for Beam
> -
>
> Key: BEAM-2083
> URL: https://issues.apache.org/jira/browse/BEAM-2083
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Henning Rohde
>
> Allow users of the Go programming language (https://golang.org/) to write
> Beam pipelines in this language. The effort is focusing on full-fledged SDK
> that leverages the Beam Fn API to bootstrap a native Go experience.
> Initial
[jira] [Commented] (BEAM-2083) Develop a Go SDK for Beam
[ https://issues.apache.org/jira/browse/BEAM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279088#comment-16279088 ] ASF GitHub Bot commented on BEAM-2083: -- herohde opened a new pull request #4217: [BEAM-2083] Add Go SDK README URL: https://github.com/apache/beam/pull/4217 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Develop a Go SDK for Beam > - > > Key: BEAM-2083 > URL: https://issues.apache.org/jira/browse/BEAM-2083 > Project: Beam > Issue Type: New Feature > Components: sdk-go >Reporter: Bill Neubauer >Assignee: Henning Rohde > > Allow users of the Go programming language (https://golang.org/) to write > Beam pipelines in this language. The effort is focusing on full-fledged SDK > that leverages the Beam Fn API to bootstrap a native Go experience. > Initial design: > https://s.apache.org/beam-go-sdk-design-rfc > Development in the go-sdk branch (pending PR#4200). Work in progress. YMMV. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-2083) Develop a Go SDK for Beam
[ https://issues.apache.org/jira/browse/BEAM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16275109#comment-16275109 ] ASF GitHub Bot commented on BEAM-2083: -- kennknowles commented on issue #4200: [BEAM-2083] Go SDK prototype URL: https://github.com/apache/beam/pull/4200#issuecomment-348632719 Test failures are not related. :-( This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Develop a Go SDK for Beam > - > > Key: BEAM-2083 > URL: https://issues.apache.org/jira/browse/BEAM-2083 > Project: Beam > Issue Type: New Feature > Components: sdk-ideas >Reporter: Bill Neubauer >Assignee: Henning Rohde > > Allow users of the Go programming language (https://golang.org/) to write > Beam pipelines in this language. > The effort is focusing on full-fledged SDK that leverages the Beam Fn API to > bootstrap a native Go experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-2083) Develop a Go SDK for Beam
[ https://issues.apache.org/jira/browse/BEAM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16275110#comment-16275110 ] ASF GitHub Bot commented on BEAM-2083: -- kennknowles commented on issue #4200: [BEAM-2083] Go SDK prototype URL: https://github.com/apache/beam/pull/4200#issuecomment-348632719 Test failures are not related. Great to see it green already. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Develop a Go SDK for Beam > - > > Key: BEAM-2083 > URL: https://issues.apache.org/jira/browse/BEAM-2083 > Project: Beam > Issue Type: New Feature > Components: sdk-ideas >Reporter: Bill Neubauer >Assignee: Henning Rohde > > Allow users of the Go programming language (https://golang.org/) to write > Beam pipelines in this language. > The effort is focusing on full-fledged SDK that leverages the Beam Fn API to > bootstrap a native Go experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-2083) Develop a Go SDK for Beam
[ https://issues.apache.org/jira/browse/BEAM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273723#comment-16273723 ] ASF GitHub Bot commented on BEAM-2083: -- herohde commented on issue #4200: [BEAM-2083] Go SDK prototype URL: https://github.com/apache/beam/pull/4200#issuecomment-348365198 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Develop a Go SDK for Beam > - > > Key: BEAM-2083 > URL: https://issues.apache.org/jira/browse/BEAM-2083 > Project: Beam > Issue Type: New Feature > Components: sdk-ideas >Reporter: Bill Neubauer >Assignee: Henning Rohde >Priority: Minor > > Allow users of the Go programming language (https://golang.org/) to write > Beam pipelines in this language. > The effort is focusing on full-fledged SDK that leverages the Beam Fn API to > bootstrap a native Go experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-2083) Develop a Go SDK for Beam
[ https://issues.apache.org/jira/browse/BEAM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273664#comment-16273664 ] ASF GitHub Bot commented on BEAM-2083: -- herohde opened a new pull request #4200: [BEAM-2083] Go SDK prototype URL: https://github.com/apache/beam/pull/4200 Prototype of a Go SDK based on the design described here: https://s.apache.org/beam-go-sdk-design-rfc Work in progress. Note that PR is to the go-sdk branch. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Develop a Go SDK for Beam > - > > Key: BEAM-2083 > URL: https://issues.apache.org/jira/browse/BEAM-2083 > Project: Beam > Issue Type: New Feature > Components: sdk-ideas >Reporter: Bill Neubauer >Assignee: Henning Rohde >Priority: Minor > > Allow users of the Go programming language (https://golang.org/) to write > Beam pipelines in this language. > The effort is focusing on full-fledged SDK that leverages the Beam Fn API to > bootstrap a native Go experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-2083) Develop a Go SDK for Beam
[ https://issues.apache.org/jira/browse/BEAM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116077#comment-16116077 ] ASF GitHub Bot commented on BEAM-2083: -- GitHub user c0b opened a pull request: https://github.com/apache/beam/pull/3694 could you allow github issues here? [dummy pr for issue comment only] _I don't understand why do you require jira ticket instead of github issues here; here I'd only want to comment on the tickets but creating an account on https://issues.apache.org for comment is a broken user experience (compared to github issues)_ - https://issues.apache.org/jira/browse/BEAM-2083 for Go SDK - https://issues.apache.org/jira/browse/BEAM-1754 for NodeJS SDK - https://issues.apache.org/jira/browse/BEAM-14 for a generic declarative DSL for any language SDK writers can use from the https://beam.apache.org/documentation/runners/capability-matrix/ I did my first test run is to see how many runners supported by existing languages (Java & Python); I did test the example wordcount with both Java and Python, from this error from Python seems like it does not have most other runners, and Python so far only support Direct and DataflowRunner, still lack important features like triggers? ValueError: Unexpected pipeline runner: ApexRunner. Valid values are DirectRunner, EagerRunner, DataflowRunner, TestDataflowRunner or the fully qualified name of a PipelineRunner subclass. so focus at DataflowRunner with Python or try another programming language: dataflow is providing REST API calls, but the difficulty here for another programming language is how to provide the job create request body? especially, how define and encode the job steps ? https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs#Job.Step from two test runs of wordcount examples, so far I found the clues: 1. with jobs list api with `view=JOB_VIEW_ALL` I can see java and python uses a different **workerHarnessContainerImage**, so I do docker pull these images to locally to look into, but where are the source code for each? are these open sourced? what is the default entrypoint `/opt/google/dataflow/boot` ? "workerHarnessContainerImage": "dataflow.gcr.io/v1beta3/beam-java-batch:beam-2.0.0" "workerHarnessContainerImage": "dataflow.gcr.io/v1beta3/python:2.0.0" ```console $ docker images --filter='reference=dataflow.gcr.io/v1beta3/*:*' REPOSITORYTAG IMAGE ID CREATED SIZE dataflow.gcr.io/v1beta3/python2.0.0 2a1e69afbef9 2 months ago1.3GB dataflow.gcr.io/v1beta3/beam-java-batch beam-2.0.0 2686ad94cb93 5 months ago393MB $ docker run -it --rm --entrypoint=/bin/bash dataflow.gcr.io/v1beta3/python:2.0.0 ... root@ddfe741352d6:/# \du -sh /usr/local/gcloud/google-cloud-sdk /usr/local/lib/python2.7/dist-packages/tensorflow /usr/local/lib/python2.7/dist-packages/scipy /usr/local/lib/python2.7/dist-packages/sklearn /opt/google/dataflow 226M/usr/local/gcloud/google-cloud-sdk 167M/usr/local/lib/python2.7/dist-packages/tensorflow 155M/usr/local/lib/python2.7/dist-packages/scipy 72M /usr/local/lib/python2.7/dist-packages/sklearn 26M /opt/google/dataflow root@ddfe741352d6:/# ls -lih /opt/google/dataflow total 26M 19005540 -r-xr-xr-x 1 root root 43K Jan 1 1970 NOTICES.shuffle 19005538 -r-xr-xr-x 1 root root 14M Jan 1 1970 boot 19005539 -r-xr-xr-x 1 root root 680K Jan 1 1970 dataflow_python_worker.tar 19005541 -r-xr-xr-x 1 root root 12M Jan 1 1970 shuffle_client.so ``` 2. the REST API only defined each step requires a `kind`, `name` and `properties`; but what's the internal structure of `properties` ? for the python one I spent some time figured out the `serialized_fn` is base64 encoded of zlib compressed of a pickle serialized object code of the python function, and Java version's serialized_fn is using another way of serialization of a function (looks like snappy compression of java byte code?), so the question is here: does this mean the `properties` is complete up to SDK Writers? if somebody is going to do Go or NodeJS, since different language has very different way of serialization of a function's code; all these will look like duplicating a lot of effort, then BEAM-14 could be a better approach? but generally, could you share more necessary documentation for SDK writters? so far I feel these are necessary: **1) define a function serialization protocol, to be used in the `steps / properties`** **2) a language specific docker image, to be used as `workerHarnessContainerImage`, this image will need to interpret the serialization
