[
https://issues.apache.org/jira/browse/BEAM-9679?focusedWorklogId=437575&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437575
]
ASF GitHub Bot logged work on BEAM-9679:
----------------------------------------
Author: ASF GitHub Bot
Created on: 27/May/20 04:09
Start Date: 27/May/20 04:09
Worklog Time Spent: 10m
Work Description: henryken commented on a change in pull request #11803:
URL: https://github.com/apache/beam/pull/11803#discussion_r429965845
##########
File path: learning/katas/go/Core
Transforms/CoGroupByKey/CoGroupByKey/pkg/task/task.go
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "fmt"
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, fruits beam.PCollection, countries
beam.PCollection) beam.PCollection {
+ fruitsKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, fruits)
+
+ countriesKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, countries)
+
+ grouped := beam.CoGroupByKey(s, fruitsKV, countriesKV)
+ return beam.ParDo(s, func(key string, f func(*string) bool, c
func(*string) bool, emit func(string)) {
+ v := &WordsAlphabet{
Review comment:
Why not using `wa` here so that not to confuse with Value?
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/task.md
##########
@@ -0,0 +1,104 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# CoGroupByKey
+
+CoGroupByKey performs a relational join of two or more key/value PCollections
that have the same
+key type.
+
+**Kata:** Implement a
[beam.CoGroupByKey](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey)
+transform that join words by the first alphabetical letter, and then produces
the string representation of the
+WordsAlphabet model.
+
+<div class="hint">
+ Refer to
+ <a
href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey";>beam.CoGroupByKey</a>
+ to solve this problem.
+</div>
+
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a
href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey";>
+ "CoGroupByKey"</a> section for more information.
+</div>
+
+<div class="hint">
+ Think of this problem in three stages. First, create key/value pairs of
PCollections called KV
+ for fruits and countries, pairing the first character with the word. Next,
apply CoGroupByKey to the KVs
+ followed by a ParDo.
+</div>
+
+<div class="hint">
+ In the last lesson we learned how to make key/value PCollections called KV.
Now we have
+ two to make from fruits and countries.
+
+ To return as a KV, you can return two values from your DoFn. The first
return value represents the Key, and
+ the second return value represents the Value. An example is shown below.
+
+```
+func doFn(element string) (string, string) {
+ key := string(element[0])
+ value := element
+ return key, value
+}
+```
+</div>
+
+<div class="hint">
+ In the last lesson we learned that
+ <a
href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#GroupByKey";>
+ beam.GroupByKey</a> takes a single KV.
+ <a
href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey";>beam.CoGroupByKey</a>
+ takes more than one KV.
+</div>
+
+<div class="hint">
+ Our final step in this problem requires a
+ <a
href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#ParDo";>beam.ParDo</a>
+ with a DoFn that's different than what we've seen in previous lessons. In
the previous step we should
+ have a PCollection acquired from CoGroupByKey. A ParDo for that PCollection
expects a DoFn that looks
+ like the following.
+
+ ```
+ func doFn(key string, aKV func(*string) bool, anotherKV func(*string) bool,
emit func(string)){
Review comment:
If I understand correctly, it seems that `func(*string) bool` should
assign the value of V to the passed-in variable pointer, not the KV?
Borrowing the concept of the Java version, these functions behave similarly
to CoGbkResult.get() based on the sequence of the passed in PCollection?
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/go.mod
##########
@@ -0,0 +1,26 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+module cogroupbykey
+
+go 1.14
Review comment:
Just realized that the GroupByKey and CoGroupByKey are not using the
same Go version as the others. Can we make all of them uniform?
##########
File path: learning/katas/go/Core
Transforms/CoGroupByKey/CoGroupByKey/pkg/task/task.go
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "fmt"
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, fruits beam.PCollection, countries
beam.PCollection) beam.PCollection {
+ fruitsKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, fruits)
+
+ countriesKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, countries)
+
+ grouped := beam.CoGroupByKey(s, fruitsKV, countriesKV)
+ return beam.ParDo(s, func(key string, f func(*string) bool, c
func(*string) bool, emit func(string)) {
+ v := &WordsAlphabet{
+ Alphabet: key,
+ }
+ f(&v.Fruit)
+ c(&v.Country)
+ emit(v.String())
+ }, grouped)
+}
+
+type WordsAlphabet struct {
+ Alphabet string
+ Fruit string
+ Country string
+}
+
+func (wa *WordsAlphabet) String() string {
+ return fmt.Sprintf("WordsAlphabet%+v", *wa)
Review comment:
Nice trick!
##########
File path: learning/katas/go/Core
Transforms/CoGroupByKey/CoGroupByKey/pkg/task/task.go
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "fmt"
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, fruits beam.PCollection, countries
beam.PCollection) beam.PCollection {
+ fruitsKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, fruits)
+
+ countriesKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, countries)
+
+ grouped := beam.CoGroupByKey(s, fruitsKV, countriesKV)
+ return beam.ParDo(s, func(key string, f func(*string) bool, c
func(*string) bool, emit func(string)) {
Review comment:
I understand the convention of Go that prefers shorter variable name,
but I personally find that `f` and `c` are cryptic.
But if it's well understood, I guess I'm just too old for this :)
Btw, I like the variable naming in the examples at the top of the godoc.
https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/task.md
##########
@@ -0,0 +1,104 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# CoGroupByKey
+
+CoGroupByKey performs a relational join of two or more key/value PCollections
that have the same
+key type.
+
+**Kata:** Implement a
[beam.CoGroupByKey](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey)
+transform that join words by the first alphabetical letter, and then produces
the string representation of the
+WordsAlphabet model.
+
+<div class="hint">
+ Refer to
+ <a
href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey";>beam.CoGroupByKey</a>
+ to solve this problem.
+</div>
+
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a
href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey";>
+ "CoGroupByKey"</a> section for more information.
+</div>
+
+<div class="hint">
+ Think of this problem in three stages. First, create key/value pairs of
PCollections called KV
+ for fruits and countries, pairing the first character with the word. Next,
apply CoGroupByKey to the KVs
+ followed by a ParDo.
+</div>
+
+<div class="hint">
+ In the last lesson we learned how to make key/value PCollections called KV.
Now we have
+ two to make from fruits and countries.
+
+ To return as a KV, you can return two values from your DoFn. The first
return value represents the Key, and
+ the second return value represents the Value. An example is shown below.
+
+```
+func doFn(element string) (string, string) {
+ key := string(element[0])
+ value := element
+ return key, value
+}
+```
+</div>
+
+<div class="hint">
+ In the last lesson we learned that
+ <a
href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#GroupByKey";>
+ beam.GroupByKey</a> takes a single KV.
+ <a
href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey";>beam.CoGroupByKey</a>
+ takes more than one KV.
+</div>
+
+<div class="hint">
+ Our final step in this problem requires a
+ <a
href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#ParDo";>beam.ParDo</a>
+ with a DoFn that's different than what we've seen in previous lessons. In
the previous step we should
+ have a PCollection acquired from CoGroupByKey. A ParDo for that PCollection
expects a DoFn that looks
+ like the following.
+
+ ```
+ func doFn(key string, aKV func(*string) bool, anotherKV func(*string) bool,
emit func(string)){
Review comment:
Is there any documentation available that explains about this DoFn
variation?
I still couldn't find a good programming guide for the Go version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 437575)
Time Spent: 2h 50m (was: 2h 40m)
> Core Transforms | Go SDK Code Katas
> -----------------------------------
>
> Key: BEAM-9679
> URL: https://issues.apache.org/jira/browse/BEAM-9679
> Project: Beam
> Issue Type: Sub-task
> Components: katas, sdk-go
> Reporter: Damon Douglas
> Assignee: Damon Douglas
> Priority: P2
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> A kata devoted to core beam transforms patterns after
> [https://github.com/apache/beam/tree/master/learning/katas/java/Core%20Transforms]
> where the take away is an individual's ability to master the following using
> an Apache Beam pipeline using the Golang SDK.
>
> ||Transform||Pull Request||Status||
> |Map|[11564|https://github.com/apache/beam/pull/11564]|Closed|
> |GroupByKey|[11734|https://github.com/apache/beam/pull/11734]|Closed|
> |CoGroupByKey|[11803|https://github.com/apache/beam/pull/11803]|Open|
> |Combine| | |
> |Flatten|[11806|https://github.com/apache/beam/pull/11806]| |
> |Partition| | |
> |Side Input| | |
> |Side Output| | |
> |Branching| | |
> |Composite Transform| | |
> |DoFn Additional Parameters| | |
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
