alxp1982 commented on code in PR #23577: URL: https://github.com/apache/beam/pull/23577#discussion_r1000733006
########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/count/example/main.go: ########## @@ -0,0 +1,60 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// beam-playground: +// name: Count +// description: Count example. +// multifile: false +// context_line: 39 +// categories: +// - Quickstart +// complexity: BASIC +// tags: +// - hellobeam + +package main + +import ( + "context" + "github.com/apache/beam/sdks/v2/go/pkg/beam" + "github.com/apache/beam/sdks/v2/go/pkg/beam/log" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/beamx" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/debug" + "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/stats" +) + +func main() { + ctx := context.Background() + + p, s := beam.NewPipelineWithRoot() + + // List of elements Review Comment: // Create input PCollection ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/count/description.md: ########## @@ -0,0 +1,49 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Count + +`Count` provides many transformations for calculating the count of values in a `PCollection`, either globally or for each key. + +Counts the number of elements within each aggregation. The Count transform has two varieties: + +You can count the number of elements in ```PCollection``` with ```CountElms()```, it will return one element. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.CountElms(s, input) +} +``` + +You can use ```Count()``` to count how many elements are associated with a particular key, the result will be one output for each key. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Count(s, input) +} +``` + Review Comment: Lets' put '## Playground exercise' section title for runnable examples everywhere ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/count/description.md: ########## @@ -0,0 +1,49 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Count + +`Count` provides many transformations for calculating the count of values in a `PCollection`, either globally or for each key. + +Counts the number of elements within each aggregation. The Count transform has two varieties: + +You can count the number of elements in ```PCollection``` with ```CountElms()```, it will return one element. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.CountElms(s, input) +} +``` + +You can use ```Count()``` to count how many elements are associated with a particular key, the result will be one output for each key. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Count(s, input) +} +``` + +You can find the full code of this example in the playground window, which you can run and experiment with. Review Comment: Let's change playground description to: You can find an example of CountElms usage in the playground window. Please try it out to better understand how it works. Can you also try using Count to return how often each number appears in the input? Have you also noticed the order in which the collection items are displayed in the console? Why is that? You can also run the example several times to see if the output remains the same or changes. And Count transforms work with strings too! Can you change the example to count the number of words in a given sentence and how often each word occurs? ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/count/description.md: ########## @@ -0,0 +1,49 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Count + +`Count` provides many transformations for calculating the count of values in a `PCollection`, either globally or for each key. + Review Comment: Replace below lines with the following: ### CountElms You can count the total number of elements in ```PCollection``` with ```CountElms()```. It will return exactly one element. For example: ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/count/description.md: ########## @@ -0,0 +1,49 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Count + +`Count` provides many transformations for calculating the count of values in a `PCollection`, either globally or for each key. + +Counts the number of elements within each aggregation. The Count transform has two varieties: + +You can count the number of elements in ```PCollection``` with ```CountElms()```, it will return one element. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.CountElms(s, input) +} +``` + Review Comment: In the above example, the output will contain a single element - '10'. ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/count/description.md: ########## @@ -0,0 +1,49 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Count + +`Count` provides many transformations for calculating the count of values in a `PCollection`, either globally or for each key. + +Counts the number of elements within each aggregation. The Count transform has two varieties: + +You can count the number of elements in ```PCollection``` with ```CountElms()```, it will return one element. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.CountElms(s, input) +} +``` + +You can use ```Count()``` to count how many elements are associated with a particular key, the result will be one output for each key. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Count(s, input) +} +``` Review Comment: The output of the above example is '(1,3),(2,2),(3,3),(5,1),(7,2)' ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/max/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Max + +Max provides a variety of different transforms for computing the maximum values in a collection, either globally or for each key. + +You can find the global maximum value from the ```PCollection``` by using ```Max()``` + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Max(s, input) +} +``` + +You can use ```MaxPerKey()``` to calculate the maximum Integer associated with each unique key (which is of type String). + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + Review Comment: Let's put an example PCollection creation and what the output would be. E.g. input:= beam.ParDo(s, func(_ []byte, emit func(int, int)){ emit(1,1) emit(1,4) emit(2,6) emit(2,3) emit(2,-4) emit(3,23) }, beam.Impulse(s)) stuff. It would also be nice to explain this part as it isn't very intuitive. ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/max/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Max + +Max provides a variety of different transforms for computing the maximum values in a collection, either globally or for each key. + +You can find the global maximum value from the ```PCollection``` by using ```Max()``` + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Max(s, input) +} +``` + +You can use ```MaxPerKey()``` to calculate the maximum Integer associated with each unique key (which is of type String). + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.MaxPerKey(s, input) +} +``` + +You can find the full code of this example in the playground window, which you can run and experiment with. Review Comment: Let's do this for playground exercise section: ### Playground exercise. You can find the full code of the above example using 'Max' in the playground window, which you can run and experiment with. **Hint** You can use the following construction to create input `PCollection` that contains key-value pairs: ``` input:= beam.ParDo(s, func(_ []byte, emit func(int, int)){ emit("Jan",-1) emit("Jan",-3) emit("Feb",-4) emit("Feb",-6) emit("March",-3) emit("March",-4) emit("April",3) }, beam.Impulse(s)) ``` We encourage you to also try 'MaxPerKey' transform, for example, to find a max temperature for each month when the input contains a key-value list, where the key contains the month, and the value is a single temperature measurement. There may be multiple measurements for any given month. ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/max/example/main.go: ########## @@ -0,0 +1,60 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// beam-playground: +// name: Max +// description: Max example. +// multifile: false +// context_line: 38 +// categories: +// - Quickstart +// complexity: BASIC +// tags: +// - hellobeam + +package main + +import ( + "context" + "github.com/apache/beam/sdks/v2/go/pkg/beam" + "github.com/apache/beam/sdks/v2/go/pkg/beam/log" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/beamx" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/debug" + "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/stats" +) + +func main() { + ctx := context.Background() + + p, s := beam.NewPipelineWithRoot() + + // List of elements Review Comment: // Create input PCollection ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/max/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Max + +Max provides a variety of different transforms for computing the maximum values in a collection, either globally or for each key. + +You can find the global maximum value from the ```PCollection``` by using ```Max()``` + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Max(s, input) +} +``` + +You can use ```MaxPerKey()``` to calculate the maximum Integer associated with each unique key (which is of type String). + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.MaxPerKey(s, input) +} +``` Review Comment: The output of the above example is {1,4},{2,6},{3,23} ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/mean/example/main.go: ########## @@ -0,0 +1,60 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// beam-playground: +// name: Mean +// description: Mean example. +// multifile: false +// context_line: 38 +// categories: +// - Quickstart +// complexity: BASIC +// tags: +// - hellobeam + +package main + +import ( + "context" + "github.com/apache/beam/sdks/v2/go/pkg/beam" + "github.com/apache/beam/sdks/v2/go/pkg/beam/log" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/beamx" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/debug" + "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/stats" +) + +func main() { + ctx := context.Background() + + p, s := beam.NewPipelineWithRoot() + + // List of elements Review Comment: //Create PCollection ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/count/example/main.go: ########## @@ -0,0 +1,60 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// beam-playground: +// name: Count +// description: Count example. +// multifile: false +// context_line: 39 +// categories: +// - Quickstart +// complexity: BASIC +// tags: +// - hellobeam + +package main + +import ( + "context" + "github.com/apache/beam/sdks/v2/go/pkg/beam" + "github.com/apache/beam/sdks/v2/go/pkg/beam/log" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/beamx" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/debug" + "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/stats" +) + +func main() { + ctx := context.Background() + + p, s := beam.NewPipelineWithRoot() + + // List of elements + input := beam.Create(s, 1, 2, 31, 4, -5, 16, 2, 7, 8, 9, -5, 10, 11) + + // The applyTransform() converts [input] to [output] + output := applyTransform(s, input) + + debug.Printf(s, "PCollection count value: %v", output) Review Comment: debug.Printf(s, "Input has %v elements", output) ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/count/description.md: ########## @@ -0,0 +1,49 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Count + +`Count` provides many transformations for calculating the count of values in a `PCollection`, either globally or for each key. Review Comment: `Count` transforms return the number of elements within input `PCollection`, either globally or for each key. The `Count` has two varieties: ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/mean/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Mean + +You can use Mean transforms to compute the arithmetic mean of the elements in a collection or the mean of the values associated with each key in a collection of key-value pairs. + +```Mean()``` returns a transformation that returns a collection whose content is the average of the elements of the input collection. If there are no elements in the input collection, 0 is returned. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Mean(s, input) +} +``` Review Comment: The output of the above example is '5.5' ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/count/example/main.go: ########## @@ -0,0 +1,60 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// beam-playground: +// name: Count +// description: Count example. +// multifile: false +// context_line: 39 +// categories: +// - Quickstart +// complexity: BASIC +// tags: +// - hellobeam + +package main + +import ( + "context" + "github.com/apache/beam/sdks/v2/go/pkg/beam" + "github.com/apache/beam/sdks/v2/go/pkg/beam/log" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/beamx" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/debug" + "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/stats" +) + +func main() { + ctx := context.Background() + + p, s := beam.NewPipelineWithRoot() + + // List of elements + input := beam.Create(s, 1, 2, 31, 4, -5, 16, 2, 7, 8, 9, -5, 10, 11) + + // The applyTransform() converts [input] to [output] + output := applyTransform(s, input) + + debug.Printf(s, "PCollection count value: %v", output) + + err := beamx.Run(ctx, p) + + if err != nil { + log.Exitf(context.Background(), "Failed to execute job: %v", err) + } +} +// Return the count of numbers from `PCollection`. Review Comment: // Returns number of elements in `PCollection`. ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/max/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Max + +Max provides a variety of different transforms for computing the maximum values in a collection, either globally or for each key. + +You can find the global maximum value from the ```PCollection``` by using ```Max()``` + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + Review Comment: input := beam.Create(s, 1,2,3,4,5,6,7,8,9,10) output := applyTransform(s, input) ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/min/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Min + +Min transforms find the minimum values globally or for each key in the input collection. + +You can find the global minimum value from the ```PCollection``` by using ```Min()``` + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + Review Comment: ... input := beam.Create(s, 1,2,3,4,5,6,7,8,9,10) output := applyTransform(s, input) ... ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/count/description.md: ########## @@ -0,0 +1,49 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Count + +`Count` provides many transformations for calculating the count of values in a `PCollection`, either globally or for each key. + +Counts the number of elements within each aggregation. The Count transform has two varieties: + +You can count the number of elements in ```PCollection``` with ```CountElms()```, it will return one element. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + Review Comment: Insert: input := beam.Create(s, 1,2,3,4,5,6,7,8,9,10) output := applyTransform(s, input) ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/min/example/main.go: ########## @@ -0,0 +1,60 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// beam-playground: +// name: Min +// description: Min example. +// multifile: false +// context_line: 38 +// categories: +// - Quickstart +// complexity: BASIC +// tags: +// - hellobeam + +package main + +import ( + "context" + "github.com/apache/beam/sdks/v2/go/pkg/beam" + "github.com/apache/beam/sdks/v2/go/pkg/beam/log" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/beamx" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/debug" + "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/stats" +) + +func main() { + ctx := context.Background() + + p, s := beam.NewPipelineWithRoot() + + // List of elements Review Comment: //Create input PCollection ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/min/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Min + +Min transforms find the minimum values globally or for each key in the input collection. + +You can find the global minimum value from the ```PCollection``` by using ```Min()``` + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Min(s, input) +} +``` + +You can use ```MinPerKey()``` to calculate the minimum Integer associated with each unique key (which is of type String). + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.MinPerKey(s, input) +} +``` + Review Comment: Let's do this for the playground exercise section: ### Playground exercise. You can find the full code of the above example using 'Min' in the playground window, which you can run and experiment with. **Hint** You can use the following construction to create input `PCollection` that contains key-value pairs: ``` input:= beam.ParDo(s, func(_ []byte, emit func(int, int)){ emit("Jan",-1) emit("Jan",-3) emit("Feb",-4) emit("Feb",-6) emit("March",-3) emit("March",-4) emit("April",3) }, beam.Impulse(s)) ``` We encourage you to also try the 'MinPerKey' transform, for example, to find a minimum temperature for each month when the input contains a key-value list, where the key includes the month, and the value is a single temperature measurement. There may be multiple measurements for any given month. ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/sum/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Sum + +You can use Sum transforms to compute the sum of the elements in a collection or the sum of the values associated with each key in a collection of key-value pairs. + Review Comment: ## Sum ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/max/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Max + +Max provides a variety of different transforms for computing the maximum values in a collection, either globally or for each key. + Review Comment: ### Max ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/max/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Max + +Max provides a variety of different transforms for computing the maximum values in a collection, either globally or for each key. + +You can find the global maximum value from the ```PCollection``` by using ```Max()``` + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Max(s, input) +} +``` + Review Comment: ### MaxPerKey ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/sum/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Sum + +You can use Sum transforms to compute the sum of the elements in a collection or the sum of the values associated with each key in a collection of key-value pairs. + +You can use ```Sum()``` to sum the elements of a ```PCollection```. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Sum(s, input) +} +``` + +You can use ```SumPerKey()```to calculate the sum Integer associated with each unique key. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.SumPerKey(s, input) +} +``` Review Comment: The output of the above example is (1,3),(2,5),(3,23) ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/max/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Max + +Max provides a variety of different transforms for computing the maximum values in a collection, either globally or for each key. + +You can find the global maximum value from the ```PCollection``` by using ```Max()``` + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Max(s, input) +} +``` Review Comment: The output of the above example is '10' ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/count/description.md: ########## @@ -0,0 +1,49 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Count + +`Count` provides many transformations for calculating the count of values in a `PCollection`, either globally or for each key. + +Counts the number of elements within each aggregation. The Count transform has two varieties: + +You can count the number of elements in ```PCollection``` with ```CountElms()```, it will return one element. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.CountElms(s, input) +} +``` + +You can use ```Count()``` to count how many elements are associated with a particular key, the result will be one output for each key. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + Review Comment: Insert: input := beam.Create(s, 1,1,2,2,3,3,3,1,5,7,7) output := applyTransform(s, input) ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/sum/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Sum + +You can use Sum transforms to compute the sum of the elements in a collection or the sum of the values associated with each key in a collection of key-value pairs. + +You can use ```Sum()``` to sum the elements of a ```PCollection```. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Sum(s, input) +} +``` + +You can use ```SumPerKey()```to calculate the sum Integer associated with each unique key. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + Review Comment: ... input:= beam.ParDo(s, func(_ []byte, emit func(int, int)){ emit(1,1) emit(1,4) emit(2,6) emit(2,4) emit(2,-4) emit(3,23) }, beam.Impulse(s)) output := applyTransform(s, input) ... ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/sum/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Sum + +You can use Sum transforms to compute the sum of the elements in a collection or the sum of the values associated with each key in a collection of key-value pairs. + +You can use ```Sum()``` to sum the elements of a ```PCollection```. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Sum(s, input) +} +``` Review Comment: The output of the above example is '55' ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/sum/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Sum + +You can use Sum transforms to compute the sum of the elements in a collection or the sum of the values associated with each key in a collection of key-value pairs. + +You can use ```Sum()``` to sum the elements of a ```PCollection```. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Sum(s, input) +} +``` + +You can use ```SumPerKey()```to calculate the sum Integer associated with each unique key. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.SumPerKey(s, input) +} +``` + +You can find the full code of this example in the playground window, which you can run and experiment with. Review Comment: Let's do this for the playground exercise section: ### Playground exercise. You can find the full code of the above example using 'Sum' in the playground window, which you can run and experiment with. **Hint** You can use the following construction to create input `PCollection` that contains key-value pairs: ``` input:= beam.ParDo(s, func(_ []byte, emit func(int, int)){ emit("electronics",25) emit("clothing",10) emit("food",45) emit("clothing",14) emit("electronics",16) emit("books",40) emit("food",16) }, beam.Impulse(s)) ``` We encourage you to also try the 'SumPerKey' transform, for example, to find a sum of shipment weights for each shipment category when the input contains a key-value list, where the key represents the shipment category, and the value is a single shipment weight measurement. There may be multiple measurements for any given shipment category. ########## learning/tour-of-beam/learning-content/go/common-transforms/filter/description.md: ########## @@ -0,0 +1,36 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Filter + +### Using Filter + +PCollection datasets can be filtered using the Filter transform. You can create a filter by supplying a predicate and, when applied, filtering out all the elements of PCollection that don’t satisfy the predicate. + +``` +import ( + "github.com/apache/fbeam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/filter" +) + Review Comment: ... input := beam.Create(s, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10) output := applyTransform(s, input) ... ########## learning/tour-of-beam/learning-content/go/common-transforms/filter/description.md: ########## @@ -0,0 +1,36 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Filter + +### Using Filter + +PCollection datasets can be filtered using the Filter transform. You can create a filter by supplying a predicate and, when applied, filtering out all the elements of PCollection that don’t satisfy the predicate. + +``` +import ( + "github.com/apache/fbeam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/filter" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return filter.Exclude(s, input, func(element int) bool { + return element % 2 == 1 + }) +} +``` Review Comment: The output of the above example will be (1,3,5,7,9) ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/sum/example/main.go: ########## @@ -0,0 +1,60 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// beam-playground: +// name: Sum +// description: Sum example. +// multifile: false +// context_line: 33 +// categories: +// - Quickstart +// complexity: BASIC +// tags: +// - hellobeam + +package main + +import ( + "context" + "github.com/apache/beam/sdks/v2/go/pkg/beam" + "github.com/apache/beam/sdks/v2/go/pkg/beam/log" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/beamx" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/debug" + "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/stats" +) + +func main() { + ctx := context.Background() + + p, s := beam.NewPipelineWithRoot() + + // List of elements Review Comment: //Create input PCollection ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/mean/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Mean + +You can use Mean transforms to compute the arithmetic mean of the elements in a collection or the mean of the values associated with each key in a collection of key-value pairs. + +```Mean()``` returns a transformation that returns a collection whose content is the average of the elements of the input collection. If there are no elements in the input collection, 0 is returned. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Mean(s, input) +} +``` + Review Comment: ### MeanPerKey ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/mean/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Mean + +You can use Mean transforms to compute the arithmetic mean of the elements in a collection or the mean of the values associated with each key in a collection of key-value pairs. + Review Comment: ### Mean ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/mean/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Mean + +You can use Mean transforms to compute the arithmetic mean of the elements in a collection or the mean of the values associated with each key in a collection of key-value pairs. + +```Mean()``` returns a transformation that returns a collection whose content is the average of the elements of the input collection. If there are no elements in the input collection, 0 is returned. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + Review Comment: ... input := beam.Create(s, 1,2,3,4,5,6,7,8,9,10) output := applyTransform(s, input) ... ########## learning/tour-of-beam/learning-content/go/common-transforms/filter/description.md: ########## @@ -0,0 +1,36 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Filter + +### Using Filter + +PCollection datasets can be filtered using the Filter transform. You can create a filter by supplying a predicate and, when applied, filtering out all the elements of PCollection that don’t satisfy the predicate. + +``` +import ( + "github.com/apache/fbeam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/filter" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return filter.Exclude(s, input, func(element int) bool { + return element % 2 == 1 + }) +} +``` + Review Comment: ### Playground exercise. You can find the complete code of the above example using 'Filter' in the playground window, which you can run and experiment with. Filter transform can be used with both text and numerical collection. For example, let's try filtering the input collection that contains words so that only words that start with the letter 'a' are returned. You can also chain several filter transforms to form more complex filtering based on several simple filters or implement more complex filtering logic within a single filter transform. For example, try both approaches to filter the same list of words such that only ones that start with a letter 'a' (regardless of the case) **and** containing more than three symbols are returned. **Hint** You can use the following code snippet to create an input `PCollection`: ``` str:= `To be, or not to be: that is the question: Whether 'tis nobler in the mind to suffer The slings and arrows of outrageous fortune, Or to take arms against a sea of troubles, And by opposing end them. To die: to sleep;` input := beam.Create(s,strings.Split(" ",str)) ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/mean/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Mean + +You can use Mean transforms to compute the arithmetic mean of the elements in a collection or the mean of the values associated with each key in a collection of key-value pairs. + +```Mean()``` returns a transformation that returns a collection whose content is the average of the elements of the input collection. If there are no elements in the input collection, 0 is returned. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Mean(s, input) +} +``` + +You can use ```MeanPerKey()``` to calculate the mean of the elements associated with each unique key. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.MeanPerKey(s, input) +} +``` + Review Comment: Let's do this for playground exercise section: ### Playground exercise. You can find the full code of the above example using 'Mean' in the playground window, which you can run and experiment with. **Hint** You can use the following construction to create input `PCollection` that contains key-value pairs: ``` input:= beam.ParDo(s, func(_ []byte, emit func(int, int)){ emit("Jan",-1) emit("Jan",-3) emit("Feb",-4) emit("Feb",-6) emit("March",-3) emit("March",-4) emit("April",3) }, beam.Impulse(s)) ``` We encourage you to also try the 'MeanPerKey' transform, for example, to find a mean temperature for each month when the input contains a key-value list, where the key includes the month, and the value is a single temperature measurement. There may be multiple measurements for any given month. ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/min/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Min + +Min transforms find the minimum values globally or for each key in the input collection. + Review Comment: ### Min ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/mean/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Mean + +You can use Mean transforms to compute the arithmetic mean of the elements in a collection or the mean of the values associated with each key in a collection of key-value pairs. + +```Mean()``` returns a transformation that returns a collection whose content is the average of the elements of the input collection. If there are no elements in the input collection, 0 is returned. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Mean(s, input) +} +``` + +You can use ```MeanPerKey()``` to calculate the mean of the elements associated with each unique key. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + Review Comment: ... input:= beam.ParDo(s, func(_ []byte, emit func(int, int)){ emit(1,1) emit(1,4) emit(2,6) emit(2,4) emit(2,-4) emit(3,23) }, beam.Impulse(s)) output := applyTransform(s, input) ... ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/mean/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Mean + +You can use Mean transforms to compute the arithmetic mean of the elements in a collection or the mean of the values associated with each key in a collection of key-value pairs. + +```Mean()``` returns a transformation that returns a collection whose content is the average of the elements of the input collection. If there are no elements in the input collection, 0 is returned. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Mean(s, input) +} +``` + +You can use ```MeanPerKey()``` to calculate the mean of the elements associated with each unique key. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.MeanPerKey(s, input) +} +``` Review Comment: Output of the above example is {1,2.5},{2,2},{3,23} ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/min/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Min + +Min transforms find the minimum values globally or for each key in the input collection. + +You can find the global minimum value from the ```PCollection``` by using ```Min()``` + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Min(s, input) +} +``` + Review Comment: ### MinPerKey ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/min/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Min + +Min transforms find the minimum values globally or for each key in the input collection. + +You can find the global minimum value from the ```PCollection``` by using ```Min()``` + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Min(s, input) +} +``` Review Comment: The output of the above example is '1' ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/min/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Min + +Min transforms find the minimum values globally or for each key in the input collection. + +You can find the global minimum value from the ```PCollection``` by using ```Min()``` + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Min(s, input) +} +``` + +You can use ```MinPerKey()``` to calculate the minimum Integer associated with each unique key (which is of type String). + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + Review Comment: ... input:= beam.ParDo(s, func(_ []byte, emit func(int, int)){ emit(1,1) emit(1,4) emit(2,6) emit(2,4) emit(2,-4) emit(3,23) }, beam.Impulse(s)) output := applyTransform(s, input) ... ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/min/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Min + +Min transforms find the minimum values globally or for each key in the input collection. + +You can find the global minimum value from the ```PCollection``` by using ```Min()``` + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Min(s, input) +} +``` + +You can use ```MinPerKey()``` to calculate the minimum Integer associated with each unique key (which is of type String). + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.MinPerKey(s, input) +} +``` Review Comment: The output of the above example is (1,1),(2,-4),(3,23) ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/sum/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Sum + +You can use Sum transforms to compute the sum of the elements in a collection or the sum of the values associated with each key in a collection of key-value pairs. + +You can use ```Sum()``` to sum the elements of a ```PCollection```. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + +func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection { + return stats.Sum(s, input) +} +``` + Review Comment: ## SumPerKey ########## learning/tour-of-beam/learning-content/go/common-transforms/aggregation/sum/description.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Sum + +You can use Sum transforms to compute the sum of the elements in a collection or the sum of the values associated with each key in a collection of key-value pairs. + +You can use ```Sum()``` to sum the elements of a ```PCollection```. + +``` +import ( + "github.com/apache/beam/sdks/go/pkg/beam" + "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats" +) + Review Comment: ... input := beam.Create(s, 1,2,3,4,5,6,7,8,9,10) output := applyTransform(s, input) ... ########## learning/tour-of-beam/learning-content/go/common-transforms/filter/example/main.go: ########## @@ -0,0 +1,65 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// beam-playground: +// name: Filter +// description: Filter example. +// multifile: false +// context_line: 38 +// categories: +// - Quickstart +// complexity: BASIC +// tags: +// - hellobeam + +package main + +import ( + "context" + "github.com/apache/beam/sdks/v2/go/pkg/beam" + "github.com/apache/beam/sdks/v2/go/pkg/beam/log" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/beamx" + "github.com/apache/beam/sdks/v2/go/pkg/beam/x/debug" + "github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/filter" +) + +func main() { + ctx := context.Background() + + p, s := beam.NewPipelineWithRoot() + + // List of elements Review Comment: //Create input PCollection -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
