This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 97f27c9cf518f34a7e9e573612ff495b29eb06c6 Author: Mergebot <[email protected]> AuthorDate: Tue May 15 21:10:42 2018 -0700 Prepare repository for deployment. --- content/documentation/dsls/sql/index.html | 11 + content/documentation/index.html | 3 +- .../sdks/feature-comparison/index.html | 11 + content/documentation/sdks/{java => go}/index.html | 51 ++-- .../documentation/sdks/java-extensions/index.html | 11 + .../documentation/sdks/java-thirdparty/index.html | 11 + content/documentation/sdks/java/index.html | 11 + content/documentation/sdks/java/nexmark/index.html | 11 + .../documentation/sdks/python-custom-io/index.html | 11 + .../sdks/python-pipeline-dependencies/index.html | 11 + .../documentation/sdks/python-streaming/index.html | 11 + .../sdks/python-type-safety/index.html | 11 + content/documentation/sdks/python/index.html | 11 + content/get-started/beam-overview/index.html | 4 +- content/get-started/downloads/index.html | 1 + content/get-started/index.html | 3 +- .../get-started/mobile-gaming-example/index.html | 6 + content/get-started/{ => quickstart-go}/index.html | 80 ++++-- content/get-started/quickstart-java/index.html | 1 + content/get-started/quickstart-py/index.html | 1 + content/get-started/wordcount-example/index.html | 317 ++++++++++++++++++++- content/images/logos/sdks/go.png | Bin 0 -> 15475 bytes content/index.html | 2 + 23 files changed, 526 insertions(+), 64 deletions(-) diff --git a/content/documentation/dsls/sql/index.html b/content/documentation/dsls/sql/index.html index ab2348b..d85f387 100644 --- a/content/documentation/dsls/sql/index.html +++ b/content/documentation/dsls/sql/index.html @@ -120,6 +120,17 @@ </li> <li> + <span class="section-nav-list-title">Go SDK</span> + <ul class="section-nav-list"> + <li><a href="/documentation/sdks/go/">Go SDK overview</a></li> + <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam"; target="_blank">Go SDK API reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + </ul> +</li> + +<li> <span class="section-nav-list-title">DSLs</span> <ul class="section-nav-list"> <li><a href="/documentation/dsls/sql/">SQL</a></li> diff --git a/content/documentation/index.html b/content/documentation/index.html index 5c78629..71d463f 100644 --- a/content/documentation/index.html +++ b/content/documentation/index.html @@ -236,6 +236,7 @@ <ul> <li><a href="/documentation/sdks/java/">Java SDK</a></li> <li><a href="/documentation/sdks/python/">Python SDK</a></li> + <li><a href="/documentation/sdks/go/">Go SDK</a></li> </ul> <h2 id="runners">Runners</h2> @@ -257,7 +258,7 @@ <p>Beam is designed to enable pipelines to be portable across different runners. However, given every runner has different capabilities, they also have different abilities to implement the core concepts in the Beam model. The <a href="/documentation/runners/capability-matrix/">Capability Matrix</a> provides a detailed comparison of runner functionality.</p> -<p>Once you have chosen which runner to use, see that runner’s page for more information about any initial runner-specific setup as well as any required or optional <code class="highlighter-rouge">PipelineOptions</code> for configuring it’s execution. You may also want to refer back to the Quickstart for <a href="/get-started/quickstart-java">Java</a> or <a href="/get-started/quickstart-py">Python</a> for instructions on executing the sample WordCount pipeline.</p> +<p>Once you have chosen which runner to use, see that runner’s page for more information about any initial runner-specific setup as well as any required or optional <code class="highlighter-rouge">PipelineOptions</code> for configuring it’s execution. You may also want to refer back to the Quickstart for <a href="/get-started/quickstart-java">Java</a>, <a href="/get-started/quickstart-py">Python</a> or <a href="/get-started/quickstart-go">Go</a> for instructions on executing the sample W [...] </div> </div> diff --git a/content/documentation/sdks/feature-comparison/index.html b/content/documentation/sdks/feature-comparison/index.html index 57f3a81..c3ae7f7 100644 --- a/content/documentation/sdks/feature-comparison/index.html +++ b/content/documentation/sdks/feature-comparison/index.html @@ -120,6 +120,17 @@ </li> <li> + <span class="section-nav-list-title">Go SDK</span> + <ul class="section-nav-list"> + <li><a href="/documentation/sdks/go/">Go SDK overview</a></li> + <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam"; target="_blank">Go SDK API reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + </ul> +</li> + +<li> <span class="section-nav-list-title">DSLs</span> <ul class="section-nav-list"> <li><a href="/documentation/dsls/sql/">SQL</a></li> diff --git a/content/documentation/sdks/java/index.html b/content/documentation/sdks/go/index.html similarity index 84% copy from content/documentation/sdks/java/index.html copy to content/documentation/sdks/go/index.html index f9d450e..4f87793 100644 --- a/content/documentation/sdks/java/index.html +++ b/content/documentation/sdks/go/index.html @@ -4,7 +4,7 @@ <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> - <title>Beam Java SDK</title> + <title>Beam Go SDK</title> <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...] "> <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400"; rel="stylesheet"> @@ -15,7 +15,7 @@ <script src="/js/fix-menu.js"></script> <script src="/js/section-nav.js"></script> <script src="/js/page-nav.js"></script> - <link rel="canonical" href="https://beam.apache.org/documentation/sdks/java/"; data-proofer-ignore> + <link rel="canonical" href="https://beam.apache.org/documentation/sdks/go/"; data-proofer-ignore> <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico"> <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml";> <script> @@ -120,6 +120,17 @@ </li> <li> + <span class="section-nav-list-title">Go SDK</span> + <ul class="section-nav-list"> + <li><a href="/documentation/sdks/go/">Go SDK overview</a></li> + <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam"; target="_blank">Go SDK API reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + </ul> +</li> + +<li> <span class="section-nav-list-title">DSLs</span> <ul class="section-nav-list"> <li><a href="/documentation/dsls/sql/">SQL</a></li> @@ -134,44 +145,28 @@ <ul class="nav"> - <li><a href="#get-started-with-the-java-sdk">Get Started with the Java SDK</a></li> - <li><a href="#supported-features">Supported Features</a></li> - <li><a href="#pipeline-io">Pipeline I/O</a></li> - <li><a href="#extensions">Extensions</a></li> + <li><a href="#get-started-with-the-go-sdk">Get Started with the Go SDK</a></li> + <li><a href="#status">Status</a></li> </ul> </nav> <div class="body__contained body__section-nav"> - <h1 id="apache-beam-java-sdk">Apache Beam Java SDK</h1> - -<p>The Java SDK for Apache Beam provides a simple, powerful API for building both batch and streaming parallel data processing pipelines in Java.</p> - -<h2 id="get-started-with-the-java-sdk">Get Started with the Java SDK</h2> - -<p>Get started with the <a href="/documentation/programming-guide/">Beam Programming Model</a> to learn the basic concepts that apply to all SDKs in Beam.</p> + <h1 id="apache-beam-go-sdk">Apache Beam Go SDK</h1> -<p>See the <a href="/documentation/sdks/javadoc/">Java API Reference</a> for more information on individual APIs.</p> +<p>The Go SDK for Apache Beam provides a simple, powerful API for building both batch and streaming parallel data processing pipelines. It is based on the following +<a href="https://s.apache.org/beam-go-sdk-design-rfc";>design</a>.</p> -<h2 id="supported-features">Supported Features</h2> +<h2 id="get-started-with-the-go-sdk">Get Started with the Go SDK</h2> -<p>The Java SDK supports all features currently supported by the Beam model.</p> +<p>Get started with the <a href="/get-started/quickstart-go">Beam Go SDK quickstart</a> to set up your development environment and run an example pipeline. Then, read through the <a href="/documentation/programming-guide">Beam programming guide</a> to learn the basic concepts that apply to all SDKs in Beam.</p> -<h2 id="pipeline-io">Pipeline I/O</h2> -<p>See the <a href="/documentation/io/built-in/">Beam-provided I/O Transforms</a> page for a list of the currently available I/O transforms.</p> +<p>See the <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam";>godoc</a> for more detailed information.</p> -<h2 id="extensions">Extensions</h2> - -<p>The Java SDK has the following extensions:</p> - -<ul> - <li><a href="/documentation/sdks/java-extensions/#join-library">join-library</a> provides inner join, outer left join, and outer right join functions.</li> - <li><a href="/documentation/sdks/java-extensions/#sorter">sorter</a> is an efficient and scalable sorter for large iterables.</li> - <li><a href="/documentation/sdks/java/nexmark">Nexmark</a> is a benchmark suite that runs in batch and streaming modes.</li> -</ul> +<h2 id="status">Status</h2> -<p>In addition several <a href="/documentation/sdks/java-thirdparty/">3rd party Java libraries</a> exist.</p> +<p>The Go SDK is currently experimental, does not yet offer any compatibility guarantees and is not recommended for production usage. It supports most features, but not all.</p> </div> </div> diff --git a/content/documentation/sdks/java-extensions/index.html b/content/documentation/sdks/java-extensions/index.html index fc24efe..ce2ca9c 100644 --- a/content/documentation/sdks/java-extensions/index.html +++ b/content/documentation/sdks/java-extensions/index.html @@ -120,6 +120,17 @@ </li> <li> + <span class="section-nav-list-title">Go SDK</span> + <ul class="section-nav-list"> + <li><a href="/documentation/sdks/go/">Go SDK overview</a></li> + <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam"; target="_blank">Go SDK API reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + </ul> +</li> + +<li> <span class="section-nav-list-title">DSLs</span> <ul class="section-nav-list"> <li><a href="/documentation/dsls/sql/">SQL</a></li> diff --git a/content/documentation/sdks/java-thirdparty/index.html b/content/documentation/sdks/java-thirdparty/index.html index 26c94fe..12da390 100644 --- a/content/documentation/sdks/java-thirdparty/index.html +++ b/content/documentation/sdks/java-thirdparty/index.html @@ -120,6 +120,17 @@ </li> <li> + <span class="section-nav-list-title">Go SDK</span> + <ul class="section-nav-list"> + <li><a href="/documentation/sdks/go/">Go SDK overview</a></li> + <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam"; target="_blank">Go SDK API reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + </ul> +</li> + +<li> <span class="section-nav-list-title">DSLs</span> <ul class="section-nav-list"> <li><a href="/documentation/dsls/sql/">SQL</a></li> diff --git a/content/documentation/sdks/java/index.html b/content/documentation/sdks/java/index.html index f9d450e..0c1ee4d 100644 --- a/content/documentation/sdks/java/index.html +++ b/content/documentation/sdks/java/index.html @@ -120,6 +120,17 @@ </li> <li> + <span class="section-nav-list-title">Go SDK</span> + <ul class="section-nav-list"> + <li><a href="/documentation/sdks/go/">Go SDK overview</a></li> + <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam"; target="_blank">Go SDK API reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + </ul> +</li> + +<li> <span class="section-nav-list-title">DSLs</span> <ul class="section-nav-list"> <li><a href="/documentation/dsls/sql/">SQL</a></li> diff --git a/content/documentation/sdks/java/nexmark/index.html b/content/documentation/sdks/java/nexmark/index.html index 6ad7370..38e9df1 100644 --- a/content/documentation/sdks/java/nexmark/index.html +++ b/content/documentation/sdks/java/nexmark/index.html @@ -120,6 +120,17 @@ </li> <li> + <span class="section-nav-list-title">Go SDK</span> + <ul class="section-nav-list"> + <li><a href="/documentation/sdks/go/">Go SDK overview</a></li> + <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam"; target="_blank">Go SDK API reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + </ul> +</li> + +<li> <span class="section-nav-list-title">DSLs</span> <ul class="section-nav-list"> <li><a href="/documentation/dsls/sql/">SQL</a></li> diff --git a/content/documentation/sdks/python-custom-io/index.html b/content/documentation/sdks/python-custom-io/index.html index 6ab083a..dae6ee7 100644 --- a/content/documentation/sdks/python-custom-io/index.html +++ b/content/documentation/sdks/python-custom-io/index.html @@ -120,6 +120,17 @@ </li> <li> + <span class="section-nav-list-title">Go SDK</span> + <ul class="section-nav-list"> + <li><a href="/documentation/sdks/go/">Go SDK overview</a></li> + <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam"; target="_blank">Go SDK API reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + </ul> +</li> + +<li> <span class="section-nav-list-title">DSLs</span> <ul class="section-nav-list"> <li><a href="/documentation/dsls/sql/">SQL</a></li> diff --git a/content/documentation/sdks/python-pipeline-dependencies/index.html b/content/documentation/sdks/python-pipeline-dependencies/index.html index 7e48c27..a9f7367 100644 --- a/content/documentation/sdks/python-pipeline-dependencies/index.html +++ b/content/documentation/sdks/python-pipeline-dependencies/index.html @@ -120,6 +120,17 @@ </li> <li> + <span class="section-nav-list-title">Go SDK</span> + <ul class="section-nav-list"> + <li><a href="/documentation/sdks/go/">Go SDK overview</a></li> + <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam"; target="_blank">Go SDK API reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + </ul> +</li> + +<li> <span class="section-nav-list-title">DSLs</span> <ul class="section-nav-list"> <li><a href="/documentation/dsls/sql/">SQL</a></li> diff --git a/content/documentation/sdks/python-streaming/index.html b/content/documentation/sdks/python-streaming/index.html index 1275208..aab1124 100644 --- a/content/documentation/sdks/python-streaming/index.html +++ b/content/documentation/sdks/python-streaming/index.html @@ -120,6 +120,17 @@ </li> <li> + <span class="section-nav-list-title">Go SDK</span> + <ul class="section-nav-list"> + <li><a href="/documentation/sdks/go/">Go SDK overview</a></li> + <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam"; target="_blank">Go SDK API reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + </ul> +</li> + +<li> <span class="section-nav-list-title">DSLs</span> <ul class="section-nav-list"> <li><a href="/documentation/dsls/sql/">SQL</a></li> diff --git a/content/documentation/sdks/python-type-safety/index.html b/content/documentation/sdks/python-type-safety/index.html index 7e98d4a..7358b84 100644 --- a/content/documentation/sdks/python-type-safety/index.html +++ b/content/documentation/sdks/python-type-safety/index.html @@ -120,6 +120,17 @@ </li> <li> + <span class="section-nav-list-title">Go SDK</span> + <ul class="section-nav-list"> + <li><a href="/documentation/sdks/go/">Go SDK overview</a></li> + <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam"; target="_blank">Go SDK API reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + </ul> +</li> + +<li> <span class="section-nav-list-title">DSLs</span> <ul class="section-nav-list"> <li><a href="/documentation/dsls/sql/">SQL</a></li> diff --git a/content/documentation/sdks/python/index.html b/content/documentation/sdks/python/index.html index 55b32e1..4d78a7d 100644 --- a/content/documentation/sdks/python/index.html +++ b/content/documentation/sdks/python/index.html @@ -120,6 +120,17 @@ </li> <li> + <span class="section-nav-list-title">Go SDK</span> + <ul class="section-nav-list"> + <li><a href="/documentation/sdks/go/">Go SDK overview</a></li> + <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam"; target="_blank">Go SDK API reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + </ul> +</li> + +<li> <span class="section-nav-list-title">DSLs</span> <ul class="section-nav-list"> <li><a href="/documentation/dsls/sql/">SQL</a></li> diff --git a/content/get-started/beam-overview/index.html b/content/get-started/beam-overview/index.html index 833ca28..ff410ff 100644 --- a/content/get-started/beam-overview/index.html +++ b/content/get-started/beam-overview/index.html @@ -96,6 +96,7 @@ <ul class="section-nav-list"> <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li> <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li> + <li><a href="/get-started/quickstart-go/">Quickstart - Go</a></li> </ul> </li> <li> @@ -141,6 +142,7 @@ <ul> <li>Java <img src="/images/logos/sdks/java.png" alt="Java logo" /></li> <li>Python <img src="/images/logos/sdks/python.png" alt="Python logo" /></li> + <li>Go <img src="/images/logos/sdks/go.png" alt="Go logo" height="45px" /></li> </ul> <p>A Scala <img src="/images/logos/sdks/scala.png" alt="Scala logo" height="45px" /> interface is also available as <a href="https://github.com/spotify/scio";>Scio</a>.</p> @@ -167,7 +169,7 @@ <ol> <li> - <p>Follow the Quickstart for the <a href="/get-started/quickstart-java">Java SDK</a> or the <a href="/get-started/quickstart-py">Python SDK</a>.</p> + <p>Follow the Quickstart for the <a href="/get-started/quickstart-java">Java SDK</a>, the <a href="/get-started/quickstart-py">Python SDK</a> or the <a href="/get-started/quickstart-go">Go SDK</a>.</p> </li> <li> <p>See the <a href="/get-started/wordcount-example">WordCount Examples Walkthrough</a> for examples that introduce various features of the SDKs.</p> diff --git a/content/get-started/downloads/index.html b/content/get-started/downloads/index.html index e250c35..cfb7dd8 100644 --- a/content/get-started/downloads/index.html +++ b/content/get-started/downloads/index.html @@ -96,6 +96,7 @@ <ul class="section-nav-list"> <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li> <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li> + <li><a href="/get-started/quickstart-go/">Quickstart - Go</a></li> </ul> </li> <li> diff --git a/content/get-started/index.html b/content/get-started/index.html index e6c2827..d0e7d2e 100644 --- a/content/get-started/index.html +++ b/content/get-started/index.html @@ -96,6 +96,7 @@ <ul class="section-nav-list"> <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li> <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li> + <li><a href="/get-started/quickstart-go/">Quickstart - Go</a></li> </ul> </li> <li> @@ -127,7 +128,7 @@ <p>Learn about the Beam model, the currently available Beam SDKs and Runners, and Beam’s native I/O connectors.</p> -<h4 id="quickstart-for-java-or-python">Quickstart for <a href="/get-started/quickstart-java">Java</a> or <a href="/get-started/quickstart-py">Python</a></h4> +<h4 id="quickstart-for-java-python-or-go">Quickstart for <a href="/get-started/quickstart-java">Java</a>, <a href="/get-started/quickstart-py">Python</a> or <a href="/get-started/quickstart-go">Go</a></h4> <p>Learn how to set up a Beam project and run a simple example Beam pipeline on your local machine.</p> diff --git a/content/get-started/mobile-gaming-example/index.html b/content/get-started/mobile-gaming-example/index.html index 54281d4..16d2665 100644 --- a/content/get-started/mobile-gaming-example/index.html +++ b/content/get-started/mobile-gaming-example/index.html @@ -96,6 +96,7 @@ <ul class="section-nav-list"> <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li> <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li> + <li><a href="/get-started/quickstart-go/">Quickstart - Go</a></li> </ul> </li> <li> @@ -198,6 +199,11 @@ <p><strong>Note</strong>: These examples assume some familiarity with the Beam programming model. If you haven’t already, we recommend familiarizing yourself with the programming model documentation and running a basic example pipeline before continuing.</p> </blockquote> +<blockquote> + <p><strong>Note</strong>: MobileGaming is not yet available for the Go SDK. There is an open issue for this +(<a href="https://issues.apache.org/jira/browse/BEAM-4293";>BEAM-4293</a>).</p> +</blockquote> + <p>Every time a user plays an instance of our hypothetical mobile game, they generate a data event. Each data event consists of the following information:</p> <ul> diff --git a/content/get-started/index.html b/content/get-started/quickstart-go/index.html similarity index 69% copy from content/get-started/index.html copy to content/get-started/quickstart-go/index.html index e6c2827..59232c3 100644 --- a/content/get-started/index.html +++ b/content/get-started/quickstart-go/index.html @@ -4,7 +4,7 @@ <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> - <title>Use Beam</title> + <title>Beam Quickstart for Go</title> <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...] "> <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400"; rel="stylesheet"> @@ -15,7 +15,7 @@ <script src="/js/fix-menu.js"></script> <script src="/js/section-nav.js"></script> <script src="/js/page-nav.js"></script> - <link rel="canonical" href="https://beam.apache.org/get-started/"; data-proofer-ignore> + <link rel="canonical" href="https://beam.apache.org/get-started/quickstart-go/"; data-proofer-ignore> <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico"> <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml";> <script> @@ -96,6 +96,7 @@ <ul class="section-nav-list"> <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li> <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li> + <li><a href="/get-started/quickstart-go/">Quickstart - Go</a></li> </ul> </li> <li> @@ -115,38 +116,81 @@ <nav class="page-nav clearfix" data-offset-top="90" data-offset-bottom="500"> +<ul class="nav"> + <li><a href="#set-up-your-environment">Set up your environment</a></li> + <li><a href="#get-the-sdk-and-the-examples">Get the SDK and the examples</a></li> + <li><a href="#run-wordcount">Run wordcount</a></li> + <li><a href="#next-steps">Next Steps</a></li> +</ul> </nav> <div class="body__contained body__section-nav"> - <h1 id="get-started-with-apache-beam">Get Started with Apache Beam</h1> -<p>Learn to use Beam to create data processing pipelines that run on supported processing back-ends:</p> + <h1 id="apache-beam-go-sdk-quickstart">Apache Beam Go SDK Quickstart</h1> -<h4 id="beam-overview"><a href="/get-started/beam-overview">Beam Overview</a></h4> +<p>This Quickstart will walk you through executing your first Beam pipeline to run <a href="/get-started/wordcount-example">WordCount</a>, written using Beam’s <a href="/documentation/sdks/go">Go SDK</a>, on a <a href="/documentation#runners">runner</a> of your choice.</p> -<p>Learn about the Beam model, the currently available Beam SDKs and Runners, and Beam’s native I/O connectors.</p> +<ul id="markdown-toc"> + <li><a href="#set-up-your-environment" id="markdown-toc-set-up-your-environment">Set up your environment</a></li> + <li><a href="#get-the-sdk-and-the-examples" id="markdown-toc-get-the-sdk-and-the-examples">Get the SDK and the examples</a></li> + <li><a href="#run-wordcount" id="markdown-toc-run-wordcount">Run wordcount</a></li> + <li><a href="#next-steps" id="markdown-toc-next-steps">Next Steps</a></li> +</ul> -<h4 id="quickstart-for-java-or-python">Quickstart for <a href="/get-started/quickstart-java">Java</a> or <a href="/get-started/quickstart-py">Python</a></h4> +<h2 id="set-up-your-environment">Set up your environment</h2> -<p>Learn how to set up a Beam project and run a simple example Beam pipeline on your local machine.</p> +<p>The Beam SDK for Go requires <code class="highlighter-rouge">go</code> version 1.10 or newer. It can be downloaded <a href="https://golang.org/";>here</a>. Check that you have version 1.10 by running:</p> -<h4 id="example-walkthroughs">Example Walkthroughs</h4> +<div class="highlighter-rouge"><pre class="highlight"><code>$ go --version +</code></pre> +</div> -<p>See detailed walkthroughs of complete Beam pipelines.</p> +<h2 id="get-the-sdk-and-the-examples">Get the SDK and the examples</h2> -<ul> - <li><a href="/get-started/wordcount-example">WordCount</a>: Simple example pipelines that demonstrate basic Beam programming, including debugging and testing</li> - <li><a href="/get-started/mobile-gaming-example">Mobile Gaming</a>: A series of more advanced pipelines that demonstrate use cases in the mobile gaming domain</li> -</ul> +<p>The easiest way to obtain the Apache Beam Go SDK is via <code class="highlighter-rouge">go get</code>:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>$ go get -u github.com/apache/beam/sdks/go/... +</code></pre> +</div> + +<p>For development of the Go SDK itself, see <a href="https://github.com/apache/beam/blob/master/sdks/go/BUILD.md";>BUILD.md</a> for details.</p> -<h4 id="downloads-and-releases"><a href="/get-started/downloads">Downloads and Releases</a></h4> +<h2 id="run-wordcount">Run wordcount</h2> -<p>Find download links and information on the latest Beam releases, including versioning and release notes.</p> +<p>The Apache Beam +<a href="https://github.com/apache/beam/tree/master/sdks/go/examples";>examples</a> +directory has many examples. All examples can be run by passing the +required arguments described in the examples.</p> -<h4 id="support"><a href="/get-started/support">Support</a></h4> +<p>For example, to run <code class="highlighter-rouge">wordcount</code>, run:</p> + +<div class="runner-direct highlighter-rouge"><pre class="highlight"><code>$ go install github.com/apache/beam/sdks/go/examples/wordcount +$ wordcount --input <PATH_TO_INPUT_FILE> --output counts +</code></pre> +</div> + +<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code>$ go install github.com/apache/beam/sdks/go/examples/wordcount +$ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ + --output gs://<your-gcs-bucket>/counts \ + --runner dataflow \ + --project your-gcp-project \ + --temp_location gs://<your-gcs-bucket>/tmp/ \ + --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515 +</code></pre> +</div> + +<h2 id="next-steps">Next Steps</h2> + +<ul> + <li>Learn more about the <a href="/documentation/sdks/go/">Beam SDK for Go</a> +and look through the <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam";>godoc</a>.</li> + <li>Walk through these WordCount examples in the <a href="/get-started/wordcount-example">WordCount Example Walkthrough</a>.</li> + <li>Dive in to some of our favorite <a href="/documentation/resources">articles and presentations</a>.</li> + <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> +</ul> -<p>Find resources, such as mailing lists and issue tracking, to help you use Beam. Ask questions and discuss topics via <a href="http://stackoverflow.com/questions/tagged/apache-beam";>Stack Overflow</a> or on Beam’s <a href="http://apachebeam.slack.com";>Slack Channel</a>.</p> +<p>Please don’t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any issues!</p> </div> </div> diff --git a/content/get-started/quickstart-java/index.html b/content/get-started/quickstart-java/index.html index a88e065..8c6c663 100644 --- a/content/get-started/quickstart-java/index.html +++ b/content/get-started/quickstart-java/index.html @@ -96,6 +96,7 @@ <ul class="section-nav-list"> <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li> <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li> + <li><a href="/get-started/quickstart-go/">Quickstart - Go</a></li> </ul> </li> <li> diff --git a/content/get-started/quickstart-py/index.html b/content/get-started/quickstart-py/index.html index f935210..56db3ad 100644 --- a/content/get-started/quickstart-py/index.html +++ b/content/get-started/quickstart-py/index.html @@ -96,6 +96,7 @@ <ul class="section-nav-list"> <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li> <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li> + <li><a href="/get-started/quickstart-go/">Quickstart - Go</a></li> </ul> </li> <li> diff --git a/content/get-started/wordcount-example/index.html b/content/get-started/wordcount-example/index.html index 19000bb..93b610b 100644 --- a/content/get-started/wordcount-example/index.html +++ b/content/get-started/wordcount-example/index.html @@ -96,6 +96,7 @@ <ul class="section-nav-list"> <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li> <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li> + <li><a href="/get-started/quickstart-go/">Quickstart - Go</a></li> </ul> </li> <li> @@ -202,6 +203,7 @@ <ul> <li data-type="language-java">Java SDK</li> <li data-type="language-py">Python SDK</li> + <li data-type="language-go">Go SDK</li> </ul> </nav> @@ -232,7 +234,7 @@ to handle both bounded and unbounded datasets.</li> read from a text file, apply transforms to tokenize and count the words, and write the data to an output text file.</p> -<p class="language-java">This example hard-codes the locations for its input and output files and doesn’t +<p class="language-java language-go">This example hard-codes the locations for its input and output files and doesn’t perform any error checking; it is intended to only show you the “bare bones” of creating a Beam pipeline. This lack of parameterization makes this particular pipeline less portable across different runners than standard Beam pipelines. In @@ -247,12 +249,20 @@ show other best practices.</p> </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="err">$</span><span class="x"> </span><span class="k">go</span><span class="x"> </span><span class="n">install</span><span class="x"> </span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">apache</span><span class="o">/</span><span class="n">beam</span><span class="o">/</span><span class="n">sdks</span><span class="o">/</span><span cla [...] +</span><span class="err">$</span><span class="x"> </span><span class="n">minimal_wordcount</span><span class="x"> +</span></code></pre> +</div> + <p class="language-java">To view the full code in Java, see <strong><a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java";>MinimalWordCount</a>.</strong></p> <p class="language-py">To view the full code in Python, see <strong><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_minimal.py";>wordcount_minimal.py</a>.</strong></p> +<p class="language-go">To view the full code in Go, see +<strong><a href="https://github.com/apache/beam/blob/master/sdks/go/examples/minimal_wordcount/minimal_wordcount.go";>wordcount_minimal.go</a>.</strong></p> + <p><strong>Key Concepts:</strong></p> <ul> @@ -270,13 +280,13 @@ excerpts from the MinimalWordCount pipeline.</p> <h3 id="creating-the-pipeline">Creating the pipeline</h3> -<p>In this example, the code first creates a <code class="highlighter-rouge">PipelineOptions</code> object. This object +<p class="language-java language-py">In this example, the code first creates a <code class="highlighter-rouge">PipelineOptions</code> object. This object lets us set various options for our pipeline, such as the pipeline runner that will execute our pipeline and any runner-specific configuration required by the chosen runner. In this example we set these options programmatically, but more often, command-line arguments are used to set <code class="highlighter-rouge">PipelineOptions</code>.</p> -<p>You can specify a runner for executing your pipeline, such as the +<p class="language-java language-py">You can specify a runner for executing your pipeline, such as the <code class="highlighter-rouge">DataflowRunner</code> or <code class="highlighter-rouge">SparkRunner</code>. If you omit specifying a runner, as in this example, your pipeline executes locally using the <code class="highlighter-rouge">DirectRunner</code>. In the next sections, we will specify the pipeline’s runner.</p> @@ -299,10 +309,14 @@ sections, we will specify the pipeline’s runner.</p> </code></pre> </div> -<p>The next step is to create a <code class="highlighter-rouge">Pipeline</code> object with the options we’ve just +<p class="language-java language-py">The next step is to create a <code class="highlighter-rouge">Pipeline</code> object with the options we’ve just constructed. The Pipeline object builds up the graph of transformations to be executed, associated with that particular pipeline.</p> +<p class="language-go">The first step is to create a <code class="highlighter-rouge">Pipeline</code> object. It builds up the graph of +transformations to be executed, associated with that particular pipeline. +The scope allows grouping into composite transforms.</p> + <div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> </code></pre> </div> @@ -311,6 +325,11 @@ executed, associated with that particular pipeline.</p> </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="n">p</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">beam</span><span class="o">.</span><span class="n">NewPipeline</span><span class="x"> +</span><span class="n">s</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">p</span><span class="o">.</span><span class="n">Root</span><span class="p">()</span><span class="x"> +</span></code></pre> +</div> + <h3 id="applying-pipeline-transforms">Applying pipeline transforms</h3> <p>The MinimalWordCount pipeline contains several transforms to read data into the @@ -344,6 +363,10 @@ data stored in a publicly accessible Google Cloud Storage bucket (“gs://”).< <span class="s">'gs://dataflow-samples/shakespeare/kinglear.txt'</span><span class="p">)</span> </code></pre> </div> + + <div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="n">lines</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">textio</span><span class="o">.</span><span class="n">Read</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="s">"gs://apache-beam-samples/shakespeare/*"</span><span class="p">)</span><span class="x"> +</span></code></pre> + </div> </li> <li> <p>This transform splits the lines in <code class="highlighter-rouge">PCollection<String></code>, where each element @@ -366,6 +389,14 @@ previous <code class="highlighter-rouge">TextIO.Read</code> transform. The <code <span class="o">|</span> <span class="s">'ExtractWords'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">r'[A-Za-z</span><span class="se">\'</span><span class="s">]+'</span><span class="p">,</span> < [...] </code></pre> </div> + + <div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="n">words</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="k">func</span><span class="p">(</span><span class="n">line</span><span class="x"> </span><span class="kt">string</span><span cl [...] + </span><span class="k">for</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">word</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">range</span><span class="x"> </span><span class="n">wordRE</span><span class="o">.</span><span class="n">FindAllString</span><span class="p">(</span><span class="n">line</span><span class="p">,</span><span class="x"> </span><span class="o">-</ [...] + </span><span class="n">emit</span><span class="p">(</span><span class="n">word</span><span class="p">)</span><span class="x"> + </span><span class="p">}</span><span class="x"> +</span><span class="p">},</span><span class="x"> </span><span class="n">lines</span><span class="p">)</span><span class="x"> +</span></code></pre> + </div> </li> <li> <p>The SDK-provided <code class="highlighter-rouge">Count</code> transform is a generic transform that takes a @@ -386,6 +417,10 @@ the associated value is the occurrence count for each.</p> <div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">combiners</span><span class="o">.</span><span class="n">Count</span><span class="o">.</span><span class="n">PerElement</span><span class="p">()</span> </code></pre> </div> + + <div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="n">counted</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">stats</span><span class="o">.</span><span class="n">Count</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="n">words</span><span class="p">)</span><span class="x"> +</span></code></pre> + </div> </li> <li> <p>The next transform formats each of the key/value pairs of unique words and @@ -405,6 +440,12 @@ transform applies a function that produces exactly one output element.</p> <div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">word_count</span><span class="p">:</span> <span class="s">'</span><span class="si">%</span><span class="s">s: </span><span class="si">%</span><span class="s">s'</span> <span class="o">%</span> <span class="p">(</span><span class="n">word_count< [...] </code></pre> </div> + + <div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="n">formatted</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="k">func</span><span class="p">(</span><span class="n">w</span><span class="x"> </span><span class="kt">string</span><span c [...] + </span><span class="k">return</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Sprintf</span><span class="p">(</span><span class="s">"%s: %v"</span><span class="p">,</span><span class="x"> </span><span class="n">w</span><span class="p">,</span><span class="x"> </span><span class="n">c</span><span class="p">)</span><span class="x"> +</span><span class="p">},</span><span class="x"> </span><span class="n">counted</span><span class="p">)</span><span class="x"> +</span></code></pre> + </div> </li> <li> <p>A text file write transform. This transform takes the final <code class="highlighter-rouge">PCollection</code> of @@ -419,17 +460,25 @@ resulting output file.</p> <div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteToText</span><span class="p">(</span><span class="s">'gs://my-bucket/counts.txt'</span><span class="p">)</span> </code></pre> </div> + + <div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="n">textio</span><span class="o">.</span><span class="n">Write</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="s">"wordcounts.txt"</span><span class="p">,</span><span class="x"> </span><span class="n">formatted</span><span class="p">)</span><span class="x"> +</span></code></pre> + </div> </li> </ol> -<p>Note that the <code class="highlighter-rouge">Write</code> transform produces a trivial result value of type <code class="highlighter-rouge">PDone</code>, +<p class="language-java language-py">Note that the <code class="highlighter-rouge">Write</code> transform produces a trivial result value of type <code class="highlighter-rouge">PDone</code>, which in this case is ignored.</p> +<p class="language-go">Note that the <code class="highlighter-rouge">Write</code> transform returns no PCollections.</p> + <h3 id="running-the-pipeline">Running the pipeline</h3> -<p>Run the pipeline by calling the <code class="highlighter-rouge">run</code> method, which sends your pipeline to be +<p class="language-java language-py">Run the pipeline by calling the <code class="highlighter-rouge">run</code> method, which sends your pipeline to be executed by the pipeline runner that you specified in your <code class="highlighter-rouge">PipelineOptions</code>.</p> +<p class="language-go">Run the pipeline by passing it to a runner.</p> + <div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">().</span><span class="na">waitUntilFinish</span><span class="o">();</span> </code></pre> </div> @@ -438,7 +487,11 @@ executed by the pipeline runner that you specified in your <code class="highligh </code></pre> </div> -<p>Note that the <code class="highlighter-rouge">run</code> method is asynchronous. For a blocking execution, call the +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="n">direct</span><span class="o">.</span><span class="n">Execute</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="n">Background</span><span class="p">(),</span><span class="x"> </span><span class="n">p</span><span class="p">)</span><span class="x"> +</span></code></pre> +</div> + +<p class="language-java language-py">Note that the <code class="highlighter-rouge">run</code> method is asynchronous. For a blocking execution, call the <span class="language-java"><code class="highlighter-rouge">waitUntilFinish</code></span> <span class="language-py"><code class="highlighter-rouge">wait_until_finish</code></span> method on the result object returned by the call to <code class="highlighter-rouge">run</code>.</p> @@ -529,6 +582,42 @@ python -m apache_beam.examples.wordcount --input gs://dataflow-samples/shakespea <p>To view the full code in Python, see <strong><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount.py";>wordcount.py</a>.</strong></p> +<p><strong>To run this example in Go:</strong></p> + +<div class="runner-direct highlighter-rouge"><pre class="highlight"><code>$ go install github.com/apache/beam/sdks/go/examples/wordcount +$ wordcount --input <PATH_TO_INPUT_FILE> --output counts +</code></pre> +</div> + +<div class="runner-apex highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Go SDK. +</code></pre> +</div> + +<div class="runner-flink-local highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Go SDK. +</code></pre> +</div> + +<div class="runner-flink-cluster highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Go SDK. +</code></pre> +</div> + +<div class="runner-spark highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Go SDK. +</code></pre> +</div> + +<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code>$ go install github.com/apache/beam/sdks/go/examples/wordcount +$ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ + --output gs://<your-gcs-bucket>/counts \ + --runner dataflow \ + --project your-gcp-project \ + --temp_location gs://<your-gcs-bucket>/tmp/ \ + --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515 +</code></pre> +</div> + +<p>To view the full code in Go, see +<strong><a href="https://github.com/apache/beam/blob/master/sdks/go/examples/wordcount/wordcount.go";>wordcount.go</a>.</strong></p> + <p><strong>New Concepts:</strong></p> <ul> @@ -542,7 +631,7 @@ pipeline code into smaller sections.</p> <h3 id="specifying-explicit-dofns">Specifying explicit DoFns</h3> -<p>When using <code class="highlighter-rouge">ParDo</code> transforms, you need to specify the processing operation that +<p class="language-java language-py">When using <code class="highlighter-rouge">ParDo</code> transforms, you need to specify the processing operation that gets applied to each element in the input <code class="highlighter-rouge">PCollection</code>. This processing operation is a subclass of the SDK class <code class="highlighter-rouge">DoFn</code>. You can create the <code class="highlighter-rouge">DoFn</code> subclasses for each <code class="highlighter-rouge">ParDo</code> inline, as an anonymous inner class instance, as is @@ -550,6 +639,13 @@ done in the previous example (MinimalWordCount). However, it’s often a good idea to define the <code class="highlighter-rouge">DoFn</code> at the global level, which makes it easier to unit test and can make the <code class="highlighter-rouge">ParDo</code> code more readable.</p> +<p class="language-go">When using <code class="highlighter-rouge">ParDo</code> transforms, you need to specify the processing operation that +gets applied to each element in the input <code class="highlighter-rouge">PCollection</code>. This processing +operation is either a named function or a struct with specially-named methods. You +can use anomynous functions (but not closures). However, it’s often a good +idea to define the <code class="highlighter-rouge">DoFn</code> at the global level, which makes it easier to unit +test and can make the <code class="highlighter-rouge">ParDo</code> code more readable.</p> + <div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// In this example, ExtractWordsFn is a DoFn that is defined as a static class:</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">ExtractWordsFn</span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">></span> <span class="o">{</span> @@ -575,17 +671,32 @@ test and can make the <code class="highlighter-rouge">ParDo</code> code more rea </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="c">// In this example, extractFn is a DoFn that is defined as a function:</span><span class="x"> + +</span><span class="k">func</span><span class="x"> </span><span class="n">extractFn</span><span class="p">(</span><span class="n">ctx</span><span class="x"> </span><span class="n">context</span><span class="o">.</span><span class="n">Context</span><span class="p">,</span><span class="x"> </span><span class="n">line</span><span class="x"> </span><span class="kt">string</span><span class="p">,</span><span class="x"> </span><span class="n">emit</span><span class="x"> </span><span class="k"> [...] + </span><span class="o">...</span><span class="x"> +</span><span class="p">}</span><span class="x"> +</span></code></pre> +</div> + <h3 id="creating-composite-transforms">Creating composite transforms</h3> -<p>If you have a processing operation that consists of multiple transforms or +<p class="language-java language-py">If you have a processing operation that consists of multiple transforms or <code class="highlighter-rouge">ParDo</code> steps, you can create it as a subclass of <code class="highlighter-rouge">PTransform</code>. Creating a <code class="highlighter-rouge">PTransform</code> subclass allows you to encapsulate complex transforms, can make your pipeline’s structure more clear and modular, and makes unit testing easier.</p> -<p>In this example, two transforms are encapsulated as the <code class="highlighter-rouge">PTransform</code> subclass +<p class="language-go">If you have a processing operation that consists of multiple transforms or +<code class="highlighter-rouge">ParDo</code> steps, you can use a normal Go function to encapsulate them. You can +furthermore use a named subscope to group them as a composite transform visible +for monitoring.</p> + +<p class="language-java language-py">In this example, two transforms are encapsulated as the <code class="highlighter-rouge">PTransform</code> subclass <code class="highlighter-rouge">CountWords</code>. <code class="highlighter-rouge">CountWords</code> contains the <code class="highlighter-rouge">ParDo</code> that runs <code class="highlighter-rouge">ExtractWordsFn</code> and the SDK-provided <code class="highlighter-rouge">Count</code> transform.</p> +<p class="language-go">In this example, two transforms are encapsulated as a <code class="highlighter-rouge">CountWords</code> function.</p> + <p>When <code class="highlighter-rouge">CountWords</code> is defined, we specify its ultimate input and output; the input is the <code class="highlighter-rouge">PCollection<String></code> for the extraction operation, and the output is the <code class="highlighter-rouge">PCollection<KV<String, Long>></code> produced by the count operation.</p> @@ -632,6 +743,18 @@ is the <code class="highlighter-rouge">PCollection<KV<String, Long>> </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="k">func</span><span class="x"> </span><span class="n">CountWords</span><span class="p">(</span><span class="n">s</span><span class="x"> </span><span class="n">beam</span><span class="o">.</span><span class="n">Scope</span><span class="p">,</span><span class="x"> </span><span class="n">lines</span><span class="x"> </span><span class="n">beam</span><span class="o">.</span><span class="n">PCollection</span> [...] + </span><span class="n">s</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">Scope</span><span class="p">(</span><span class="s">"CountWords"</span><span class="p">)</span><span class="x"> + + </span><span class="c">// Convert lines of text into individual words.</span><span class="x"> + </span><span class="n">col</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="n">extractFn</span><span class="p">,</span><span class="x"> </span><span class="n">lines</span><span class="p">)</span><span class="x"> + + </span><span class="c">// Count the number of times each word occurs.</span><span class="x"> + </span><span class="k">return</span><span class="x"> </span><span class="n">stats</span><span class="o">.</span><span class="n">Count</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="n">col</span><span class="p">)</span><span class="x"> +</span><span class="p">}</span><span class="x"> +</span></code></pre> +</div> + <h3 id="using-parameterizable-pipelineoptions">Using parameterizable PipelineOptions</h3> <p>You can hard-code various execution options when you run your pipeline. However, @@ -639,9 +762,11 @@ the more common way is to define your own configuration options via command-line argument parsing. Defining your configuration options via the command-line makes the code more easily portable across different runners.</p> -<p>Add arguments to be processed by the command-line parser, and specify default +<p class="language-java language-py">Add arguments to be processed by the command-line parser, and specify default values for them. You can then access the options values in your pipeline code.</p> +<p class="language-go">You can use the standard <code class="highlighter-rouge">flag</code> package for this purpose.</p> + <div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="kd">interface</span> <span class="nc">WordCountOptions</span> <span class="kd">extends</span> <span class="n">PipelineOptions</span> <span class="o">{</span> <span class="nd">@Description</span><span class="o">(</span><span class="s">"Path of the file to read from"</span><span class="o">)</span> <span class="nd">@Default</span><span class="o">.</span><span class="na">String</span><span class="o">(</span><span class="s">"gs://dataflow-samples/shakespeare/kinglear.txt"</span><span class="o">)</span> @@ -672,6 +797,18 @@ values for them. You can then access the options values in your pipeline code.</ </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="k">var</span><span class="x"> </span><span class="n">input</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">flag</span><span class="o">.</span><span class="n">String</span><span class="p">(</span><span class="s">"input"</span><span class="p">,</span><span class="x"> </span><span class="s">"gs://apache-beam-samples/shakespeare/kinglear.txt"</span><span class="p [...] + +</span><span class="k">func</span><span class="x"> </span><span class="n">main</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x"> + </span><span class="o">...</span><span class="x"> + </span><span class="n">p</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">beam</span><span class="o">.</span><span class="n">NewPipeline</span><span class="p">()</span><span class="x"> + </span><span class="n">s</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">p</span><span class="o">.</span><span class="n">Root</span><span class="p">()</span><span class="x"> + + </span><span class="n">lines</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">textio</span><span class="o">.</span><span class="n">Read</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="o">*</span><span class="n">input</span><span class="p">)</span><span class="x"> + </span><span class="o">...</span><span class="x"> +</span></code></pre> +</div> + <h2 id="debuggingwordcount-example">DebuggingWordCount example</h2> <p>The DebuggingWordCount example demonstrates some best practices for @@ -752,6 +889,42 @@ python -m apache_beam.examples.wordcount_debugging --input gs://dataflow-samples <p>To view the full code in Python, see <strong><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_debugging.py";>wordcount_debugging.py</a>.</strong></p> +<p><strong>To run this example in Go:</strong></p> + +<div class="runner-direct highlighter-rouge"><pre class="highlight"><code>$ go install github.com/apache/beam/sdks/go/examples/debugging_wordcount +$ debugging_wordcount --input <PATH_TO_INPUT_FILE> --output counts +</code></pre> +</div> + +<div class="runner-apex highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Go SDK. +</code></pre> +</div> + +<div class="runner-flink-local highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Go SDK. +</code></pre> +</div> + +<div class="runner-flink-cluster highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Go SDK. +</code></pre> +</div> + +<div class="runner-spark highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Go SDK. +</code></pre> +</div> + +<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code>$ go install github.com/apache/beam/sdks/go/examples/debugging_wordcount +$ debugging_wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ + --output gs://<your-gcs-bucket>/counts \ + --runner dataflow \ + --project your-gcp-project \ + --temp_location gs://<your-gcs-bucket>/tmp/ \ + --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515 +</code></pre> +</div> + +<p>To view the full code in Go, see +<strong><a href="https://github.com/apache/beam/blob/master/sdks/go/examples/debugging_wordcount/debugging_wordcount.go";>debugging_wordcount.go</a>.</strong></p> + <p><strong>New Concepts:</strong></p> <ul> @@ -825,6 +998,23 @@ pipeline code into smaller sections.</p> </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="k">type</span><span class="x"> </span><span class="n">filterFn</span><span class="x"> </span><span class="k">struct</span><span class="x"> </span><span class="p">{</span><span class="x"> + </span><span class="o">...</span><span class="x"> +</span><span class="p">}</span><span class="x"> + +</span><span class="k">func</span><span class="x"> </span><span class="p">(</span><span class="n">f</span><span class="x"> </span><span class="o">*</span><span class="n">filterFn</span><span class="p">)</span><span class="x"> </span><span class="n">ProcessElement</span><span class="p">(</span><span class="n">ctx</span><span class="x"> </span><span class="n">context</span><span class="o">.</span><span class="n">Context</span><span class="p">,</span><span class="x"> </span><span class="n"> [...] + </span><span class="k">if</span><span class="x"> </span><span class="n">f</span><span class="o">.</span><span class="n">re</span><span class="o">.</span><span class="n">MatchString</span><span class="p">(</span><span class="n">word</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x"> + </span><span class="c">// Log at the "INFO" level each element that we match.</span><span class="x"> + </span><span class="n">log</span><span class="o">.</span><span class="n">Infof</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span><span class="x"> </span><span class="s">"Matched: %v"</span><span class="p">,</span><span class="x"> </span><span class="n">word</span><span class="p">)</span><span class="x"> + </span><span class="n">emit</span><span class="p">(</span><span class="n">word</span><span class="p">,</span><span class="x"> </span><span class="n">count</span><span class="p">)</span><span class="x"> + </span><span class="p">}</span><span class="x"> </span><span class="k">else</span><span class="x"> </span><span class="p">{</span><span class="x"> + </span><span class="c">// Log at the "DEBUG" level each element that is not matched.</span><span class="x"> + </span><span class="n">log</span><span class="o">.</span><span class="n">Debugf</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span><span class="x"> </span><span class="s">"Did not match: %v"</span><span class="p">,</span><span class="x"> </span><span class="n">word</span><span class="p">)</span><span class="x"> + </span><span class="p">}</span><span class="x"> +</span><span class="p">}</span><span class="x"> +</span></code></pre> +</div> + <h4 id="direct-runner">Direct Runner</h4> <p>When executing your pipeline with the <code class="highlighter-rouge">DirectRunner</code>, you can print log @@ -841,7 +1031,7 @@ that Cloud Dataflow has spun up to complete your job. Logging statements in your pipeline’s <code class="highlighter-rouge">DoFn</code> instances will appear in Stackdriver Logging as your pipeline runs.</p> -<p>You can also control the worker log levels. Cloud Dataflow workers that execute +<p class="language-java language-py">You can also control the worker log levels. Cloud Dataflow workers that execute user code are configured to log to Stackdriver Logging by default at “INFO” log level and higher. You can override log levels for specific logging namespaces by specifying: <code class="highlighter-rouge">--workerLogLevelOverrides={"Name1":"Level1","Name2":"Level2",...}</code>. @@ -850,7 +1040,7 @@ when executing a pipeline using the Cloud Dataflow service, Stackdriver Logging will contain only “DEBUG” or higher level logs for the package in addition to the default “INFO” or higher level logs.</p> -<p>The default Cloud Dataflow worker logging configuration can be overridden by +<p class="language-java language-py">The default Cloud Dataflow worker logging configuration can be overridden by specifying <code class="highlighter-rouge">--defaultWorkerLogLevel=<one of TRACE, DEBUG, INFO, WARN, ERROR></code>. For example, by specifying <code class="highlighter-rouge">--defaultWorkerLogLevel=DEBUG</code> when executing a pipeline with the Cloud Dataflow service, Cloud Logging will contain all “DEBUG” @@ -880,17 +1070,21 @@ or DEBUG significantly increases the amount of logs output.</p> <h3 id="testing-your-pipeline-with-asserts">Testing your pipeline with asserts</h3> -<p><span class="language-java"><code class="highlighter-rouge">PAssert</code></span><span class="language-py"><code class="highlighter-rouge">assert_that</code></span> +<p class="language-java language-py"><span class="language-java"><code class="highlighter-rouge">PAssert</code></span><span class="language-py"><code class="highlighter-rouge">assert_that</code></span> is a set of convenient PTransforms in the style of Hamcrest’s collection matchers that can be used when writing pipeline level tests to validate the contents of PCollections. Asserts are best used in unit tests with small data sets.</p> +<p class="language-go">The <code class="highlighter-rouge">passert</code> package contains convenient PTransforms that can be used when +writing pipeline level tests to validate the contents of PCollections. Asserts +are best used in unit tests with small data sets.</p> + <p class="language-java">The following example verifies that the set of filtered words matches our expected counts. The assert does not produce any output, and the pipeline only succeeds if all of the expectations are met.</p> -<p class="language-py">The following example verifies that two collections contain the same values. The +<p class="language-py language-go">The following example verifies that two collections contain the same values. The assert does not produce any output, and the pipeline only succeeds if all of the expectations are met.</p> @@ -913,6 +1107,11 @@ expectations are met.</p> </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="o">...</span><span class="x"> +</span><span class="n">passert</span><span class="o">.</span><span class="n">Equals</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="n">formatted</span><span class="p">,</span><span class="x"> </span><span class="s">"Flourish: 3"</span><span class="p">,</span><span class="x"> </span><span class="s">"stomach: 1"</span><span class="p">)</span><span class="x"> +</span></code></pre> +</div> + <p class="language-java">See <a href="https://github.com/apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/DebuggingWordCountTest.java";>DebuggingWordCountTest</a> for an example unit test.</p> @@ -1012,6 +1211,42 @@ python -m apache_beam.examples.windowed_wordcount --input YOUR_INPUT_FILE \ <p>To view the full code in Python, see <strong><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/windowed_wordcount.py";>windowed_wordcount.py</a>.</strong></p> +<p><strong>To run this example in Python:</strong></p> + +<div class="runner-direct highlighter-rouge"><pre class="highlight"><code>$ go install github.com/apache/beam/sdks/go/examples/windowed_wordcount +$ windowed_wordcount --input <PATH_TO_INPUT_FILE> --output counts +</code></pre> +</div> + +<div class="runner-apex highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Go SDK. +</code></pre> +</div> + +<div class="runner-flink-local highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Go SDK. +</code></pre> +</div> + +<div class="runner-flink-cluster highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Go SDK. +</code></pre> +</div> + +<div class="runner-spark highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Go SDK. +</code></pre> +</div> + +<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code>$ go install github.com/apache/beam/sdks/go/examples/windowed_wordcount +$ windowed_wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ + --output gs://<your-gcs-bucket>/counts \ + --runner dataflow \ + --project your-gcp-project \ + --temp_location gs://<your-gcs-bucket>/tmp/ \ + --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515 +</code></pre> +</div> + +<p>To view the full code in Go, see +<strong><a href="https://github.com/apache/beam/blob/master/sdks/go/examples/windowed_wordcount/windowed_wordcount.go";>windowed_wordcount.go</a>.</strong></p> + <h3 id="unbounded-and-bounded-pipeline-input-modes">Unbounded and bounded pipeline input modes</h3> <p>Beam allows you to create a single pipeline that can handle both bounded and @@ -1042,6 +1277,17 @@ file:</p> </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="k">func</span><span class="x"> </span><span class="n">main</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x"> + </span><span class="o">...</span><span class="x"> + </span><span class="n">p</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">beam</span><span class="o">.</span><span class="n">NewPipeline</span><span class="p">()</span><span class="x"> + </span><span class="n">s</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">p</span><span class="o">.</span><span class="n">Root</span><span class="p">()</span><span class="x"> + + </span><span class="n">lines</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">textio</span><span class="o">.</span><span class="n">Read</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="o">*</span><span class="n">input</span><span class="p">)</span><span class="x"> + </span><span class="o">...</span><span class="x"> +</span><span class="p">}</span><span class="x"> +</span></code></pre> +</div> + <h3 id="adding-timestamps-to-data">Adding timestamps to data</h3> <p>Each element in a <code class="highlighter-rouge">PCollection</code> has an associated <a href="/documentation/programming-guide#element-timestamps">timestamp</a>. @@ -1063,6 +1309,10 @@ each element in the <code class="highlighter-rouge">PCollection</code>.</p> </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="n">timestampedLines</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="o">&</span><span class="n">addTimestampFn</span><span class="p">{</span><span class="n">Min</span><span class="o">:< [...] +</span></code></pre> +</div> + <p>Below is the code for <code class="highlighter-rouge">AddTimestampFn</code>, a <code class="highlighter-rouge">DoFn</code> invoked by <code class="highlighter-rouge">ParDo</code>, that sets the data element of the timestamp given the element itself. For example, if the elements were log lines, this <code class="highlighter-rouge">ParDo</code> could parse the time out of the log string @@ -1100,6 +1350,20 @@ timestamp sometime in a 2-hour period.</p> </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="k">type</span><span class="x"> </span><span class="n">addTimestampFn</span><span class="x"> </span><span class="k">struct</span><span class="x"> </span><span class="p">{</span><span class="x"> + </span><span class="n">Min</span><span class="x"> </span><span class="n">beam</span><span class="o">.</span><span class="n">EventTime</span><span class="x"> </span><span class="s">`json:"min"`</span><span class="x"> +</span><span class="p">}</span><span class="x"> + +</span><span class="k">func</span><span class="x"> </span><span class="p">(</span><span class="n">f</span><span class="x"> </span><span class="o">*</span><span class="n">addTimestampFn</span><span class="p">)</span><span class="x"> </span><span class="n">ProcessElement</span><span class="p">(</span><span class="n">x</span><span class="x"> </span><span class="n">beam</span><span class="o">.</span><span class="n">X</span><span class="p">)</span><span class="x"> </span><span class="p">(</sp [...] + </span><span class="n">timestamp</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">f</span><span class="o">.</span><span class="n">Min</span><span class="o">.</span><span class="n">Add</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">Duration</span><span class="p">(</span><span class="n">rand</span><span class="o">.</span><span class="n">Int63n</span><span class="p">(</span><span class="m">2</sp [...] + </span><span class="k">return</span><span class="x"> </span><span class="n">timestamp</span><span class="p">,</span><span class="x"> </span><span class="n">x</span><span class="x"> +</span><span class="p">}</span><span class="x"> +</span></code></pre> +</div> + +<p class="language-go">Note that the use of the <code class="highlighter-rouge">beam.X</code> “type variable” allows the transform to be +used for any type.</p> + <h3 id="windowing">Windowing</h3> <p>Beam uses a concept called <strong>Windowing</strong> to subdivide a <code class="highlighter-rouge">PCollection</code> into @@ -1121,6 +1385,11 @@ defaults to 1 minute (you can change this with a command-line option).</p> </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="n">windowedLines</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="n">window</span><span class="o">.</span><span class="n">NewFixedWindows</span><span class="p">(</span><span class="n"> [...] + +</span></code></pre> +</div> + <h3 id="reusing-ptransforms-over-windowed-pcollections">Reusing PTransforms over windowed PCollections</h3> <p>You can reuse existing PTransforms that were created for manipulating simple @@ -1134,6 +1403,10 @@ PCollections over windowed PCollections as well.</p> </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="n">counted</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">wordcount</span><span class="o">.</span><span class="n">CountWords</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="x"> </span><span class="n">windowedLines</span><span class="p">)</span><span class="x"> +</span></code></pre> +</div> + <h2 id="streamingwordcount-example">StreamingWordCount example</h2> <p>The StreamingWordCount example is a streaming pipeline that reads Pub/Sub @@ -1196,6 +1469,13 @@ python -m apache_beam.examples.streaming_wordcount \ <p>To view the full code in Python, see <strong><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/streaming_wordcount.py";>streaming_wordcount.py</a>.</strong></p> +<p><strong>To run this example in Go:</strong></p> + +<blockquote> + <p><strong>Note:</strong> StreamingWordCount is not yet available for the Go SDK. There is an open issue for this +(<a href="https://issues.apache.org/jira/browse/BEAM-4292";>BEAM-4292</a>).</p> +</blockquote> + <h3 id="reading-an-unbounded-data-set">Reading an unbounded data set</h3> <p>This example uses an unbounded data set as input. The code reads Pub/Sub @@ -1213,6 +1493,10 @@ messages from a Pub/Sub subscription or topic using <span class="n">lines</span> <span class="o">=</span> <span class="n">p</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadStringsFromPubSub</span><span class="p">(</span><span class="n">topic</span><span class="o">=</span><span class="n">known_args</span><span class="o">.</span><span class="n">input_topic</span><span class="p">)</span> </code></pre> </div> + +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="x"> </span><span class="c">// This example is not currently available for the Beam SDK for Go.</span><span class="x"> +</span></code></pre> +</div> <h3 id="writing-unbounded-results">Writing unbounded results</h3> <p>When the input is unbounded, the same is true of the output <code class="highlighter-rouge">PCollection</code>. As @@ -1231,6 +1515,9 @@ using <a href="/documentation/sdks/pydoc/2.4.0/apache_beam.io.gcp.pubsub.html#ap <span class="n">output</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteStringsToPubSub</span><span class="p">(</span><span class="n">known_args</span><span class="o">.</span><span class="n">output_topic</span><span class="p">)</span> </code></pre> </div> +<div class="language-go highlighter-rouge"><pre class="highlight"><code><span class="x"> </span><span class="c">// This example is not currently available for the Beam SDK for Go.</span><span class="x"> +</span></code></pre> +</div> </div> diff --git a/content/images/logos/sdks/go.png b/content/images/logos/sdks/go.png new file mode 100644 index 0000000..d6d98c3 Binary files /dev/null and b/content/images/logos/sdks/go.png differ diff --git a/content/index.html b/content/index.html index 8c132ce..85e8fbd 100644 --- a/content/index.html +++ b/content/index.html @@ -102,6 +102,7 @@ <div class="hero__ctas"> <a class="button" href="/get-started/quickstart-java/">Java Quickstart</a> <a class="button" href="/get-started/quickstart-py/">Python Quickstart</a> + <a class="button" href="/get-started/quickstart-go/">Go Quickstart</a> </div> </div> </div> @@ -276,6 +277,7 @@ <div class="ctas__ctas"> <a class="button" href="/get-started/quickstart-java/">Java Quickstart</a> <a class="button" href="/get-started/quickstart-py/">Python Quickstart</a> + <a class="button" href="/get-started/quickstart-go/">Go Quickstart</a> </div> </div> -- To stop receiving notification emails like this one, please contact [email protected].
