This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7fcfe6a Publishing website 2021/07/27 18:01:39 at commit 69822e4
7fcfe6a is described below
commit 7fcfe6a62e375a661d76c0b1fe0159f0c1a15471
Author: jenkins <[email protected]>
AuthorDate: Tue Jul 27 18:01:40 2021 +0000
Publishing website 2021/07/27 18:01:39 at commit 69822e4
---
website/generated-content/documentation/index.xml | 34 ++++++++++++++++++++++
.../documentation/programming-guide/index.html | 12 ++++----
website/generated-content/sitemap.xml | 2 +-
3 files changed, 42 insertions(+), 6 deletions(-)
diff --git a/website/generated-content/documentation/index.xml
b/website/generated-content/documentation/index.xml
index 28f2ae0..8690ebb 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -6624,6 +6624,14 @@ transform at any point while constructing your pipeline
to create a new
<div class="highlight"><pre class="chroma"><code class="language-py"
data-lang="py"><span class="n">lines</span> <span
class="o">=</span> <span class="n">pipeline</span> <span
class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">io</span><span
class="o">.</span><span class="n">ReadFromText</span><span
class="p">(</span><span
class="s1">&#39;gs://some/inputData.txt&#39;</spa [...]
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">lines</span> <span
class="o">:=</span> <span class="nx">textio</span><span
class="p">.</span><span class="nf">Read</span><span
class="p">(</span><span class="nx">scope</span><span
class="p">,</span> <span class="err">&#39;</span><span
class="nx">gs</span><span class="p">:</span><span
class="o">//</span>&l [...]
+</div>
+</div>
<h3 id="pipeline-io-writing-data">5.2. Writing output data</h3>
<p>Write transforms write the data in a <code>PCollection</code> to
an external data source.
You will most often use write transforms at the end of your pipeline to output
@@ -6645,6 +6653,14 @@ a <code>PCollection</code>'s data at any point in
your pipeline.</p>
<div class="highlight"><pre class="chroma"><code class="language-py"
data-lang="py"><span class="n">output</span> <span
class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">io</span><span
class="o">.</span><span class="n">WriteToText</span><span
class="p">(</span><span
class="s1">&#39;gs://some/outputData&#39;</span><span
class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">textio</span><span
class="p">.</span><span class="nf">Write</span><span
class="p">(</span><span class="nx">scope</span><span
class="p">,</span> <span class="err">&#39;</span><span
class="nx">gs</span><span class="p">:</span><span
class="o">//</span><span class="nx">some</span><span
class="o">/</span><s [...]
+</div>
+</div>
<h3 id="file-based-data">5.3. File-based input and output data</h3>
<h4 id="file-based-reading-multiple-locations">5.3.1. Reading from multiple
locations</h4>
<p>Many read transforms support reading from multiple input files matching
a glob
@@ -6670,6 +6686,14 @@ suffix &ldquo;.csv&rdquo; in the given
location:</p>
<span class="s1">&#39;path/to/input-*.csv&#39;</span><span
class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">lines</span> <span
class="o">:=</span> <span class="nx">textio</span><span
class="p">.</span><span class="nf">Read</span><span
class="p">(</span><span class="nx">scope</span><span
class="p">,</span> <span
class="s">&#34;path/to/input-*.csv&#34;</span><span
class="p">)</span></code></pre></div>
+</div>
+</div>
<p>To read data from disparate sources into a single
<code>PCollection</code>, read each one
independently and then use the <a href="#flatten">Flatten</a> transform
to create a single
<code>PCollection</code>.</p>
@@ -6700,6 +6724,16 @@ location. Each file has the prefix
&ldquo;numbers&rdquo;, a numeric tag,
<span class="s1">&#39;/path/to/numbers&#39;</span><span
class="p">,</span> <span class="n">file_name_suffix</span><span
class="o">=</span><span
class="s1">&#39;.csv&#39;</span><span
class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="c1">// The Go SDK textio doesn&#39;t support
sharding on writes yet.
+</span><span class="c1">// See
https://issues.apache.org/jira/browse/BEAM-12664 for ways
+</span><span class="c1"></span><span class="o">//</span>
<span class="nx">to</span> <span class="nx">contribute</span>
<span class="nx">a</span> <span class="nx">solution</span><span
class="p">.</span></code></pre></div>
+</div>
+</div>
<h3 id="provided-io-transforms">5.4. Beam-provided I/O transforms</h3>
<p>See the <a href="/documentation/io/built-in/">Beam-provided I/O
Transforms</a>
page for a list of the currently available I/O transforms.</p>
diff --git
a/website/generated-content/documentation/programming-guide/index.html
b/website/generated-content/documentation/programming-guide/index.html
index 589910c..1e327ad 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -1844,16 +1844,16 @@ built-in transforms, you can <a
href=/documentation/io/developing-io-overview/>i
transforms</a>.</p><h3 id=pipeline-io-reading-data>5.1. Reading input
data</h3><p>Read transforms read data from an external source and return a
<code>PCollection</code>
representation of the data for use by your pipeline. You can use a read
transform at any point while constructing your pipeline to create a new
-<code>PCollection</code>, though it will be most common at the start of your
pipeline.</p><div class="language-java snippet"><div class="notebook-skip
code-snippet"><a class=copy type=button data-bs-toggle=tooltip
data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PCollection</span><span
class=o><</span><span class=n>String</span><span class=o>></span> <s [...]
+<code>PCollection</code>, though it will be most common at the start of your
pipeline.</p><div class="language-java snippet"><div class="notebook-skip
code-snippet"><a class=copy type=button data-bs-toggle=tooltip
data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PCollection</span><span
class=o><</span><span class=n>String</span><span class=o>></span> <s [...]
You will most often use write transforms at the end of your pipeline to output
your pipeline’s final results. However, you can use a write transform to
output
-a <code>PCollection</code>'s data at any point in your pipeline.</p><div
class="language-java snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-java
data-lang=java><span class=n>output</span><span class=o>.</span><span
class=na>apply</span><span class=o>(</span><span class=n>TextIO</span><span cla
[...]
+a <code>PCollection</code>'s data at any point in your pipeline.</p><div
class="language-java snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-java
data-lang=java><span class=n>output</span><span class=o>.</span><span
class=na>apply</span><span class=o>(</span><span class=n>TextIO</span><span cla
[...]
operator you provide. Note that glob operators are filesystem-specific and obey
filesystem-specific consistency models. The following TextIO example uses a
glob
operator (<code>*</code>) to read all matching input files that have prefix
“input-” and the
suffix “.csv” in the given location:</p><div class="language-java
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>p</span><span
class=o>.</span><span class=na>apply</span><span class=o>(</span><span
class=s>"ReadFromText"</span><span class=o> [...]
<span class=n>TextIO</span><span class=o>.</span><span
class=na>read</span><span class=o>().</span><span class=na>from</span><span
class=o>(</span><span
class=s>"protocol://my_bucket/path/to/input-*.csv"</span><span
class=o>));</span></code></pre></div></div></div><div class="language-py
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div cl [...]
- <span class=s1>'path/to/input-*.csv'</span><span
class=p>)</span></code></pre></div></div></div><p>To read data from disparate
sources into a single <code>PCollection</code>, read each one
+ <span class=s1>'path/to/input-*.csv'</span><span
class=p>)</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-go data-lang=go><span class=nx>lines</span> <span
class=o>:=</span> <span class=nx>textio</span><span class=p>.</spa [...]
independently and then use the <a href=#flatten>Flatten</a> transform to
create a single
<code>PCollection</code>.</p><h4 id=file-based-writing-multiple-files>5.3.2.
Writing to multiple output files</h4><p>For file-based output data, write
transforms write to multiple output files by
default. When you pass an output file name to a write transform, the file name
@@ -1863,7 +1863,9 @@ location. Each file has the prefix “numbers”,
a numeric tag, and the
“.csv”.</p><div class="language-java snippet"><div
class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>records</span><span
class=o>.</span><span class=na>apply</span><span class=o>(</span><span
class=s>"WriteToText"</span><span class=o>,</span>
<span class=n>TextIO</span><span class=o>.</span><span
class=na>write</span><span class=o>().</span><span class=na>to</span><span
class=o>(</span><span
class=s>"protocol://my_bucket/path/to/numbers"</span><span
class=o>)</span>
<span class=o>.</span><span class=na>withSuffix</span><span
class=o>(</span><span class=s>".csv"</span><span
class=o>));</span></code></pre></div></div></div><div class="language-py
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-py data-lang=py><span class=n>filtered_wor [...]
- <span class=s1>'/path/to/numbers'</span><span class=p>,</span>
<span class=n>file_name_suffix</span><span class=o>=</span><span
class=s1>'.csv'</span><span
class=p>)</span></code></pre></div></div></div><h3
id=provided-io-transforms>5.4. Beam-provided I/O transforms</h3><p>See the <a
href=/documentation/io/built-in/>Beam-provided I/O Transforms</a>
+ <span class=s1>'/path/to/numbers'</span><span class=p>,</span>
<span class=n>file_name_suffix</span><span class=o>=</span><span
class=s1>'.csv'</span><span
class=p>)</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-g [...]
+</span><span class=c1>// See https://issues.apache.org/jira/browse/BEAM-12664
for ways
+</span><span class=c1></span><span class=o>//</span> <span class=nx>to</span>
<span class=nx>contribute</span> <span class=nx>a</span> <span
class=nx>solution</span><span
class=p>.</span></code></pre></div></div></div><h3
id=provided-io-transforms>5.4. Beam-provided I/O transforms</h3><p>See the <a
href=/documentation/io/built-in/>Beam-provided I/O Transforms</a>
page for a list of the currently available I/O transforms.</p><h2
id=schemas>6. Schemas</h2><p>Often, the types of the records being processed
have an obvious structure. Common Beam sources produce
JSON, Avro, Protocol Buffer, or database row objects; all of these types have
well defined structures,
structures that can often be determined by examining the type. Even within a
SDK pipeline, Simple Java POJOs
@@ -3919,7 +3921,7 @@ kafka_records = (
ImplicitSchemaPayloadBuilder({'data': u'0'}),
<Address of expansion service>))
assert_that(res, equal_to(['0a', '0b']))
- </code></pre></div></div></li><li><p>After the job has been submitted to
the Beam runner, shutdown the expansion service by terminating the expansion
service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3.
Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the
Direct runner can be used with multi-language pipelines.</p><p>Google Cloud
Dataflow supports multi-language pipelines through the Dataflow Runner v2
backend architecture.</p><div cla [...]
+ </code></pre></div></div></li><li><p>After the job has been submitted to
the Beam runner, shutdown the expansion service by terminating the expansion
service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3.
Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the
Direct runner can be used with multi-language pipelines.</p><p>Google Cloud
Dataflow supports multi-language pipelines through the Dataflow Runner v2
backend architecture.</p><div cla [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam
logo, and the Apache feather logo are either registered trademarks or
trademarks of The Apache Software Foundation. All other products or name brands
are trademarks of their respective holders, including The Apache Software
Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml
b/website/generated-content/sitemap.xml
index a52845a..f5d773e 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.31.0/</loc><lastmod>2021-06-22T18:45:24-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-07-01T15:48:01-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-07-01T15:48:01-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-07-01T15:48:01-07:00</lastmod></url><url><loc>/blog/b
[...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.31.0/</loc><lastmod>2021-06-22T18:45:24-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-07-01T15:48:01-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-07-01T15:48:01-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-07-01T15:48:01-07:00</lastmod></url><url><loc>/blog/b
[...]
\ No newline at end of file
