This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 0c9a60b Publishing website 2021/01/28 18:03:18 at commit f24ebd3
0c9a60b is described below
commit 0c9a60bf7f267a2be47f72539392a9ab77a76547
Author: jenkins <[email protected]>
AuthorDate: Thu Jan 28 18:03:18 2021 +0000
Publishing website 2021/01/28 18:03:18 at commit f24ebd3
---
website/generated-content/documentation/index.xml | 49 +++++++++++++++++++-
.../io/built-in/google-bigquery/index.html | 53 +++++++++++++++++++++-
2 files changed, 100 insertions(+), 2 deletions(-)
diff --git a/website/generated-content/documentation/index.xml
b/website/generated-content/documentation/index.xml
index 4faffd6..f51e8f6 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -12734,7 +12734,54 @@ GitHub</a>.</p>
</div>
<p>The following code snippet reads with a query string.</p>
<div class=language-java>
-<div class="highlight"><pre class="chroma"><code
class="language-java" data-lang="java"><span class="o">//</span> <span
class="n">Snippet</span> <span class="n">not</span> <span
class="n">yet</span> <span class="nf">available</span> <span
class="o">(</span><span class="n">BEAM</span><span
class="o">-</span><span class="n">7034</span><span
class="o">).</span></code></pre></div>
+<div class="highlight"><pre class="chroma"><code
class="language-java" data-lang="java"><span class="kn">import</span>
<span
class="nn">org.apache.beam.examples.snippets.transforms.io.gcp.bigquery.BigQueryMyData.MyData</span><span
class="o">;</span>
+<span class="kn">import</span> <span
class="nn">org.apache.beam.sdk.Pipeline</span><span class="o">;</span>
+<span class="kn">import</span> <span
class="nn">org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO</span><span
class="o">;</span>
+<span class="kn">import</span> <span
class="nn">org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method</span><span
class="o">;</span>
+<span class="kn">import</span> <span
class="nn">org.apache.beam.sdk.transforms.MapElements</span><span
class="o">;</span>
+<span class="kn">import</span> <span
class="nn">org.apache.beam.sdk.values.PCollection</span><span
class="o">;</span>
+<span class="kn">import</span> <span
class="nn">org.apache.beam.sdk.values.TypeDescriptor</span><span
class="o">;</span>
+<span class="kd">class</span> <span
class="nc">BigQueryReadFromQueryWithBigQueryStorageAPI</span> <span
class="o">{</span>
+<span class="kd">public</span> <span class="kd">static</span>
<span class="n">PCollection</span><span
class="o">&lt;</span><span class="n">MyData</span><span
class="o">&gt;</span> <span
class="nf">readFromQueryWithBigQueryStorageAPI</span><span
class="o">(</span>
+<span class="n">String</span> <span
class="n">project</span><span class="o">,</span> <span
class="n">String</span> <span class="n">dataset</span><span
class="o">,</span> <span class="n">String</span> <span
class="n">table</span><span class="o">,</span> <span
class="n">String</span> <span class="n">query</span><span
class="o">,</span> <span class="n">Pipeline</span> <span
class="n">pipeline</span><span [...]
+<span class="c1">// String project = &#34;my-project-id&#34;;
+</span><span class="c1"></span> <span class="c1">// String dataset
= &#34;my_bigquery_dataset_id&#34;;
+</span><span class="c1"></span> <span class="c1">// String table =
&#34;my_bigquery_table_id&#34;;
+</span><span class="c1"></span>
+<span class="c1">// Pipeline pipeline = Pipeline.create();
+</span><span class="c1"></span>
+<span class="cm">/*
+</span><span class="cm"> String query =
String.format(&#34;SELECT\n&#34; +
+</span><span class="cm"> &#34; string_field,\n&#34; +
+</span><span class="cm"> &#34; int64_field,\n&#34; +
+</span><span class="cm"> &#34; float64_field,\n&#34; +
+</span><span class="cm"> &#34; numeric_field,\n&#34; +
+</span><span class="cm"> &#34; bool_field,\n&#34; +
+</span><span class="cm"> &#34; bytes_field,\n&#34; +
+</span><span class="cm"> &#34; date_field,\n&#34; +
+</span><span class="cm"> &#34; datetime_field,\n&#34; +
+</span><span class="cm"> &#34; time_field,\n&#34; +
+</span><span class="cm"> &#34; timestamp_field,\n&#34; +
+</span><span class="cm"> &#34; geography_field,\n&#34; +
+</span><span class="cm"> &#34; array_field,\n&#34; +
+</span><span class="cm"> &#34; struct_field\n&#34; +
+</span><span class="cm"> &#34;FROM\n&#34; +
+</span><span class="cm"> &#34; `%s:%s.%s`&#34;, project,
dataset, table)
+</span><span class="cm"> */</span>
+<span class="n">PCollection</span><span
class="o">&lt;</span><span class="n">MyData</span><span
class="o">&gt;</span> <span class="n">rows</span> <span
class="o">=</span>
+<span class="n">pipeline</span>
+<span class="o">.</span><span class="na">apply</span><span
class="o">(</span>
+<span class="s">&#34;Read from BigQuery
table&#34;</span><span class="o">,</span>
+<span class="n">BigQueryIO</span><span class="o">.</span><span
class="na">readTableRows</span><span class="o">()</span>
+<span class="o">.</span><span class="na">fromQuery</span><span
class="o">(</span><span class="n">query</span><span
class="o">)</span>
+<span class="o">.</span><span
class="na">usingStandardSql</span><span class="o">()</span>
+<span class="o">.</span><span class="na">withMethod</span><span
class="o">(</span><span class="n">Method</span><span
class="o">.</span><span class="na">DIRECT_READ</span><span
class="o">))</span>
+<span class="o">.</span><span class="na">apply</span><span
class="o">(</span>
+<span class="s">&#34;TableRows to MyData&#34;</span><span
class="o">,</span>
+<span class="n">MapElements</span><span class="o">.</span><span
class="na">into</span><span class="o">(</span><span
class="n">TypeDescriptor</span><span class="o">.</span><span
class="na">of</span><span class="o">(</span><span
class="n">MyData</span><span class="o">.</span><span
class="na">class</span><span class="o">)).</span><span
class="na">via</span><span class="o">(</span><span
class="n">MyData< [...]
+<span class="k">return</span> <span class="n">rows</span><span
class="o">;</span>
+<span class="o">}</span>
+<span class="o">}</span></code></pre></div>
</div>
<div class=language-py>
<div class="highlight"><pre class="chroma"><code class="language-py"
data-lang="py"><span class="c1"># The SDK for Python does not support the
BigQuery Storage API.</span></code></pre></div>
diff --git
a/website/generated-content/documentation/io/built-in/google-bigquery/index.html
b/website/generated-content/documentation/io/built-in/google-bigquery/index.html
index 36e329f..2bf87cd 100644
---
a/website/generated-content/documentation/io/built-in/google-bigquery/index.html
+++
b/website/generated-content/documentation/io/built-in/google-bigquery/index.html
@@ -305,7 +305,58 @@ GitHub</a>.</p><div class=language-java><div
class=highlight><pre class=chroma><
<span class=k>return</span> <span class=n>rows</span><span class=o>;</span>
<span class=o>}</span>
-<span class=o>}</span></code></pre></div></div><div class=language-py><div
class=highlight><pre class=chroma><code class=language-py data-lang=py><span
class=c1># The SDK for Python does not support the BigQuery Storage
API.</span></code></pre></div></div><p>The following code snippet reads with a
query string.</p><div class=language-java><div class=highlight><pre
class=chroma><code class=language-java data-lang=java><span class=o>//</span>
<span class=n>Snippet</span> <span class=n>not< [...]
+<span class=o>}</span></code></pre></div></div><div class=language-py><div
class=highlight><pre class=chroma><code class=language-py data-lang=py><span
class=c1># The SDK for Python does not support the BigQuery Storage
API.</span></code></pre></div></div><p>The following code snippet reads with a
query string.</p><div class=language-java><div class=highlight><pre
class=chroma><code class=language-java data-lang=java><span
class=kn>import</span> <span class=nn>org.apache.beam.examples.sn [...]
+<span class=kn>import</span> <span
class=nn>org.apache.beam.sdk.Pipeline</span><span class=o>;</span>
+<span class=kn>import</span> <span
class=nn>org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO</span><span
class=o>;</span>
+<span class=kn>import</span> <span
class=nn>org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method</span><span
class=o>;</span>
+<span class=kn>import</span> <span
class=nn>org.apache.beam.sdk.transforms.MapElements</span><span class=o>;</span>
+<span class=kn>import</span> <span
class=nn>org.apache.beam.sdk.values.PCollection</span><span class=o>;</span>
+<span class=kn>import</span> <span
class=nn>org.apache.beam.sdk.values.TypeDescriptor</span><span class=o>;</span>
+
+<span class=kd>class</span> <span
class=nc>BigQueryReadFromQueryWithBigQueryStorageAPI</span> <span
class=o>{</span>
+ <span class=kd>public</span> <span class=kd>static</span> <span
class=n>PCollection</span><span class=o><</span><span
class=n>MyData</span><span class=o>></span> <span
class=nf>readFromQueryWithBigQueryStorageAPI</span><span class=o>(</span>
+ <span class=n>String</span> <span class=n>project</span><span
class=o>,</span> <span class=n>String</span> <span class=n>dataset</span><span
class=o>,</span> <span class=n>String</span> <span class=n>table</span><span
class=o>,</span> <span class=n>String</span> <span class=n>query</span><span
class=o>,</span> <span class=n>Pipeline</span> <span
class=n>pipeline</span><span class=o>)</span> <span class=o>{</span>
+
+ <span class=c1>// String project = "my-project-id";
+</span><span class=c1></span> <span class=c1>// String dataset =
"my_bigquery_dataset_id";
+</span><span class=c1></span> <span class=c1>// String table =
"my_bigquery_table_id";
+</span><span class=c1></span>
+ <span class=c1>// Pipeline pipeline = Pipeline.create();
+</span><span class=c1></span>
+ <span class=cm>/*
+</span><span class=cm> String query = String.format("SELECT\n" +
+</span><span class=cm> " string_field,\n" +
+</span><span class=cm> " int64_field,\n" +
+</span><span class=cm> " float64_field,\n" +
+</span><span class=cm> " numeric_field,\n" +
+</span><span class=cm> " bool_field,\n" +
+</span><span class=cm> " bytes_field,\n" +
+</span><span class=cm> " date_field,\n" +
+</span><span class=cm> " datetime_field,\n" +
+</span><span class=cm> " time_field,\n" +
+</span><span class=cm> " timestamp_field,\n" +
+</span><span class=cm> " geography_field,\n" +
+</span><span class=cm> " array_field,\n" +
+</span><span class=cm> " struct_field\n" +
+</span><span class=cm> "FROM\n" +
+</span><span class=cm> " `%s:%s.%s`", project, dataset, table)
+</span><span class=cm> */</span>
+
+ <span class=n>PCollection</span><span class=o><</span><span
class=n>MyData</span><span class=o>></span> <span class=n>rows</span> <span
class=o>=</span>
+ <span class=n>pipeline</span>
+ <span class=o>.</span><span class=na>apply</span><span
class=o>(</span>
+ <span class=s>"Read from BigQuery table"</span><span
class=o>,</span>
+ <span class=n>BigQueryIO</span><span class=o>.</span><span
class=na>readTableRows</span><span class=o>()</span>
+ <span class=o>.</span><span class=na>fromQuery</span><span
class=o>(</span><span class=n>query</span><span class=o>)</span>
+ <span class=o>.</span><span
class=na>usingStandardSql</span><span class=o>()</span>
+ <span class=o>.</span><span
class=na>withMethod</span><span class=o>(</span><span
class=n>Method</span><span class=o>.</span><span
class=na>DIRECT_READ</span><span class=o>))</span>
+ <span class=o>.</span><span class=na>apply</span><span
class=o>(</span>
+ <span class=s>"TableRows to MyData"</span><span
class=o>,</span>
+ <span class=n>MapElements</span><span class=o>.</span><span
class=na>into</span><span class=o>(</span><span
class=n>TypeDescriptor</span><span class=o>.</span><span
class=na>of</span><span class=o>(</span><span class=n>MyData</span><span
class=o>.</span><span class=na>class</span><span class=o>)).</span><span
class=na>via</span><span class=o>(</span><span class=n>MyData</span><span
class=o>::</span><span class=n>fromTableRow</span><span class=o>));</span>
+
+ <span class=k>return</span> <span class=n>rows</span><span class=o>;</span>
+ <span class=o>}</span>
+<span class=o>}</span></code></pre></div></div><div class=language-py><div
class=highlight><pre class=chroma><code class=language-py data-lang=py><span
class=c1># The SDK for Python does not support the BigQuery Storage
API.</span></code></pre></div></div><h2 id=writing-to-bigquery>Writing to
BigQuery</h2><p>BigQueryIO allows you to write to BigQuery tables. If you are
using the Beam SDK
for Java, you can also write different rows to different
tables.</p><blockquote><p>BigQueryIO write transforms use APIs that are subject
to BigQuery’s
<a href=https://cloud.google.com/bigquery/quota-policy>Quota</a> and
<a href=https://cloud.google.com/bigquery/pricing>Pricing</a>
policies.</p></blockquote><p>When you apply a write transform, you must provide
the following information
