This is an automated email from the ASF dual-hosted git repository.
altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 27f0dba [BEAM-9770] BigQueryIO Patterns. Add Document update to
snippit update.
new 1a5dd77 Merge pull request #11815 from rezarokni/BEAM-9770
27f0dba is described below
commit 27f0dbae8080b93801ce56ef208470e41673062a
Author: rarokni <raro...@users.noreply.github.com>
AuthorDate: Tue May 26 17:23:44 2020 +0800
[BEAM-9770] BigQueryIO Patterns. Add Document update to snippit update.
---
.../en/documentation/patterns/bigqueryio.md | 46 ++++++++++++++++++++++
.../content/en/documentation/patterns/overview.md | 3 ++
.../partials/section-menu/en/documentation.html | 1 +
3 files changed, 50 insertions(+)
diff --git a/website/www/site/content/en/documentation/patterns/bigqueryio.md
b/website/www/site/content/en/documentation/patterns/bigqueryio.md
new file mode 100644
index 0000000..49393b8
--- /dev/null
+++ b/website/www/site/content/en/documentation/patterns/bigqueryio.md
@@ -0,0 +1,46 @@
+---
+layout: section
+title: "BigQuery patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/bigqueryio/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Google BigQuery patterns
+
+The samples on this page show you common patterns for use with BigQueryIO.
+
+{{< language-switcher java py >}}
+
+## BigQueryIO deadletter pattern
+In production systems, it is useful to implement the deadletter pattern with
BigQueryIO outputting any elements which had errors during processing by
BigQueryIO into another PCollection for further processing.
+The samples below print the errors, but in a production system they can be
sent to a deadletter table for later correction.
+
+{{< paragraph class="language-java" >}}
+When using `STREAMING_INSERTS` you can use the `WriteResult` object to access
a `PCollection` with the `TableRows` that failed to be inserted into BigQuery.
+If you also set the `withExtendedErrorInfo` property , you will be able to
access a `PCollection<BigQueryInsertError>` from the `WriteResult`. The
`PCollection` will then include a reference to the table, the data row and the
`InsertErrors`. Which errors are added to the deadletter queue is determined
via the `InsertRetryPolicy`.
+{{< /paragraph >}}
+
+{{< paragraph class="language-py" >}}
+In the result tuple you can access `FailedRows` to access the failed inserts.
+{{< /paragraph >}}
+
+{{< highlight java >}}
+{{< github_sample
"/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java"
BigQueryIODeadLetter >}}
+{{< /highlight >}}
+
+{{< highlight py >}}
+{{< github_sample
"/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py"
BigQueryIODeadLetter >}}
+{{< /highlight >}}
diff --git a/website/www/site/content/en/documentation/patterns/overview.md
b/website/www/site/content/en/documentation/patterns/overview.md
index a610c3c..561c8f0 100644
--- a/website/www/site/content/en/documentation/patterns/overview.md
+++ b/website/www/site/content/en/documentation/patterns/overview.md
@@ -35,6 +35,9 @@ Pipeline patterns demonstrate common Beam use cases. Pipeline
patterns are based
**Custom window patterns** - Patterns for windowing functions
* [Using data to dynamically set session window
gaps](/documentation/patterns/custom-windows/#using-data-to-dynamically-set-session-window-gaps)
+**BigQuery patterns** - Patterns for BigQueryIO
+* [Google BigQuery
patterns](/documentation/patterns/bigqueryio/#google-bigquery-patterns)
+
## Contributing a pattern
To contribute a new pipeline pattern, create an issue with the
[`pipeline-patterns`
label](https://issues.apache.org/jira/browse/BEAM-7449?jql=labels%20%3D%20pipeline-patterns)
and add details to the issue description. See [Get started
contributing](/contribute/) for more information.
diff --git
a/website/www/site/layouts/partials/section-menu/en/documentation.html
b/website/www/site/layouts/partials/section-menu/en/documentation.html
index a44e045..9baa95c 100644
--- a/website/www/site/layouts/partials/section-menu/en/documentation.html
+++ b/website/www/site/layouts/partials/section-menu/en/documentation.html
@@ -276,6 +276,7 @@
<li><a href="/documentation/patterns/pipeline-options/">Pipeline
options</a></li>
<li><a href="/documentation/patterns/custom-io/">Custom I/O</a></li>
<li><a href="/documentation/patterns/custom-windows/">Custom
windows</a></li>
+ <li><a href="/documentation/patterns/bigqueryio/">BigQueryIO</a></li>
</ul>
</li>
