This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 9bfa8b7 Publishing website 2020/10/24 00:02:43 at commit 56e9ce2
9bfa8b7 is described below
commit 9bfa8b7fe3063731d0466646dd8152a15be45d10
Author: jenkins <[email protected]>
AuthorDate: Sat Oct 24 00:02:43 2020 +0000
Publishing website 2020/10/24 00:02:43 at commit 56e9ce2
---
website/generated-content/beam-2.25.0.md | 85 ++++++++++++++++++++++
website/generated-content/documentation/index.xml | 34 ++++-----
.../documentation/programming-guide/index.html | 36 ++++-----
website/generated-content/sitemap.xml | 2 +-
4 files changed, 121 insertions(+), 36 deletions(-)
diff --git a/website/generated-content/beam-2.25.0.md
b/website/generated-content/beam-2.25.0.md
new file mode 100644
index 0000000..d757147
--- /dev/null
+++ b/website/generated-content/beam-2.25.0.md
@@ -0,0 +1,85 @@
+---
+title: "Apache Beam 2.25.0"
+date: 2020-10-23 14:00:00 -0800
+categories:
+ - blog
+authors:
+ - Robin Qiu
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+We are happy to present the new 2.25.0 release of Apache Beam. This release
includes both improvements and new functionality.
+See the [download page](/get-started/downloads/#2250-2020-10-23) for this
release.
+For more information on changes in 2.25.0, check out the
+[detailed release
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12347147).
+
+## Highlights
+
+* Splittable DoFn is now the default for executing the Read transform for Java
based runners (Direct, Flink, Jet, Samza, Twister2). The expected output of the
Read transform is unchanged. Users can opt-out using
`--experiments=use_deprecated_read`. The Apache Beam community is looking for
feedback for this change as the community is planning to make this change
permanent with no opt-out. If you run into an issue requiring the opt-out,
please send an e-mail to [[email protected]](mailt [...]
+
+## I/Os
+
+* Added cross-language support to Java's KinesisIO, now available in the
Python module `apache_beam.io.kinesis`
([BEAM-10138](https://issues.apache.org/jira/browse/BEAM-10138),
[BEAM-10137](https://issues.apache.org/jira/browse/BEAM-10137)).
+* Update Snowflake JDBC dependency for SnowflakeIO
([BEAM-10864](https://issues.apache.org/jira/browse/BEAM-10864))
+* Added cross-language support to Java's SnowflakeIO.Write, now available in
the Python module `apache_beam.io.snowflake`
([BEAM-9898](https://issues.apache.org/jira/browse/BEAM-9898)).
+* Added delete function to Java's `ElasticsearchIO#Write`. Now, Java's
ElasticsearchIO can be used to selectively delete documents using
`withIsDeleteFn` function
([BEAM-5757](https://issues.apache.org/jira/browse/BEAM-5757)).
+* Java SDK: Added new IO connector for InfluxDB - InfluxDbIO
([BEAM-2546](https://issues.apache.org/jira/browse/BEAM-2546)).
+
+## New Features / Improvements
+
+* Support for repeatable fields in JSON decoder for `ReadFromBigQuery` added.
(Python) ([BEAM-10524](https://issues.apache.org/jira/browse/BEAM-10524))
+* Added an opt-in, performance-driven runtime type checking system for the
Python SDK ([BEAM-10549](https://issues.apache.org/jira/browse/BEAM-10549)).
+ More details will be in an upcoming [blog
post](https://beam.apache.org/blog/python-performance-runtime-type-checking/index.html).
+* Added support for Python 3 type annotations on PTransforms using typed
PCollections ([BEAM-10258](https://issues.apache.org/jira/browse/BEAM-10258)).
+ More details will be in an upcoming [blog
post](https://beam.apache.org/blog/python-improved-annotations/index.html).
+* Improved the Interactive Beam API where recording streaming jobs now start a
long running background recording job. Running ib.show() or ib.collect()
samples from the recording
([BEAM-10603](https://issues.apache.org/jira/browse/BEAM-10603)).
+* In Interactive Beam, ib.show() and ib.collect() now have "n" and "duration"
as parameters. These mean read only up to "n" elements and up to "duration"
seconds of data read from the recording
([BEAM-10603](https://issues.apache.org/jira/browse/BEAM-10603)).
+* Initial preview of
[Dataframes](https://s.apache.org/simpler-python-pipelines-2020#slide=id.g905ac9257b_1_21)
support.
+ See also example at apache_beam/examples/wordcount_dataframe.py
+* Fixed support for type hints on `@ptransform_fn` decorators in the Python
SDK.
+ ([BEAM-4091](https://issues.apache.org/jira/browse/BEAM-4091))
+ This has not enabled by default to preserve backwards compatibility; use the
+ `--type_check_additional=ptransform_fn` flag to enable. It may be enabled by
+ default in future versions of Beam.
+
+## Breaking Changes
+
+* Python 2 and Python 3.5 support dropped
([BEAM-10644](https://issues.apache.org/jira/browse/BEAM-10644),
[BEAM-9372](https://issues.apache.org/jira/browse/BEAM-9372)).
+* Pandas 1.x allowed. Older version of Pandas may still be used, but may not
be as well tested.
+
+## Deprecations
+
+* Python transform ReadFromSnowflake has been moved from
`apache_beam.io.external.snowflake` to `apache_beam.io.snowflake`. The previous
path will be removed in the future versions.
+
+## Known Issues
+
+* Dataflow streaming timers once against not strictly time ordered when set
earlier mid-bundle, as the fix for
[BEAM-8543](https://issues.apache.org/jira/browse/BEAM-8543) introduced more
severe bugs and has been rolled back.
+
+
+## List of Contributors
+
+According to git shortlog, the following people contributed to the 2.25.0
release. Thank you to all contributors!
+
+Ahmet Altay, Alan Myrvold, Aldair Coronel Ruiz, Alexey Romanenko, Andrew
Pilloud, Ankur Goenka,
+Ayoub ENNASSIRI, Bipin Upadhyaya, Boyuan Zhang, Brian Hulette, Brian
Michalski, Chad Dombrova,
+Chamikara Jayalath, Damon Douglas, Daniel Oliveira, David Cavazos, David
Janicek, Doug Roeper, Eric
+Roshan-Eisner, Etta Rapp, Eugene Kirpichov, Filipe Regadas, Heejong Lee, Ihor
Indyk, Irvi Firqotul
+Aini, Ismaël Mejía, Jan Lukavský, Jayendra, Jiadai Xia, Jithin Sukumar, Jozsef
Bartok, Kamil
+Gałuszka, Kamil Wasilewski, Kasia Kucharczyk, Kenneth Jung, Kenneth Knowles,
Kevin Puthusseri, Kevin
+Sijo Puthusseri, KevinGG, Kyle Weaver, Leiyi Zhang, Lourens Naudé, Luke Cwik,
Matthew Ouyang,
+Maximilian Michels, Michal Walenia, Milan Cermak, Monica Song, Nelson Osacky,
Neville Li, Ning Kang,
+Pablo Estrada, Piotr Szuberski, Qihang, Rehman, Reuven Lax, Robert Bradshaw,
Robert Burke, Rui Wang,
+Saavan Nanavati, Sam Bourne, Sam Rohde, Sam Whittle, Sergiy Kolesnikov, Sindy
Li, Siyuan Chen, Steve
+Niemitz, Terry Xian, Thomas Weise, Tobiasz Kędzierski, Truc Le, Tyson
Hamilton, Udi Meiri, Valentyn
+Tymofieiev, Yichi Zhang, Yifan Mai, Yueyang Qiu, annaqin418, danielxjd,
dennis, dp, fuyuwei,
+lostluck, nehsyc, odeshpande, odidev, pulasthi, purbanow, rworley-monster,
sclukas77, terryxian78,
+tvalentyn, yoshiki.obata
diff --git a/website/generated-content/documentation/index.xml
b/website/generated-content/documentation/index.xml
index 0ffa63e..837871e 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -6293,7 +6293,7 @@ potentially while the pipeline is running. There could be
different reasons for
<li>Retrieve an accurate count of the number of elements that have been
processed;</li>
<li>&hellip;and so on.</li>
</ul>
-<h3 id="101-the-main-concepts-of-beam-metrics">10.1 The main concepts of
Beam metrics</h3>
+<h3 id="101-the-main-concepts-of-beam-metrics">10.1. The main concepts of
Beam metrics</h3>
<ul>
<li><strong>Named</strong>. Each metric has a name which consists of
a namespace and an actual name. The
namespace can be used to differentiate between multiple metrics with the same
name and also
@@ -6314,7 +6314,7 @@ transform reported, as well as aggregating the metric
across the entire pipeline
<p><strong>Note:</strong> It is runner-dependent whether metrics are
accessible during pipeline execution or only
after jobs have completed.</p>
</blockquote>
-<h3 id="types-of-metrics">10.2 Types of metrics</h3>
+<h3 id="types-of-metrics">10.2. Types of metrics</h3>
<p>There are three types of metrics that are supported for the moment:
<code>Counter</code>, <code>Distribution</code> and
<code>Gauge</code>.</p>
<p><strong>Counter</strong>: A metric that reports a single long
value and can be incremented or decremented.</p>
@@ -6350,7 +6350,7 @@ collected from many workers the value may not be the
absolute last, but one of t
<span class="o">...</span>
<span class="o">}</span></code></pre></div>
</div>
-<h3 id="querying-metrics">10.3 Querying metrics</h3>
+<h3 id="querying-metrics">10.3. Querying metrics</h3>
<p><code>PipelineResult</code> has a method
<code>metrics()</code> which returns a <code>MetricResults</code>
object that allows
accessing metrics. The main method available in
<code>MetricResults</code> allows querying for all metrics
matching a given filter.</p>
@@ -6373,7 +6373,7 @@ matching a given filter.</p>
<span class="n">T</span> <span
class="nf">getAttempted</span><span class="o">();</span>
<span class="o">}</span></code></pre></div>
</div>
-<h3 id="using-metrics">10.4 Using metrics in pipeline</h3>
+<h3 id="using-metrics">10.4. Using metrics in pipeline</h3>
<p>Below, there is a simple example of how to use a
<code>Counter</code> metric in a user pipeline.</p>
<div class=language-java>
<div class="highlight"><pre class="chroma"><code
class="language-java" data-lang="java"><span class="c1">// creating a
pipeline with custom metrics DoFn
@@ -6404,7 +6404,7 @@ matching a given filter.</p>
<span class="o">}</span>
<span class="o">}</span></code></pre></div>
</div>
-<h3 id="export-metrics">10.5 Export metrics</h3>
+<h3 id="export-metrics">10.5. Export metrics</h3>
<p>Beam metrics can be exported to external sinks. If a metrics sink is set
up in the configuration, the runner will push metrics to it at a default 5s
period.
The configuration is held in the <a
href="https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html";>MetricsOptions</a>
class.
It contains push period configuration and also sink specific options such as
type and URL. As for now only the REST HTTP and the Graphite sinks are
supported and only
@@ -6434,7 +6434,7 @@ to other nodes in the graph. A <code>DoFn</code>
can declare multiple stat
<p class="language-py">In Python DoFn declares states to be accessed by
creating <code>StateSpec</code> class member variables representing each
state. Each
<code>StateSpec</code> is initialized with a name, this name is unique
to a ParDo in the graph and has no relation
to other nodes in the graph. A <code>DoFn</code> can declare multiple
state variables.</p>
-<h3 id="types-of-state">11.1 Types of state</h3>
+<h3 id="types-of-state">11.1. Types of state</h3>
<p>Beam provides several types of state:</p>
<h4 id="valuestate">ValueState</h4>
<p>A ValueState is a scalar state value. For each key in the input, a
ValueState will store a typed value that can be
@@ -6519,7 +6519,7 @@ bags larger than available memory.</p>
<span class="n">_</span> <span class="o">=</span> <span
class="p">(</span><span class="n">p</span> <span
class="o">|</span> <span class="s1">&#39;Read per
user&#39;</span> <span class="o">&gt;&gt;</span> <span
class="n">ReadPerUser</span><span class="p">()</span>
<span class="o">|</span> <span class="s1">&#39;Bag state
pardo&#39;</span> <span class="o">&gt;&gt;</span> <span
class="n">beam</span><span class="o">.</span><span
class="n">ParDo</span><span class="p">(</span><span
class="n">BagStateDoFn</span><span
class="p">()))</span></code></pre></div>
</div>
-<h3 id="deferred-state-reads">11.2 Deferred state reads</h3>
+<h3 id="deferred-state-reads">11.2. Deferred state reads</h3>
<p>When a <code>DoFn</code> contains multiple state specifications,
reading each one in order can be slow. Calling the <code>read()</code>
function
on a state can cause the runner to perform a blocking read. Performing
multiple blocking reads in sequence adds latency
to element processing. If you know that a state will always be read, you can
annotate it as @AlwaysFetched, and then the
@@ -6565,12 +6565,12 @@ be read in the future, allowing multiple state reads to
be batched together.<
<span class="o">}</span>
<span class="o">}));</span></code></pre></div>
</div>
-<h3 id="timers">11.3 Timers</h3>
+<h3 id="timers">11.3. Timers</h3>
<p>Beam provides a per-key timer callback API. This allows for delayed
processing of data stored using the state API.
Timers can be set to callback at either an event-time or a processing-time
timestamp. Every timer is identified with a
TimerId. A given timer for a key can only be set for a single timestamp.
Calling set on a timer overwrites the previous
firing time for that key&rsquo;s timer.</p>
-<h4 id="event-time-timers">11.3.1 Event-time timers</h4>
+<h4 id="event-time-timers">11.3.1. Event-time timers</h4>
<p>Event-time timers fire when the input watermark for the DoFn passes the
time at which the timer is set, meaning that
the runner believes that there are no more elements to be processed with
timestamps before the timer timestamp. This
allows for event-time aggregations.</p>
@@ -6611,7 +6611,7 @@ allows for event-time aggregations.</p>
<span class="n">_</span> <span class="o">=</span> <span
class="p">(</span><span class="n">p</span> <span
class="o">|</span> <span class="s1">&#39;Read per
user&#39;</span> <span class="o">&gt;&gt;</span> <span
class="n">ReadPerUser</span><span class="p">()</span>
<span class="o">|</span> <span class="s1">&#39;EventTime timer
pardo&#39;</span> <span class="o">&gt;&gt;</span> <span
class="n">beam</span><span class="o">.</span><span
class="n">ParDo</span><span class="p">(</span><span
class="n">EventTimerDoFn</span><span
class="p">()))</span></code></pre></div>
</div>
-<h4 id="processing-time-timers">11.3.2 Processing-time timers</h4>
+<h4 id="processing-time-timers">11.3.2. Processing-time timers</h4>
<p>Processing-time timers fire when the real wall-clock time passes. This
is often used to create larger batches of data
before processing. It can also be used to schedule events that should occur at
a specific time. Just like with
event-time timers, processing-time timers are per key - each key has a
separate copy of the timer.</p>
@@ -6649,7 +6649,7 @@ to the current time. In Java, the
<code>Timer.offset</code> and <code>T
<span class="n">_</span> <span class="o">=</span> <span
class="p">(</span><span class="n">p</span> <span
class="o">|</span> <span class="s1">&#39;Read per
user&#39;</span> <span class="o">&gt;&gt;</span> <span
class="n">ReadPerUser</span><span class="p">()</span>
<span class="o">|</span> <span class="s1">&#39;ProcessingTime
timer pardo&#39;</span> <span class="o">&gt;&gt;</span>
<span class="n">beam</span><span class="o">.</span><span
class="n">ParDo</span><span class="p">(</span><span
class="n">ProcessingTimerDoFn</span><span
class="p">()))</span></code></pre></div>
</div>
-<h4 id="dynamic-timer-tags">11.3.3 Dynamic timer tags</h4>
+<h4 id="dynamic-timer-tags">11.3.3. Dynamic timer tags</h4>
<p>Beam also supports dynamically setting a timer tag using
<code>TimerMap</code>. This allows for setting multiple different timers
in a <code>DoFn</code> and allowing for the timer tags to be dynamically
chosen - e.g. based on data in the input elements. A
timer with a specific tag can only be set to a single timestamp, so setting
the timer again has the effect of
@@ -6674,7 +6674,7 @@ id, and timers in different timer families are
independent.</p>
<div class=language-python>
<div class="highlight"><pre class="chroma"><code
class="language-python" data-lang="python"><span class="n">To</span>
<span class="n">be</span> <span class="n">supported</span><span
class="p">,</span> <span class="n">See</span> <span
class="n">BEAM</span><span class="o">-</span><span
class="mi">9602</span></code></pre></div>
</div>
-<h4 id="timer-output-timestamps">11.3.4 Timer output timestamps</h4>
+<h4 id="timer-output-timestamps">11.3.4. Timer output timestamps</h4>
<p>By default, event-time timers will hold the output watermark of the
<code>ParDo</code> to the timestamp of the timer. This means
that if a timer is set to 12pm, any windowed aggregations or event-time timers
later in the pipeline graph that finish
after 12pm will not expire. The timestamp of the timer is also the default
output timestamp for the timer callback. This
@@ -6765,10 +6765,10 @@ past the timestamp of the minimum element. The
following code demonstrates this.
<span class="o">}</span>
<span class="o">}));</span></code></pre></div>
</div>
-<h3 id="garbage-collecting-state">11.4 Garbage collecting state</h3>
+<h3 id="garbage-collecting-state">11.4. Garbage collecting state</h3>
<p>Per-key state needs to be garbage collected, or eventually the
increasing size of state may negatively impact
performance. There are two common strategies for garbage collecting
state.</p>
-<h5 id="using-windows-for-garbage-collection">11.4.1 <strong>Using
windows for garbage collection</strong></h5>
+<h5 id="using-windows-for-garbage-collection">11.4.1. <strong>Using
windows for garbage collection</strong></h5>
<p>All state and timers for a key is scoped to the window it is in. This
means that depending on the timestamp of the
input element the ParDo will see different values for the state depending on
the window that element falls into. In
addition, once the input watermark passes the end of the window, the runner
should garbage collect all state for that
@@ -6802,7 +6802,7 @@ garbage-collection strategy.</p>
</div>
<p>This <code>ParDo</code> stores state per day. Once the pipeline is
done processing data for a given day, all the state for that
day is garbage collected.</p>
-<h5 id="using-timers-for-garbage-collection">11.4.1 <strong>Using timers
For garbage collection</strong></h5>
+<h5 id="using-timers-for-garbage-collection">11.4.1. <strong>Using
timers For garbage collection</strong></h5>
<p>In some cases, it is difficult to find a windowing strategy that models
the desired garbage-collection strategy. For
example, a common desire is to garbage collect state for a key once no
activity has been seen on the key for some time.
This can be done by updating a timer that garbage collects state. For
example</p>
@@ -6865,7 +6865,7 @@ This can be done by updating a timer that garbage
collects state. For example<
<span class="n">_</span> <span class="o">=</span> <span
class="p">(</span><span class="n">p</span> <span
class="o">|</span> <span class="s1">&#39;Read per
user&#39;</span> <span class="o">&gt;&gt;</span> <span
class="n">ReadPerUser</span><span class="p">()</span>
<span class="o">|</span> <span class="s1">&#39;User
DoFn&#39;</span> <span class="o">&gt;&gt;</span> <span
class="n">beam</span><span class="o">.</span><span
class="n">ParDo</span><span class="p">(</span><span
class="n">UserDoFn</span><span
class="p">()))</span></code></pre></div>
</div>
-<h3 id="state-timers-examples">11.5 State and timers examples</h3>
+<h3 id="state-timers-examples">11.5. State and timers examples</h3>
<p>Following are some example uses of state and timers</p>
<h4 id="joining-clicks-and-views">11.5.1. Joining clicks and views</h4>
<p>In this example, the pipeline is processing data from an e-commerce
site&rsquo;s home page. There are two input streams:
@@ -6938,7 +6938,7 @@ event before the view event. The one hour join timeout
should be based on event
<span class="o">}</span>
<span class="o">}));</span></code></pre></div>
</div>
-<h4 id="batching-rpcs">11.5.2 Batching RPCs</h4>
+<h4 id="batching-rpcs">11.5.2. Batching RPCs</h4>
<p>In this example, input elements are being forwarded to an external RPC
service. The RPC accepts batch requests -
multiple events for the same user can be batched in a single RPC call. Since
this RPC service also imposes rate limits,
we want to batch ten seconds worth of events together in order to reduce the
number of calls.</p>
diff --git
a/website/generated-content/documentation/programming-guide/index.html
b/website/generated-content/documentation/programming-guide/index.html
index 58831b2..fceb292 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -1,7 +1,7 @@
<!doctype html><html lang=en class=no-js><head><meta charset=utf-8><meta
http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport
content="width=device-width,initial-scale=1"><title>Beam Programming
Guide</title><meta name=description content="Apache Beam is an open source,
unified model and set of language-specific SDKs for defining and executing data
processing workflows, and also data ingestion and integration flows, supporting
Enterprise Integration Patterns (EIPs) and Domain [...]
<span class=sr-only>Toggle navigation</span>
<span class=icon-bar></span><span class=icon-bar></span><span
class=icon-bar></span></button>
-<a href=/ class=navbar-brand><img alt=Brand style=height:25px
src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask
closed"></div><div id=navbar class="navbar-container closed"><ul class="nav
navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a
href=/documentation/>Documentation</a></li><li><a
href=/documentation/sdks/java/>Languages</a></li><li><a
href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a
href=/roadmap/>Roadmap</a></li>< [...]
+<a href=/ class=navbar-brand><img alt=Brand style=height:25px
src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask
closed"></div><div id=navbar class="navbar-container closed"><ul class="nav
navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a
href=/documentation/>Documentation</a></li><li><a
href=/documentation/sdks/java/>Languages</a></li><li><a
href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a
href=/roadmap/>Roadmap</a></li>< [...]
Beam SDKs to create data processing pipelines. It provides guidance for using
the Beam SDK classes to build and test your pipeline. It is not intended as an
exhaustive reference, but as a language-agnostic, high-level guide to
@@ -2166,7 +2166,7 @@ elements, or after a minute.</p><div
class=language-java><div class=highlight><p
<span class=n>trigger</span><span class=o>=</span><span
class=n>Repeatedly</span><span class=p>(</span>
<span class=n>AfterAny</span><span class=p>(</span><span
class=n>AfterCount</span><span class=p>(</span><span class=mi>100</span><span
class=p>),</span> <span class=n>AfterProcessingTime</span><span
class=p>(</span><span class=mi>1</span> <span class=o>*</span> <span
class=mi>60</span><span class=p>))),</span>
<span class=n>accumulation_mode</span><span class=o>=</span><span
class=n>AccumulationMode</span><span class=o>.</span><span
class=n>DISCARDING</span><span class=p>)</span></code></pre></div></div><h2
id=metrics>10. Metrics</h2><p>In the Beam model, metrics provide some insight
into the current state of a user pipeline,
-potentially while the pipeline is running. There could be different reasons
for that, for instance:</p><ul><li>Check the number of errors encountered while
running a specific step in the pipeline;</li><li>Monitor the number of RPCs
made to backend service;</li><li>Retrieve an accurate count of the number of
elements that have been processed;</li><li>…and so on.</li></ul><h3
id=101-the-main-concepts-of-beam-metrics>10.1 The main concepts of Beam
metrics</h3><ul><li><strong>Named</str [...]
+potentially while the pipeline is running. There could be different reasons
for that, for instance:</p><ul><li>Check the number of errors encountered while
running a specific step in the pipeline;</li><li>Monitor the number of RPCs
made to backend service;</li><li>Retrieve an accurate count of the number of
elements that have been processed;</li><li>…and so on.</li></ul><h3
id=101-the-main-concepts-of-beam-metrics>10.1. The main concepts of Beam
metrics</h3><ul><li><strong>Named</st [...]
namespace can be used to differentiate between multiple metrics with the same
name and also
allows querying for all metrics within a specific
namespace.</li><li><strong>Scoped</strong>. Each metric is reported against a
specific step in the pipeline, indicating what
code was running when the metric was incremented.</li><li><strong>Dynamically
Created</strong>. Metrics may be created during runtime without pre-declaring
them, in
@@ -2176,7 +2176,7 @@ fallback behavior is to drop the metric updates rather
than failing the pipeline
doesn’t support some part of querying metrics, the runner will not return the
associated data.</li></ul><p>Reported metrics are implicitly scoped to the
transform within the pipeline that reported them.
This allows reporting the same metric name in multiple places and identifying
the value each
transform reported, as well as aggregating the metric across the entire
pipeline.</p><blockquote><p><strong>Note:</strong> It is runner-dependent
whether metrics are accessible during pipeline execution or only
-after jobs have completed.</p></blockquote><h3 id=types-of-metrics>10.2 Types
of metrics</h3><p>There are three types of metrics that are supported for the
moment: <code>Counter</code>, <code>Distribution</code> and
+after jobs have completed.</p></blockquote><h3 id=types-of-metrics>10.2. Types
of metrics</h3><p>There are three types of metrics that are supported for the
moment: <code>Counter</code>, <code>Distribution</code> and
<code>Gauge</code>.</p><p><strong>Counter</strong>: A metric that reports a
single long value and can be incremented or decremented.</p><div
class=language-java><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>Counter</span> <span
class=n>counter</span> <span class=o>=</span> <span class=n>Metrics</span><span
class=o>.</span><span class=na>counter</span><span class=o>(</span> <span
class=s>"namespace"</span><span class=o>,</span> <span [...]
<span class=nd>@ProcessElement</span>
@@ -2201,7 +2201,7 @@ collected from many workers the value may not be the
absolute last, but one of t
<span class=c1>// create a gauge (latest value received) of the values
</span><span class=c1></span> <span class=n>gauge</span><span
class=o>.</span><span class=na>set</span><span class=o>(</span><span
class=n>element</span><span class=o>);</span>
<span class=o>...</span>
-<span class=o>}</span></code></pre></div></div><h3 id=querying-metrics>10.3
Querying metrics</h3><p><code>PipelineResult</code> has a method
<code>metrics()</code> which returns a <code>MetricResults</code> object that
allows
+<span class=o>}</span></code></pre></div></div><h3 id=querying-metrics>10.3.
Querying metrics</h3><p><code>PipelineResult</code> has a method
<code>metrics()</code> which returns a <code>MetricResults</code> object that
allows
accessing metrics. The main method available in <code>MetricResults</code>
allows querying for all metrics
matching a given filter.</p><div class=language-java><div class=highlight><pre
class=chroma><code class=language-java data-lang=java><span
class=kd>public</span> <span class=kd>interface</span> <span
class=nc>PipelineResult</span> <span class=o>{</span>
<span class=n>MetricResults</span> <span class=nf>metrics</span><span
class=o>();</span>
@@ -2222,7 +2222,7 @@ matching a given filter.</p><div class=language-java><div
class=highlight><pre c
<span class=n>String</span> <span class=nf>getStep</span><span
class=o>();</span>
<span class=n>T</span> <span class=nf>getCommitted</span><span
class=o>();</span>
<span class=n>T</span> <span class=nf>getAttempted</span><span
class=o>();</span>
-<span class=o>}</span></code></pre></div></div><h3 id=using-metrics>10.4 Using
metrics in pipeline</h3><p>Below, there is a simple example of how to use a
<code>Counter</code> metric in a user pipeline.</p><div
class=language-java><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=c1>// creating a pipeline with
custom metrics DoFn
+<span class=o>}</span></code></pre></div></div><h3 id=using-metrics>10.4.
Using metrics in pipeline</h3><p>Below, there is a simple example of how to use
a <code>Counter</code> metric in a user pipeline.</p><div
class=language-java><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=c1>// creating a pipeline with
custom metrics DoFn
</span><span class=c1></span><span class=n>pipeline</span>
<span class=o>.</span><span class=na>apply</span><span class=o>(...)</span>
<span class=o>.</span><span class=na>apply</span><span
class=o>(</span><span class=n>ParDo</span><span class=o>.</span><span
class=na>of</span><span class=o>(</span><span class=k>new</span> <span
class=n>MyMetricsDoFn</span><span class=o>()));</span>
@@ -2253,7 +2253,7 @@ matching a given filter.</p><div class=language-java><div
class=highlight><pre c
</span><span class=c1></span> <span class=n>counter</span><span
class=o>.</span><span class=na>inc</span><span class=o>();</span>
<span class=n>context</span><span class=o>.</span><span
class=na>output</span><span class=o>(</span><span class=n>context</span><span
class=o>.</span><span class=na>element</span><span class=o>());</span>
<span class=o>}</span>
-<span class=o>}</span></code></pre></div></div><h3 id=export-metrics>10.5
Export metrics</h3><p>Beam metrics can be exported to external sinks. If a
metrics sink is set up in the configuration, the runner will push metrics to it
at a default 5s period.
+<span class=o>}</span></code></pre></div></div><h3 id=export-metrics>10.5.
Export metrics</h3><p>Beam metrics can be exported to external sinks. If a
metrics sink is set up in the configuration, the runner will push metrics to it
at a default 5s period.
The configuration is held in the <a
href=https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html>MetricsOptions</a>
class.
It contains push period configuration and also sink specific options such as
type and URL. As for now only the REST HTTP and the Graphite sinks are
supported and only
Flink and Spark runners support metrics export.</p><p>Also Beam metrics are
exported to inner Spark and Flink dashboards to be consulted in their
respective UI.</p><h2 id=state-and-timers>11. State and
Timers</h2><p>Beam’s windowing and triggering facilities provide a
powerful abstraction for grouping and aggregating unbounded input
@@ -2273,7 +2273,7 @@ processing, and are recommended to test for
correctness.</p><p class=language-ja
state must be named using the <code>StateId</code> annotation; this name is
unique to a ParDo in the graph and has no relation
to other nodes in the graph. A <code>DoFn</code> can declare multiple state
variables.</p><p class=language-py>In Python DoFn declares states to be
accessed by creating <code>StateSpec</code> class member variables representing
each state. Each
<code>StateSpec</code> is initialized with a name, this name is unique to a
ParDo in the graph and has no relation
-to other nodes in the graph. A <code>DoFn</code> can declare multiple state
variables.</p><h3 id=types-of-state>11.1 Types of state</h3><p>Beam provides
several types of state:</p><h4 id=valuestate>ValueState</h4><p>A ValueState is
a scalar state value. For each key in the input, a ValueState will store a
typed value that can be
+to other nodes in the graph. A <code>DoFn</code> can declare multiple state
variables.</p><h3 id=types-of-state>11.1. Types of state</h3><p>Beam provides
several types of state:</p><h4 id=valuestate>ValueState</h4><p>A ValueState is
a scalar state value. For each key in the input, a ValueState will store a
typed value that can be
read and modified inside the DoFn’s <code>@ProcessElement</code> or
<code>@OnTimer</code> methods. If the type of the ValueState has a coder
registered, then Beam will automatically infer the coder for the state value.
Otherwise, a coder can be explicitly
specified when creating the ValueState. For example, the following ParDo
creates a single state variable that
@@ -2338,7 +2338,7 @@ bags larger than available memory.</p><div
class=language-java><div class=highli
<span class=n>state</span><span class=o>.</span><span
class=n>clear</span><span class=p>()</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'Bag state pardo'</span>
<span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>BagStateDoFn</span><span
class=p>()))</span></code></pre></div></div><h3 id=deferred-state-reads>11.2
Deferred state reads</h3><p>When a <code>DoFn</code> contains multiple state
specifications, reading each one in order can be slow. Calling the
<code>read()</code> function
+ <span class=o>|</span> <span class=s1>'Bag state pardo'</span>
<span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>BagStateDoFn</span><span
class=p>()))</span></code></pre></div></div><h3 id=deferred-state-reads>11.2.
Deferred state reads</h3><p>When a <code>DoFn</code> contains multiple state
specifications, reading each one in order can be slow. Calling the
<code>read()</code> function
on a state can cause the runner to perform a blocking read. Performing
multiple blocking reads in sequence adds latency
to element processing. If you know that a state will always be read, you can
annotate it as @AlwaysFetched, and then the
runner can prefetch all of the states necessary. For example:</p><div
class=language-java><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PCollection</span><span
class=o><</span><span class=n>KV</span><span class=o><</span><span
class=n>String</span><span class=o>,</span> <span class=n>ValueT</span><span
class=o>>></span> <span class=n>perUser</span> <span class=o>=</span>
<span class=n>readPerUser</span><span class=o>();</span>
@@ -2378,10 +2378,10 @@ be read in the future, allowing multiple state reads to
be batched together.</p>
<span class=n>processState2</span><span class=o>(</span><span
class=n>state2</span><span class=o>.</span><span class=na>read</span><span
class=o>());</span>
<span class=n>processState3</span><span class=o>(</span><span
class=n>state3</span><span class=o>.</span><span class=na>read</span><span
class=o>());</span>
<span class=o>}</span>
-<span class=o>}));</span></code></pre></div></div><h3 id=timers>11.3
Timers</h3><p>Beam provides a per-key timer callback API. This allows for
delayed processing of data stored using the state API.
+<span class=o>}));</span></code></pre></div></div><h3 id=timers>11.3.
Timers</h3><p>Beam provides a per-key timer callback API. This allows for
delayed processing of data stored using the state API.
Timers can be set to callback at either an event-time or a processing-time
timestamp. Every timer is identified with a
TimerId. A given timer for a key can only be set for a single timestamp.
Calling set on a timer overwrites the previous
-firing time for that key’s timer.</p><h4 id=event-time-timers>11.3.1
Event-time timers</h4><p>Event-time timers fire when the input watermark for
the DoFn passes the time at which the timer is set, meaning that
+firing time for that key’s timer.</p><h4 id=event-time-timers>11.3.1.
Event-time timers</h4><p>Event-time timers fire when the input watermark for
the DoFn passes the time at which the timer is set, meaning that
the runner believes that there are no more elements to be processed with
timestamps before the timer timestamp. This
allows for event-time aggregations.</p><div class=language-java><div
class=highlight><pre class=chroma><code class=language-java
data-lang=java><span class=n>PCollection</span><span class=o><</span><span
class=n>KV</span><span class=o><</span><span class=n>String</span><span
class=o>,</span> <span class=n>ValueT</span><span class=o>>></span> <span
class=n>perUser</span> <span class=o>=</span> <span
class=n>readPerUser</span><span class=o>();</span>
<span class=n>perUser</span><span class=o>.</span><span
class=na>apply</span><span class=o>(</span><span class=n>ParDo</span><span
class=o>.</span><span class=na>of</span><span class=o>(</span><span
class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span
class=n>KV</span><span class=o><</span><span class=n>String</span><span
class=o>,</span> <span class=n>ValueT</span><span class=o>>,</span> <span
class=n>OutputT</span><span class=o>>()</span> <span class=o>{</span>
@@ -2419,7 +2419,7 @@ allows for event-time aggregations.</p><div
class=language-java><div class=highl
<span class=n>state</span><span class=o>.</span><span
class=n>clear</span><span class=p>()</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'EventTime timer
pardo'</span> <span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>EventTimerDoFn</span><span
class=p>()))</span></code></pre></div></div><h4
id=processing-time-timers>11.3.2 Processing-time timers</h4><p>Processing-time
timers fire when the real wall-clock time passes. This is often used to create
larger batches of data
+ <span class=o>|</span> <span class=s1>'EventTime timer
pardo'</span> <span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>EventTimerDoFn</span><span
class=p>()))</span></code></pre></div></div><h4
id=processing-time-timers>11.3.2. Processing-time timers</h4><p>Processing-time
timers fire when the real wall-clock time passes. This is often used to create
larger batches of data
before processing. It can also be used to schedule events that should occur at
a specific time. Just like with
event-time timers, processing-time timers are per key - each key has a
separate copy of the timer.</p><p>While processing-time timers can be set to an
absolute timestamp, it is very common to set them to an offset relative
to the current time. In Java, the <code>Timer.offset</code> and
<code>Timer.setRelative</code> methods can be used to accomplish this.</p><div
class=language-java><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PCollection</span><span
class=o><</span><span class=n>KV</span><span class=o><</span><span
class=n>String</span><span class=o>,</span> <span class=n>ValueT</span><span
class=o>>></span> <span class=n>perUser</span> <span cl [...]
@@ -2453,7 +2453,7 @@ to the current time. In Java, the
<code>Timer.offset</code> and <code>Timer.setR
<span class=n>state</span><span class=o>.</span><span
class=n>clear</span><span class=p>()</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'ProcessingTime timer
pardo'</span> <span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>ProcessingTimerDoFn</span><span
class=p>()))</span></code></pre></div></div><h4 id=dynamic-timer-tags>11.3.3
Dynamic timer tags</h4><p>Beam also supports dynamically setting a timer tag
using <code>TimerMap</code>. This allows for setting multiple different timers
+ <span class=o>|</span> <span class=s1>'ProcessingTime timer
pardo'</span> <span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>ProcessingTimerDoFn</span><span
class=p>()))</span></code></pre></div></div><h4 id=dynamic-timer-tags>11.3.3.
Dynamic timer tags</h4><p>Beam also supports dynamically setting a timer tag
using <code>TimerMap</code>. This allows for setting multiple different timers
in a <code>DoFn</code> and allowing for the timer tags to be dynamically
chosen - e.g. based on data in the input elements. A
timer with a specific tag can only be set to a single timestamp, so setting
the timer again has the effect of
overwriting the previous expiration time for the timer with that tag. Each
<code>TimerMap</code> is identified with a timer family
@@ -2472,7 +2472,7 @@ id, and timers in different timer families are
independent.</p><div class=langua
<span class=nd>@OnTimerFamily</span><span class=o>(</span><span
class=s>"actionTimers"</span><span class=o>)</span> <span
class=kd>public</span> <span class=kt>void</span> <span
class=nf>onTimer</span><span class=o>(</span><span class=nd>@TimerId</span>
<span class=n>String</span> <span class=n>timerId</span><span class=o>)</span>
<span class=o>{</span>
<span class=n>LOG</span><span class=o>.</span><span
class=na>info</span><span class=o>(</span><span class=s>"Timer fired with
id "</span> <span class=o>+</span> <span class=n>timerId</span><span
class=o>);</span>
<span class=o>}</span>
-<span class=o>}));</span></code></pre></div></div><div
class=language-python><div class=highlight><pre class=chroma><code
class=language-python data-lang=python><span class=n>To</span> <span
class=n>be</span> <span class=n>supported</span><span class=p>,</span> <span
class=n>See</span> <span class=n>BEAM</span><span class=o>-</span><span
class=mi>9602</span></code></pre></div></div><h4
id=timer-output-timestamps>11.3.4 Timer output timestamps</h4><p>By default,
event-time timers will hol [...]
+<span class=o>}));</span></code></pre></div></div><div
class=language-python><div class=highlight><pre class=chroma><code
class=language-python data-lang=python><span class=n>To</span> <span
class=n>be</span> <span class=n>supported</span><span class=p>,</span> <span
class=n>See</span> <span class=n>BEAM</span><span class=o>-</span><span
class=mi>9602</span></code></pre></div></div><h4
id=timer-output-timestamps>11.3.4. Timer output timestamps</h4><p>By default,
event-time timers will ho [...]
that if a timer is set to 12pm, any windowed aggregations or event-time timers
later in the pipeline graph that finish
after 12pm will not expire. The timestamp of the timer is also the default
output timestamp for the timer callback. This
means that any elements output from the onTimer method will have a timestamp
equal to the timestamp of the timer firing.
@@ -2559,8 +2559,8 @@ past the timestamp of the minimum element. The following
code demonstrates this.
<span class=c1>// Note that the timer has now fired.
</span><span class=c1></span> <span class=n>timerTimestamp</span><span
class=o>.</span><span class=na>clear</span><span class=o>();</span>
<span class=o>}</span>
-<span class=o>}));</span></code></pre></div></div><h3
id=garbage-collecting-state>11.4 Garbage collecting state</h3><p>Per-key state
needs to be garbage collected, or eventually the increasing size of state may
negatively impact
-performance. There are two common strategies for garbage collecting
state.</p><h5 id=using-windows-for-garbage-collection>11.4.1 <strong>Using
windows for garbage collection</strong></h5><p>All state and timers for a key
is scoped to the window it is in. This means that depending on the timestamp of
the
+<span class=o>}));</span></code></pre></div></div><h3
id=garbage-collecting-state>11.4. Garbage collecting state</h3><p>Per-key state
needs to be garbage collected, or eventually the increasing size of state may
negatively impact
+performance. There are two common strategies for garbage collecting
state.</p><h5 id=using-windows-for-garbage-collection>11.4.1. <strong>Using
windows for garbage collection</strong></h5><p>All state and timers for a key
is scoped to the window it is in. This means that depending on the timestamp of
the
input element the ParDo will see different values for the state depending on
the window that element falls into. In
addition, once the input watermark passes the end of the window, the runner
should garbage collect all state for that
window. (note: if allowed lateness is set to a positive value for the window,
the runner must wait for the watermark to
@@ -2586,7 +2586,7 @@ garbage-collection strategy.</p><p>For example, given the
following:</p><div cla
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
<span class=o>|</span> <span class=s1>'Windowing'</span> <span
class=o>>></span> <span class=n>beam</span><span class=o>.</span><span
class=n>WindowInto</span><span class=p>(</span><span
class=n>FixedWindows</span><span class=p>(</span><span class=mi>60</span> <span
class=o>*</span> <span class=mi>60</span> <span class=o>*</span> <span
class=mi>24</span><span class=p>))</span>
<span class=o>|</span> <span class=s1>'DoFn'</span> <span
class=o>>></span> <span class=n>beam</span><span class=o>.</span><span
class=n>ParDo</span><span class=p>(</span><span class=n>StateDoFn</span><span
class=p>()))</span></code></pre></div></div><p>This <code>ParDo</code> stores
state per day. Once the pipeline is done processing data for a given day, all
the state for that
-day is garbage collected.</p><h5 id=using-timers-for-garbage-collection>11.4.1
<strong>Using timers For garbage collection</strong></h5><p>In some cases, it
is difficult to find a windowing strategy that models the desired
garbage-collection strategy. For
+day is garbage collected.</p><h5
id=using-timers-for-garbage-collection>11.4.1. <strong>Using timers For garbage
collection</strong></h5><p>In some cases, it is difficult to find a windowing
strategy that models the desired garbage-collection strategy. For
example, a common desire is to garbage collect state for a key once no
activity has been seen on the key for some time.
This can be done by updating a timer that garbage collects state. For
example</p><div class=language-java><div class=highlight><pre
class=chroma><code class=language-java data-lang=java><span
class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span
class=o><</span><span class=n>String</span><span class=o>,</span> <span
class=n>ValueT</span><span class=o>>></span> <span class=n>perUser</span>
<span class=o>=</span> <span class=n>readPerUser</span><span clas [...]
<span class=n>perUser</span><span class=o>.</span><span
class=na>apply</span><span class=o>(</span><span class=n>ParDo</span><span
class=o>.</span><span class=na>of</span><span class=o>(</span><span
class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span
class=n>KV</span><span class=o><</span><span class=n>String</span><span
class=o>,</span> <span class=n>ValueT</span><span class=o>>,</span> <span
class=n>OutputT</span><span class=o>>()</span> <span class=o>{</span>
@@ -2651,7 +2651,7 @@ This can be done by updating a timer that garbage
collects state. For example</p
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'User DoFn'</span> <span
class=o>>></span> <span class=n>beam</span><span class=o>.</span><span
class=n>ParDo</span><span class=p>(</span><span class=n>UserDoFn</span><span
class=p>()))</span></code></pre></div></div><h3 id=state-timers-examples>11.5
State and timers examples</h3><p>Following are some example uses of state and
timers</p><h4 id=joining-clicks-and-views>11.5.1. Joining clicks and
views</h4><p>In this example, the p [...]
+ <span class=o>|</span> <span class=s1>'User DoFn'</span> <span
class=o>>></span> <span class=n>beam</span><span class=o>.</span><span
class=n>ParDo</span><span class=p>(</span><span class=n>UserDoFn</span><span
class=p>()))</span></code></pre></div></div><h3 id=state-timers-examples>11.5.
State and timers examples</h3><p>Following are some example uses of state and
timers</p><h4 id=joining-clicks-and-views>11.5.1. Joining clicks and
views</h4><p>In this example, the [...]
a stream of views, representing suggested product links displayed to the user
on the home page, and a stream of
clicks, representing actual user clicks on these links. The goal of the
pipeline is to join click events with view
events, outputting a new joined event that contains information from both
events. Each link has a unique identifier
@@ -2723,7 +2723,7 @@ event before the view event. The one hour join timeout
should be based on event
<span class=n>clickState</span><span class=o>.</span><span
class=na>clear</span><span class=o>();</span>
<span class=n>maxTimestampState</span><span class=o>.</span><span
class=na>clear</span><span class=o>();</span>
<span class=o>}</span>
- <span class=o>}));</span></code></pre></div></div><h4 id=batching-rpcs>11.5.2
Batching RPCs</h4><p>In this example, input elements are being forwarded to an
external RPC service. The RPC accepts batch requests -
+ <span class=o>}));</span></code></pre></div></div><h4
id=batching-rpcs>11.5.2. Batching RPCs</h4><p>In this example, input elements
are being forwarded to an external RPC service. The RPC accepts batch requests -
multiple events for the same user can be batched in a single RPC call. Since
this RPC service also imposes rate limits,
we want to batch ten seconds worth of events together in order to reduce the
number of calls.</p><div class=language-java><div class=highlight><pre
class=chroma><code class=language-java data-lang=java><span
class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span
class=o><</span><span class=n>String</span><span class=o>,</span> <span
class=n>ValueT</span><span class=o>>></span> <span class=n>perUser</span>
<span class=o>=</span> <span class=n>readPerUser< [...]
<span class=n>perUser</span><span class=o>.</span><span
class=na>apply</span><span class=o>(</span><span class=n>ParDo</span><span
class=o>.</span><span class=na>of</span><span class=o>(</span><span
class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span
class=n>KV</span><span class=o><</span><span class=n>String</span><span
class=o>,</span> <span class=n>ValueT</span><span class=o>>,</span> <span
class=n>OutputT</span><span class=o>>()</span> <span class=o>{</span>
diff --git a/website/generated-content/sitemap.xml
b/website/generated-content/sitemap.xml
index 9a3febc..cd183e8 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.24.0/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/p
[...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.24.0/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/p
[...]
\ No newline at end of file
