This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a470b0a395b Publishing website 2022/05/03 04:17:00 at commit 0daef62
a470b0a395b is described below
commit a470b0a395b3ee0f038a6742e0911b73b081c4be
Author: jenkins <[email protected]>
AuthorDate: Tue May 3 04:17:01 2022 +0000
Publishing website 2022/05/03 04:17:00 at commit 0daef62
---
website/generated-content/documentation/index.xml | 47 +++++++++++++++++-
.../documentation/programming-guide/index.html | 56 +++++++++++++++++++++-
website/generated-content/sitemap.xml | 2 +-
3 files changed, 101 insertions(+), 4 deletions(-)
diff --git a/website/generated-content/documentation/index.xml
b/website/generated-content/documentation/index.xml
index 336a66df3f5..f535f993226 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -11627,7 +11627,52 @@ watermark estimator implementation. You can also
provide your own watermark esti
<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
<img src="/images/copy-icon.svg"/>
</a>
-<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">11105.</span></code></pre></div>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="c1">// WatermarkState is a custom type.`
+</span><span class="c1">//
+</span><span class="c1">// It is optional to write your own state type
when making a custom estimator.
+</span><span class="c1"></span><span class="kd">type</span>
<span class="nx">WatermarkState</span> <span
class="kd">struct</span> <span class="p">{</span>
+<span class="nx">Watermark</span> <span
class="nx">time</span><span class="p">.</span><span
class="nx">Time</span>
+<span class="p">}</span>
+<span class="c1">// CustomWatermarkEstimator is a custom watermark
estimator.
+</span><span class="c1">// You may use any type here, including some of
Beam&#39;s built in watermark estimator types,
+</span><span class="c1">// e.g. sdf.WallTimeWatermarkEstimator,
sdf.TimestampObservingWatermarkEstimator, and sdf.ManualWatermarkEstimator
+</span><span class="c1"></span><span class="kd">type</span>
<span class="nx">CustomWatermarkEstimator</span> <span
class="kd">struct</span> <span class="p">{</span>
+<span class="nx">state</span> <span
class="nx">WatermarkState</span>
+<span class="p">}</span>
+<span class="c1">// CurrentWatermark returns the current watermark and is
invoked on DoFn splits and self-checkpoints.
+</span><span class="c1">// Watermark estimators must implement
CurrentWatermark() time.Time
+</span><span class="c1"></span><span class="kd">func</span>
<span class="p">(</span><span class="nx">e</span> <span
class="o">*</span><span
class="nx">CustomWatermarkEstimator</span><span class="p">)</span>
<span class="nf">CurrentWatermark</span><span class="p">()</span>
<span class="nx">time</span><span class="p">.</span><span
class="nx">Time</span> <span class="p">{</span>
+<span class="k">return</span> <span class="nx">e</span><span
class="p">.</span><span class="nx">state</span><span
class="p">.</span><span class="nx">Watermark</span>
+<span class="p">}</span>
+<span class="c1">// ObserveTimestamp is called on the output timestamps of
all
+</span><span class="c1">// emitted elements to update the watermark. It
is optional
+</span><span class="c1"></span><span class="kd">func</span>
<span class="p">(</span><span class="nx">e</span> <span
class="o">*</span><span
class="nx">CustomWatermarkEstimator</span><span class="p">)</span>
<span class="nf">ObserveTimestamp</span><span
class="p">(</span><span class="nx">ts</span> <span
class="nx">time</span><span class="p">.</span><span
class="nx">Time</span><span class="p">)</span> &l [...]
+<span class="nx">e</span><span class="p">.</span><span
class="nx">state</span><span class="p">.</span><span
class="nx">Watermark</span> <span class="p">=</span> <span
class="nx">ts</span>
+<span class="p">}</span>
+<span class="c1">// InitialWatermarkEstimatorState defines an initial state
used to initialize the watermark
+</span><span class="c1">// estimator. It is optional. If this is not
defined, WatermarkEstimatorState may not be
+</span><span class="c1">// defined and CreateWatermarkEstimator must not
take in parameters.
+</span><span class="c1"></span><span class="kd">func</span>
<span class="p">(</span><span class="nx">fn</span> <span
class="o">*</span><span class="nx">weDoFn</span><span
class="p">)</span> <span
class="nf">InitialWatermarkEstimatorState</span><span
class="p">(</span><span class="nx">et</span> <span
class="nx">beam</span><span class="p">.</span><span
class="nx">EventTime</span><span class="p">,</span> [...]
+<span class="c1">// Return some watermark state
+</span><span class="c1"></span> <span class="k">return</span>
<span class="nx">WatermarkState</span><span
class="p">{</span><span class="nx">Watermark</span><span
class="p">:</span> <span class="nx">time</span><span
class="p">.</span><span class="nf">Now</span><span
class="p">()}</span>
+<span class="p">}</span>
+<span class="c1">// CreateWatermarkEstimator creates the watermark
estimator used by this Splittable DoFn.
+</span><span class="c1">// Must take in a state parameter if
InitialWatermarkEstimatorState is defined, otherwise takes no parameters.
+</span><span class="c1"></span><span class="kd">func</span>
<span class="p">(</span><span class="nx">fn</span> <span
class="o">*</span><span class="nx">weDoFn</span><span
class="p">)</span> <span
class="nf">CreateWatermarkEstimator</span><span
class="p">(</span><span class="nx">initialState</span> <span
class="nx">WatermarkState</span><span class="p">)</span> <span
class="o">*</span><span class="nx">Custo [...]
+<span class="k">return</span> <span
class="o">&amp;</span><span
class="nx">CustomWatermarkEstimator</span><span
class="p">{</span><span class="nx">state</span><span
class="p">:</span> <span class="nx">initialState</span><span
class="p">}</span>
+<span class="p">}</span>
+<span class="c1">// WatermarkEstimatorState returns the state used to
resume future watermark estimation
+</span><span class="c1">// after a checkpoint/split. It is required if
InitialWatermarkEstimatorState is defined,
+</span><span class="c1">// otherwise it must not be defined.
+</span><span class="c1"></span><span class="kd">func</span>
<span class="p">(</span><span class="nx">fn</span> <span
class="o">*</span><span class="nx">weDoFn</span><span
class="p">)</span> <span
class="nf">WatermarkEstimatorState</span><span
class="p">(</span><span class="nx">e</span> <span
class="o">*</span><span
class="nx">CustomWatermarkEstimator</span><span class="p">)</span>
<span class="nx">Waterma [...]
+<span class="k">return</span> <span class="nx">e</span><span
class="p">.</span><span class="nx">state</span>
+<span class="p">}</span>
+<span class="c1">// ProcessElement is the method to execute for each
element.
+</span><span class="c1">// It can optionally take in a watermark
estimator.
+</span><span class="c1"></span><span class="kd">func</span>
<span class="p">(</span><span class="nx">fn</span> <span
class="o">*</span><span class="nx">weDoFn</span><span
class="p">)</span> <span class="nf">ProcessElement</span><span
class="p">(</span><span class="nx">e</span> <span
class="o">*</span><span
class="nx">CustomWatermarkEstimator</span><span class="p">,</span>
<span class="nx">element</span [...]
+<span class="c1">// ...
+</span><span class="c1"></span> <span
class="nx">e</span><span class="p">.</span><span
class="nx">state</span><span class="p">.</span><span
class="nx">Watermark</span> <span class="p">=</span> <span
class="nx">time</span><span class="p">.</span><span
class="nf">Now</span><span class="p">()</span>
+<span class="p">}</span></code></pre></div>
</div>
</div>
<h3 id="truncating-during-drain">12.6. Truncating during drain</h3>
diff --git
a/website/generated-content/documentation/programming-guide/index.html
b/website/generated-content/documentation/programming-guide/index.html
index 9ca51a73d33..578f66a485d 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -4116,7 +4116,59 @@ watermark estimator implementation. You can also provide
your own watermark esti
<span class=n>watermark_estimator</span><span class=o>=</span><span
class=n>beam</span><span class=o>.</span><span class=n>DoFn</span><span
class=o>.</span><span class=n>WatermarkEstimatorParam</span><span
class=p>(</span>
<span class=n>MyWatermarkEstimatorProvider</span><span
class=p>())):</span>
<span class=c1># The current watermark can be inspected.</span>
- <span class=n>watermark_estimator</span><span class=o>.</span><span
class=n>current_watermark</span><span
class=p>()</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-go data-lang=go><span class=nx>This</span> <span
class=nx>is</span [...]
+ <span class=n>watermark_estimator</span><span class=o>.</span><span
class=n>current_watermark</span><span
class=p>()</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-go data-lang=go><span class=c1>// WatermarkState is a custom
type.`
+</span><span class=c1>//
+</span><span class=c1>// It is optional to write your own state type when
making a custom estimator.
+</span><span class=c1></span><span class=kd>type</span> <span
class=nx>WatermarkState</span> <span class=kd>struct</span> <span
class=p>{</span>
+ <span class=nx>Watermark</span> <span class=nx>time</span><span
class=p>.</span><span class=nx>Time</span>
+<span class=p>}</span>
+
+<span class=c1>// CustomWatermarkEstimator is a custom watermark estimator.
+</span><span class=c1>// You may use any type here, including some of
Beam's built in watermark estimator types,
+</span><span class=c1>// e.g. sdf.WallTimeWatermarkEstimator,
sdf.TimestampObservingWatermarkEstimator, and sdf.ManualWatermarkEstimator
+</span><span class=c1></span><span class=kd>type</span> <span
class=nx>CustomWatermarkEstimator</span> <span class=kd>struct</span> <span
class=p>{</span>
+ <span class=nx>state</span> <span class=nx>WatermarkState</span>
+<span class=p>}</span>
+
+<span class=c1>// CurrentWatermark returns the current watermark and is
invoked on DoFn splits and self-checkpoints.
+</span><span class=c1>// Watermark estimators must implement
CurrentWatermark() time.Time
+</span><span class=c1></span><span class=kd>func</span> <span
class=p>(</span><span class=nx>e</span> <span class=o>*</span><span
class=nx>CustomWatermarkEstimator</span><span class=p>)</span> <span
class=nf>CurrentWatermark</span><span class=p>()</span> <span
class=nx>time</span><span class=p>.</span><span class=nx>Time</span> <span
class=p>{</span>
+ <span class=k>return</span> <span class=nx>e</span><span
class=p>.</span><span class=nx>state</span><span class=p>.</span><span
class=nx>Watermark</span>
+<span class=p>}</span>
+
+<span class=c1>// ObserveTimestamp is called on the output timestamps of all
+</span><span class=c1>// emitted elements to update the watermark. It is
optional
+</span><span class=c1></span><span class=kd>func</span> <span
class=p>(</span><span class=nx>e</span> <span class=o>*</span><span
class=nx>CustomWatermarkEstimator</span><span class=p>)</span> <span
class=nf>ObserveTimestamp</span><span class=p>(</span><span class=nx>ts</span>
<span class=nx>time</span><span class=p>.</span><span class=nx>Time</span><span
class=p>)</span> <span class=p>{</span>
+ <span class=nx>e</span><span class=p>.</span><span
class=nx>state</span><span class=p>.</span><span class=nx>Watermark</span>
<span class=p>=</span> <span class=nx>ts</span>
+<span class=p>}</span>
+
+<span class=c1>// InitialWatermarkEstimatorState defines an initial state used
to initialize the watermark
+</span><span class=c1>// estimator. It is optional. If this is not defined,
WatermarkEstimatorState may not be
+</span><span class=c1>// defined and CreateWatermarkEstimator must not take in
parameters.
+</span><span class=c1></span><span class=kd>func</span> <span
class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span
class=nx>weDoFn</span><span class=p>)</span> <span
class=nf>InitialWatermarkEstimatorState</span><span class=p>(</span><span
class=nx>et</span> <span class=nx>beam</span><span class=p>.</span><span
class=nx>EventTime</span><span class=p>,</span> <span class=nx>rest</span>
<span class=nx>offsetrange</span><span class=p>.</span><span
class=nx>Restriction</span> [...]
+ <span class=c1>// Return some watermark state
+</span><span class=c1></span> <span class=k>return</span> <span
class=nx>WatermarkState</span><span class=p>{</span><span
class=nx>Watermark</span><span class=p>:</span> <span class=nx>time</span><span
class=p>.</span><span class=nf>Now</span><span class=p>()}</span>
+<span class=p>}</span>
+
+<span class=c1>// CreateWatermarkEstimator creates the watermark estimator
used by this Splittable DoFn.
+</span><span class=c1>// Must take in a state parameter if
InitialWatermarkEstimatorState is defined, otherwise takes no parameters.
+</span><span class=c1></span><span class=kd>func</span> <span
class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span
class=nx>weDoFn</span><span class=p>)</span> <span
class=nf>CreateWatermarkEstimator</span><span class=p>(</span><span
class=nx>initialState</span> <span class=nx>WatermarkState</span><span
class=p>)</span> <span class=o>*</span><span
class=nx>CustomWatermarkEstimator</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=o>&</span><span
class=nx>CustomWatermarkEstimator</span><span class=p>{</span><span
class=nx>state</span><span class=p>:</span> <span
class=nx>initialState</span><span class=p>}</span>
+<span class=p>}</span>
+
+<span class=c1>// WatermarkEstimatorState returns the state used to resume
future watermark estimation
+</span><span class=c1>// after a checkpoint/split. It is required if
InitialWatermarkEstimatorState is defined,
+</span><span class=c1>// otherwise it must not be defined.
+</span><span class=c1></span><span class=kd>func</span> <span
class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span
class=nx>weDoFn</span><span class=p>)</span> <span
class=nf>WatermarkEstimatorState</span><span class=p>(</span><span
class=nx>e</span> <span class=o>*</span><span
class=nx>CustomWatermarkEstimator</span><span class=p>)</span> <span
class=nx>WatermarkState</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=nx>e</span><span
class=p>.</span><span class=nx>state</span>
+<span class=p>}</span>
+
+<span class=c1>// ProcessElement is the method to execute for each element.
+</span><span class=c1>// It can optionally take in a watermark estimator.
+</span><span class=c1></span><span class=kd>func</span> <span
class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span
class=nx>weDoFn</span><span class=p>)</span> <span
class=nf>ProcessElement</span><span class=p>(</span><span class=nx>e</span>
<span class=o>*</span><span class=nx>CustomWatermarkEstimator</span><span
class=p>,</span> <span class=nx>element</span> <span
class=kt>string</span><span class=p>)</span> <span class=p>{</span>
+ <span class=c1>// ...
+</span><span class=c1></span> <span class=nx>e</span><span
class=p>.</span><span class=nx>state</span><span class=p>.</span><span
class=nx>Watermark</span> <span class=p>=</span> <span
class=nx>time</span><span class=p>.</span><span class=nf>Now</span><span
class=p>()</span>
+<span class=p>}</span></code></pre></div></div></div><h3
id=truncating-during-drain>12.6. Truncating during drain</h3><p>Runners which
support draining pipelines need the ability to drain SDFs; otherwise, the
pipeline may never stop. By default, bounded restrictions process the
remainder of the restriction while
unbounded restrictions finish processing at the next SDF-initiated checkpoint
or runner-initiated split.
You are able to override this default behavior by defining the appropriate
method on the restriction
@@ -4340,7 +4392,7 @@ functions as shortcuts for single, unnamed inputs/outputs
or define a map for na
<span class=nx>outT</span> <span class=o>:=</span> <span
class=nx>beam</span><span class=p>.</span><span
class=nf>UnnamedOutput</span><span class=p>(</span><span
class=nx>typex</span><span class=p>.</span><span class=nf>New</span><span
class=p>(</span><span class=nx>reflectx</span><span class=p>.</span><span
class=nx>String</span><span class=p>))</span>
<span class=nx>res</span> <span class=o>:=</span> <span
class=nx>beam</span><span class=p>.</span><span
class=nf>CrossLanguage</span><span class=p>(</span><span class=nx>s</span><span
class=p>,</span> <span class=nx>urn</span><span class=p>,</span> <span
class=nx>payload</span><span class=p>,</span> <span
class=nx>expansionAddr</span><span class=p>,</span> <span
class=nx>beam</span><span class=p>.</span><span
class=nf>UnnamedInput</span><span class=p>(</span><span class=nx>inputPCol</spa
[...]
</code></pre></div></li><li><p>After the job has been submitted to the Beam
runner, shutdown the expansion service by
-terminating the expansion service process.</p></li></ol><h3
id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently,
portable runners such as Flink, Spark, and the direct runner can be used with
multi-language pipelines.</p><p>Dataflow supports multi-language pipelines
through the Dataflow Runner v2 backend architecture.</p><h3
id=x-lang-transform-tips-troubleshooting>13.4 Tips and
Troubleshooting</h3><p>For additional tips and troubleshooting information, see
<a href=ht [...]
+terminating the expansion service process.</p></li></ol><h3
id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently,
portable runners such as Flink, Spark, and the direct runner can be used with
multi-language pipelines.</p><p>Dataflow supports multi-language pipelines
through the Dataflow Runner v2 backend architecture.</p><h3
id=x-lang-transform-tips-troubleshooting>13.4 Tips and
Troubleshooting</h3><p>For additional tips and troubleshooting information, see
<a href=ht [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam
logo, and the Apache feather logo are either registered trademarks or
trademarks of The Apache Software Foundation. All other products or name brands
are trademarks of their respective holders, including The Apache Software
Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml
b/website/generated-content/sitemap.xml
index 6eb3a9963ae..e2bc4573dab 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/categories/blog/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url><loc>/blog/beam-sql-with-notebooks/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url>
[...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/categories/blog/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url><loc>/blog/beam-sql-with-notebooks/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url>
[...]
\ No newline at end of file
