This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push: new 6ad0c23 Publishing website 2020/02/06 01:18:56 at commit a8af0e1 6ad0c23 is described below commit 6ad0c23c530bae840823f1266cf8c15163fa4ea3 Author: jenkins <bui...@apache.org> AuthorDate: Thu Feb 6 01:18:56 2020 +0000 Publishing website 2020/02/06 01:18:56 at commit a8af0e1 --- .../documentation/programming-guide/index.html | 24 ++++++++++++++-------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html index 178854b..7d1dfae 100644 --- a/website/generated-content/documentation/programming-guide/index.html +++ b/website/generated-content/documentation/programming-guide/index.html @@ -3263,10 +3263,6 @@ elements.</p> <h4 id="managing-late-data">7.4.1. Managing late data</h4> -<blockquote> - <p><strong>Note:</strong> Managing late data is not supported in the Beam SDK for Python.</p> -</blockquote> - <p>You can allow late data by invoking the <code class="highlighter-rouge">.withAllowedLateness</code> operation when you set your <code class="highlighter-rouge">PCollection</code>’s windowing strategy. The following code example demonstrates a windowing strategy that will allow late data up to two days after @@ -3278,6 +3274,14 @@ the end of a window.</p> <span class="o">.</span><span class="na">withAllowedLateness</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardDays</span><span class="o">(</span><span class="mi">2</span><span class="o">)));</span> </code></pre></div></div> +<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">pc</span> <span class="o">=</span> <span class="p">[</span><span class="n">Initial</span> <span class="n">PCollection</span><span class="p">]</span> + <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span> + <span class="n">FixedWindows</span><span class="p">(</span><span class="mi">60</span><span class="p">),</span> + <span class="n">trigger</span><span class="o">=</span><span class="n">trigger_fn</span><span class="p">,</span> + <span class="n">accumulation_mode</span><span class="o">=</span><span class="n">accumulation_mode</span><span class="p">,</span> + <span class="n">timestamp_combiner</span><span class="o">=</span><span class="n">timestamp_combiner</span><span class="p">,</span> + <span class="n">allowed_lateness</span><span class="o">=</span><span class="n">Duration</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">2</span><span class="o">*</span><span class="mi">24</span><span class="o">*</span><span class="mi">60</span><span class="o">*</span><span class="mi">60</span><span class="p">))</span> <span class="c"># 2 days</span> +</code></pre></div></div> <p>When you set <code class="highlighter-rouge">.withAllowedLateness</code> on a <code class="highlighter-rouge">PCollection</code>, that allowed lateness propagates forward to any subsequent <code class="highlighter-rouge">PCollection</code> derived from the first <code class="highlighter-rouge">PCollection</code> you applied allowed lateness to. If you want to change the allowed @@ -3555,10 +3559,6 @@ on each firing:</p> <h4 id="handling-late-data">8.4.2. Handling late data</h4> -<blockquote> - <p>The Beam SDK for Python does not currently support allowed lateness.</p> -</blockquote> - <p>If you want your pipeline to process data that arrives after the watermark passes the end of the window, you can apply an <em>allowed lateness</em> when you set your windowing configuration. This gives your trigger the opportunity to react @@ -3574,7 +3574,13 @@ windowing function:</p> <span class="o">.</span><span class="na">plusDelayOf</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span> <span class="o">.</span><span class="na">withAllowedLateness</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">30</span><span class="o">));</span> </code></pre></div></div> -<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c"># The Beam SDK for Python does not currently support allowed lateness.</span> +<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">pc</span> <span class="o">=</span> <span class="p">[</span><span class="n">Initial</span> <span class="n">PCollection</span><span class="p">]</span> + <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span> + <span class="n">FixedWindows</span><span class="p">(</span><span class="mi">60</span><span class="p">),</span> + <span class="n">trigger</span><span class="o">=</span><span class="n">AfterProcessingTime</span><span class="p">(</span><span class="mi">60</span><span class="p">),</span> + <span class="n">allowed_lateness</span><span class="o">=</span><span class="mi">1800</span><span class="p">)</span> <span class="c"># 30 minutes</span> + <span class="o">|</span> <span class="o">...</span> + </code></pre></div></div> <p>This allowed lateness propagates to all <code class="highlighter-rouge">PCollection</code>s derived as a result of