Regenerate website
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/5a7282fe Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/5a7282fe Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/5a7282fe Branch: refs/heads/asf-site Commit: 5a7282fe42e9a396673aacfac8f466284dffa1ea Parents: 6289505 Author: Ahmet Altay <al...@altay-macbookpro2.roam.corp.google.com> Authored: Tue May 2 17:59:11 2017 -0700 Committer: Ahmet Altay <al...@altay-macbookpro2.roam.corp.google.com> Committed: Tue May 2 17:59:11 2017 -0700 ---------------------------------------------------------------------- content/documentation/io/built-in/index.html | 2 +- .../documentation/programming-guide/index.html | 46 +++++++++++++++++--- .../sdks/python-type-safety/index.html | 10 ++--- 3 files changed, 45 insertions(+), 13 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/beam-site/blob/5a7282fe/content/documentation/io/built-in/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/io/built-in/index.html b/content/documentation/io/built-in/index.html index 639a877..cf262d3 100644 --- a/content/documentation/io/built-in/index.html +++ b/content/documentation/io/built-in/index.html @@ -170,7 +170,7 @@ <td>Java</td> <td> <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java";>AvroIO</a></p> - <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hdfs";>Apache Hadoop HDFS</a></p> + <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system";>Apache Hadoop File System</a></p> <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java";>TextIO</a></p> <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/";>XML</a></p> </td> http://git-wip-us.apache.org/repos/asf/beam-site/blob/5a7282fe/content/documentation/programming-guide/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/programming-guide/index.html b/content/documentation/programming-guide/index.html index b57f89c..5ad4bea 100644 --- a/content/documentation/programming-guide/index.html +++ b/content/documentation/programming-guide/index.html @@ -1200,18 +1200,18 @@ guest, [[], [order4]] <span class="k">yield</span> <span class="n">element</span> <span class="k">else</span><span class="p">:</span> <span class="c"># Emit this word's long length to the 'above_cutoff_lengths' output.</span> - <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">OutputValue</span><span class="p">(</span> + <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">TaggedOutput</span><span class="p">(</span> <span class="s">'above_cutoff_lengths'</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">element</span><span class="p">))</span> <span class="k">if</span> <span class="n">element</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">marker</span><span class="p">):</span> <span class="c"># Emit this word to a different output with the 'marked strings' tag.</span> - <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">OutputValue</span><span class="p">(</span><span class="s">'marked strings'</span><span class="p">,</span> <span class="n">element</span><span class="p">)</span> + <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">TaggedOutput</span><span class="p">(</span><span class="s">'marked strings'</span><span class="p">,</span> <span class="n">element</span><span class="p">)</span> <span class="c"># Producing multiple outputs is also available in Map and FlatMap.</span> <span class="c"># Here is an example that uses FlatMap and shows that the tags do not need to be specified ahead of time.</span> <span class="k">def</span> <span class="nf">even_odd</span><span class="p">(</span><span class="n">x</span><span class="p">):</span> - <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">OutputValue</span><span class="p">(</span><span class="s">'odd'</span> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="k">else</span> <span class="s">'even'</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span> + <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">TaggedOutput</span><span class="p">(</span><span class="s">'odd'</span> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="k">else</span> <span class="s">'even'</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">10</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="k">yield</span> <span class="n">x</span> @@ -1267,7 +1267,17 @@ guest, [[], [order4]] </code></pre> </div> -<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="n">Python</span> <span class="n">code</span> <span class="n">snippet</span> <span class="n">coming</span> <span class="n">soon</span> <span class="p">(</span><span class="n">BEAM</span><span class="o">-</span><span class="mi">1926</span><span class="p">)</span> +<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># The CountWords Composite Transform inside the WordCount pipeline.</span> +<span class="k">class</span> <span class="nc">CountWords</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span> + + <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span> + <span class="k">return</span> <span class="p">(</span><span class="n">pcoll</span> + <span class="c"># Convert lines of text into individual words.</span> + <span class="o">|</span> <span class="s">'ExtractWords'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">ExtractWordsFn</span><span class="p">())</span> + <span class="c"># Count the number of times each word occurs.</span> + <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">combiners</span><span class="o">.</span><span class="n">Count</span><span class="o">.</span><span class="n">PerElement</span><span class="p">()</span> + <span class="c"># Format each word and count into a printable string.</span> + <span class="o">|</span> <span class="s">'FormatCounts'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">FormatCountsFn</span><span class="p">()))</span> </code></pre> </div> @@ -1286,7 +1296,10 @@ guest, [[], [order4]] </code></pre> </div> -<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="n">Python</span> <span class="n">code</span> <span class="n">snippet</span> <span class="n">coming</span> <span class="n">soon</span> <span class="p">(</span><span class="n">BEAM</span><span class="o">-</span><span class="mi">1926</span><span class="p">)</span> +<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ComputeWordLengths</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span> + <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span> + <span class="c"># transform logic goes here</span> + <span class="k">return</span> <span class="n">pcoll</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">))</span> </code></pre> </div> @@ -1307,7 +1320,10 @@ guest, [[], [order4]] </code></pre> </div> -<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="n">Python</span> <span class="n">code</span> <span class="n">snippet</span> <span class="n">coming</span> <span class="n">soon</span> <span class="p">(</span><span class="n">BEAM</span><span class="o">-</span><span class="mi">1926</span><span class="p">)</span> +<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ComputeWordLengths</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span> + <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span> + <span class="c"># transform logic goes here</span> + <span class="k">return</span> <span class="n">pcoll</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">))</span> </code></pre> </div> @@ -1920,7 +1936,7 @@ Subsequent transforms, however, are applied to the result of the <code class="hi </code></pre> </div> -<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="n">The</span> <span class="n">Beam</span> <span class="n">SDK</span> <span class="k">for</span> <span class="n">Python</span> <span class="n">does</span> <span class="ow">not</span> <span class="n">support</span> <span class="n">triggers</span><span class="o">.</span> +<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="c"># The Beam SDK for Python does not support triggers.</span> </code></pre> </div> @@ -1956,6 +1972,10 @@ Subsequent transforms, however, are applied to the result of the <code class="hi </code></pre> </div> +<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="c"># The Beam SDK for Python does not support triggers.</span> +</code></pre> +</div> + <p>This code sample sets a time-based trigger for a <code class="highlighter-rouge">PCollection</code>, which emits results one minute after the first element in that window has been processed. The last line in the code sample, <code class="highlighter-rouge">.discardingFiredPanes()</code>, is the windowâs <strong>accumulation mode</strong>.</p> <h4 id="window-accumulation-modes">Window Accumulation Modes</h4> @@ -2009,6 +2029,10 @@ Subsequent transforms, however, are applied to the result of the <code class="hi </code></pre> </div> +<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="c"># The Beam SDK for Python does not support triggers.</span> +</code></pre> +</div> + <p>This allowed lateness propagates to all <code class="highlighter-rouge">PCollection</code>s derived as a result of applying transforms to the original <code class="highlighter-rouge">PCollection</code>. If you want to change the allowed lateness later in your pipeline, you can apply <code class="highlighter-rouge">Window.withAllowedLateness()</code> again, explicitly.</p> <h3 id="a-namecomposite-triggersacomposite-triggers"><a name="composite-triggers"></a>Composite Triggers</h3> @@ -2055,6 +2079,10 @@ Subsequent transforms, however, are applied to the result of the <code class="hi </code></pre> </div> +<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="c"># The Beam SDK for Python does not support triggers.</span> +</code></pre> +</div> + <h4 id="other-composite-triggers">Other Composite Triggers</h4> <p>You can also build other sorts of composite triggers. The following example code shows a simple composite trigger that fires whenever the pane has at least 100 elements, or after a minute.</p> @@ -2065,6 +2093,10 @@ Subsequent transforms, however, are applied to the result of the <code class="hi </code></pre> </div> +<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="c"># The Beam SDK for Python does not support triggers.</span> +</code></pre> +</div> + </div> http://git-wip-us.apache.org/repos/asf/beam-site/blob/5a7282fe/content/documentation/sdks/python-type-safety/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/sdks/python-type-safety/index.html b/content/documentation/sdks/python-type-safety/index.html index 24900b8..cd364cf 100644 --- a/content/documentation/sdks/python-type-safety/index.html +++ b/content/documentation/sdks/python-type-safety/index.html @@ -167,7 +167,9 @@ <p>Consider the following example, in which <code class="highlighter-rouge">numbers</code> is a <code class="highlighter-rouge">PCollection</code> of <code class="highlighter-rouge">str</code> values:</p> -<div class="highlighter-rouge"><pre class="highlight"><code>numbers = p | beam.Create(['1', '2', '3']) +<div class="highlighter-rouge"><pre class="highlight"><code>p = TestPipeline(options=PipelineOptions(pipeline_type_check=True)) + +numbers = p | beam.Create(['1', '2', '3']) </code></pre> </div> @@ -189,8 +191,7 @@ <p>To specify type hints inline, use the methods <code class="highlighter-rouge">with_input_types</code> and <code class="highlighter-rouge">with_output_types</code>. The following example code declares an input type hint inline:</p> -<div class="highlighter-rouge"><pre class="highlight"><code>p.options.view_as(TypeOptions).pipeline_type_check = True -evens = numbers | beam.Filter(lambda x: x % 2 == 0).with_input_types(int) +<div class="highlighter-rouge"><pre class="highlight"><code>evens = numbers | beam.Filter(lambda x: x % 2 == 0).with_input_types(int) </code></pre> </div> @@ -284,8 +285,7 @@ p.run() <p>However, if you enable runtime type checking, the code passes at pipeline construction and fails at runtime. To enable runtime type checking, set the pipeline option <code class="highlighter-rouge">runtime_type_check</code> to <code class="highlighter-rouge">True</code>.</p> -<div class="highlighter-rouge"><pre class="highlight"><code>p.options.view_as(TypeOptions).runtime_type_check = True -p | beam.Create(['a']) | beam.Map(lambda x: 3).with_output_types(str) +<div class="highlighter-rouge"><pre class="highlight"><code>p | beam.Create(['a']) | beam.Map(lambda x: 3).with_output_types(str) p.run() </code></pre> </div>
