This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new edc7d7f Publishing website 2019/08/13 01:56:17 at commit 0c56384
edc7d7f is described below
commit edc7d7f25c73a64dffd274d770b6e0b8f609a205
Author: jenkins <[email protected]>
AuthorDate: Tue Aug 13 01:56:17 2019 +0000
Publishing website 2019/08/13 01:56:17 at commit 0c56384
---
.../python/elementwise/flatmap/index.html | 497 ++++++++++++++++++++-
1 file changed, 490 insertions(+), 7 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/flatmap/index.html
b/website/generated-content/documentation/transforms/python/elementwise/flatmap/index.html
index 1e349b0..0558852 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/flatmap/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/flatmap/index.html
@@ -437,7 +437,19 @@
<ul class="nav">
- <li><a href="#examples">Examples</a></li>
+ <li><a href="#examples">Examples</a>
+ <ul>
+ <li><a href="#example-1-flatmap-with-a-predefined-function">Example 1:
FlatMap with a predefined function</a></li>
+ <li><a href="#example-2-flatmap-with-a-function">Example 2: FlatMap with
a function</a></li>
+ <li><a href="#example-3-flatmap-with-a-lambda-function">Example 3:
FlatMap with a lambda function</a></li>
+ <li><a href="#example-4-flatmap-with-a-generator">Example 4: FlatMap
with a generator</a></li>
+ <li><a href="#example-5-flatmaptuple-for-key-value-pairs">Example 5:
FlatMapTuple for key-value pairs</a></li>
+ <li><a href="#example-6-flatmap-with-multiple-arguments">Example 6:
FlatMap with multiple arguments</a></li>
+ <li><a href="#example-7-flatmap-with-side-inputs-as-singletons">Example
7: FlatMap with side inputs as singletons</a></li>
+ <li><a href="#example-8-flatmap-with-side-inputs-as-iterators">Example
8: FlatMap with side inputs as iterators</a></li>
+ <li><a
href="#example-9-flatmap-with-side-inputs-as-dictionaries">Example 9: FlatMap
with side inputs as dictionaries</a></li>
+ </ul>
+ </li>
<li><a href="#related-transforms">Related transforms</a></li>
</ul>
@@ -460,20 +472,481 @@ limitations under the License.
-->
<h1 id="flatmap">FlatMap</h1>
-<table align="left">
- <a target="_blank" class="button"
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap";>
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+<table>
+ <td>
+ <a class="button" target="_blank"
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap";>
<img src="https://beam.apache.org/images/logos/sdks/python.png";
width="20px" height="20px" alt="Pydoc" />
- Pydoc
+ Pydoc
</a>
+ </td>
</table>
-<p><br />
-Applies a simple 1-to-many mapping function over each element in the
collection.
+<p><br /></p>
+
+<p>Applies a simple 1-to-many mapping function over each element in the
collection.
The many elements are flattened into the resulting collection.</p>
<h2 id="examples">Examples</h2>
-<p>See <a href="https://issues.apache.org/jira/browse/BEAM-7389";>BEAM-7389</a>
for updates.</p>
+
+<p>In the following examples, we create a pipeline with a <code
class="highlighter-rouge">PCollection</code> of produce with their icon, name,
and duration.
+Then, we apply <code class="highlighter-rouge">FlatMap</code> in multiple ways
to yield zero or more elements per each input element into the resulting <code
class="highlighter-rouge">PCollection</code>.</p>
+
+<p><code class="highlighter-rouge">FlatMap</code> accepts a function that
returns an <code class="highlighter-rouge">iterable</code>,
+where each of the output <code class="highlighter-rouge">iterable</code>’s
elements is an element of the resulting <code
class="highlighter-rouge">PCollection</code>.</p>
+
+<h3 id="example-1-flatmap-with-a-predefined-function">Example 1: FlatMap with
a predefined function</h3>
+
+<p>We use the function <code class="highlighter-rouge">str.split</code> which
takes a single <code class="highlighter-rouge">str</code> element and outputs a
<code class="highlighter-rouge">list</code> of <code
class="highlighter-rouge">str</code>s.
+This pipeline splits the input element using whitespaces, creating a list of
zero or more elements.</p>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span
class="kn">import</span> <span class="nn">apache_beam</span> <span
class="kn">as</span> <span class="nn">beam</span>
+
+<span class="k">with</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Pipeline</span><span class="p">()</span>
<span class="k">as</span> <span class="n">pipeline</span><span
class="p">:</span>
+ <span class="n">plants</span> <span class="o">=</span> <span
class="p">(</span>
+ <span class="n">pipeline</span>
+ <span class="o">|</span> <span class="s">'Gardening plants'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">Create</span><span class="p">([</span>
+ <span class="s">'🍓Strawberry 🥕Carrot 🍆Eggplant'</span><span
class="p">,</span>
+ <span class="s">'🍅Tomato 🥔Potato'</span><span class="p">,</span>
+ <span class="p">])</span>
+ <span class="o">|</span> <span class="s">'Split words'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span
class="nb">str</span><span class="o">.</span><span class="n">split</span><span
class="p">)</span>
+ <span class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Map</span><span class="p">(</span><span
class="k">print</span><span class="p">)</span>
+ <span class="p">)</span>
+</code></pre>
+</div>
+
+<p>Output <code class="highlighter-rouge">PCollection</code> after <code
class="highlighter-rouge">FlatMap</code>:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+]
+</code></pre>
+</div>
+
+<table>
+ <td>
+ <a class="button" target="_blank"
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py";>
+ <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png";
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+ </a>
+ </td>
+</table>
+<p><br /></p>
+
+<h3 id="example-2-flatmap-with-a-function">Example 2: FlatMap with a
function</h3>
+
+<p>We define a function <code class="highlighter-rouge">split_words</code>
which splits an input <code class="highlighter-rouge">str</code> element using
the delimiter <code class="highlighter-rouge">','</code> and outputs a <code
class="highlighter-rouge">list</code> of <code
class="highlighter-rouge">str</code>s.</p>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span
class="kn">import</span> <span class="nn">apache_beam</span> <span
class="kn">as</span> <span class="nn">beam</span>
+
+<span class="k">def</span> <span class="nf">split_words</span><span
class="p">(</span><span class="n">text</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">text</span><span
class="o">.</span><span class="n">split</span><span class="p">(</span><span
class="s">','</span><span class="p">)</span>
+
+<span class="k">with</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Pipeline</span><span class="p">()</span>
<span class="k">as</span> <span class="n">pipeline</span><span
class="p">:</span>
+ <span class="n">plants</span> <span class="o">=</span> <span
class="p">(</span>
+ <span class="n">pipeline</span>
+ <span class="o">|</span> <span class="s">'Gardening plants'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">Create</span><span class="p">([</span>
+ <span class="s">'🍓Strawberry,🥕Carrot,🍆Eggplant'</span><span
class="p">,</span>
+ <span class="s">'🍅Tomato,🥔Potato'</span><span class="p">,</span>
+ <span class="p">])</span>
+ <span class="o">|</span> <span class="s">'Split words'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span
class="n">split_words</span><span class="p">)</span>
+ <span class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Map</span><span class="p">(</span><span
class="k">print</span><span class="p">)</span>
+ <span class="p">)</span>
+</code></pre>
+</div>
+
+<p>Output <code class="highlighter-rouge">PCollection</code> after <code
class="highlighter-rouge">FlatMap</code>:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+]
+</code></pre>
+</div>
+
+<table>
+ <td>
+ <a class="button" target="_blank"
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py";>
+ <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png";
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+ </a>
+ </td>
+</table>
+<p><br /></p>
+
+<h3 id="example-3-flatmap-with-a-lambda-function">Example 3: FlatMap with a
lambda function</h3>
+
+<p>For this example, we want to flatten a <code
class="highlighter-rouge">PCollection</code> of lists of <code
class="highlighter-rouge">str</code>s into a <code
class="highlighter-rouge">PCollection</code> of <code
class="highlighter-rouge">str</code>s.
+Each input element is already an <code
class="highlighter-rouge">iterable</code>, where each element is what we want
in the resulting <code class="highlighter-rouge">PCollection</code>.
+We use a lambda function that returns the same input element it received.</p>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span
class="kn">import</span> <span class="nn">apache_beam</span> <span
class="kn">as</span> <span class="nn">beam</span>
+
+<span class="k">with</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Pipeline</span><span class="p">()</span>
<span class="k">as</span> <span class="n">pipeline</span><span
class="p">:</span>
+ <span class="n">plants</span> <span class="o">=</span> <span
class="p">(</span>
+ <span class="n">pipeline</span>
+ <span class="o">|</span> <span class="s">'Gardening plants'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">Create</span><span class="p">([</span>
+ <span class="p">[</span><span class="s">'🍓Strawberry'</span><span
class="p">,</span> <span class="s">'🥕Carrot'</span><span class="p">,</span>
<span class="s">'🍆Eggplant'</span><span class="p">],</span>
+ <span class="p">[</span><span class="s">'🍅Tomato'</span><span
class="p">,</span> <span class="s">'🥔Potato'</span><span class="p">],</span>
+ <span class="p">])</span>
+ <span class="o">|</span> <span class="s">'Flatten lists'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span
class="k">lambda</span> <span class="n">elements</span><span class="p">:</span>
<span class="n">elements</span><span class="p">)</span>
+ <span class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Map</span><span class="p">(</span><span
class="k">print</span><span class="p">)</span>
+ <span class="p">)</span>
+</code></pre>
+</div>
+
+<p>Output <code class="highlighter-rouge">PCollection</code> after <code
class="highlighter-rouge">FlatMap</code>:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+]
+</code></pre>
+</div>
+
+<table>
+ <td>
+ <a class="button" target="_blank"
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py";>
+ <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png";
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+ </a>
+ </td>
+</table>
+<p><br /></p>
+
+<h3 id="example-4-flatmap-with-a-generator">Example 4: FlatMap with a
generator</h3>
+
+<p>For this example, we want to flatten a <code
class="highlighter-rouge">PCollection</code> of lists of <code
class="highlighter-rouge">str</code>s into a <code
class="highlighter-rouge">PCollection</code> of <code
class="highlighter-rouge">str</code>s.
+We use a generator to iterate over the input list and yield each of the
elements.
+Each yielded result in the generator is an element in the resulting <code
class="highlighter-rouge">PCollection</code>.</p>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span
class="kn">import</span> <span class="nn">apache_beam</span> <span
class="kn">as</span> <span class="nn">beam</span>
+
+<span class="k">def</span> <span class="nf">generate_elements</span><span
class="p">(</span><span class="n">elements</span><span class="p">):</span>
+ <span class="k">for</span> <span class="n">element</span> <span
class="ow">in</span> <span class="n">elements</span><span class="p">:</span>
+ <span class="k">yield</span> <span class="n">element</span>
+
+<span class="k">with</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Pipeline</span><span class="p">()</span>
<span class="k">as</span> <span class="n">pipeline</span><span
class="p">:</span>
+ <span class="n">plants</span> <span class="o">=</span> <span
class="p">(</span>
+ <span class="n">pipeline</span>
+ <span class="o">|</span> <span class="s">'Gardening plants'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">Create</span><span class="p">([</span>
+ <span class="p">[</span><span class="s">'🍓Strawberry'</span><span
class="p">,</span> <span class="s">'🥕Carrot'</span><span class="p">,</span>
<span class="s">'🍆Eggplant'</span><span class="p">],</span>
+ <span class="p">[</span><span class="s">'🍅Tomato'</span><span
class="p">,</span> <span class="s">'🥔Potato'</span><span class="p">],</span>
+ <span class="p">])</span>
+ <span class="o">|</span> <span class="s">'Flatten lists'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span
class="n">generate_elements</span><span class="p">)</span>
+ <span class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Map</span><span class="p">(</span><span
class="k">print</span><span class="p">)</span>
+ <span class="p">)</span>
+</code></pre>
+</div>
+
+<p>Output <code class="highlighter-rouge">PCollection</code> after <code
class="highlighter-rouge">FlatMap</code>:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+]
+</code></pre>
+</div>
+
+<table>
+ <td>
+ <a class="button" target="_blank"
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py";>
+ <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png";
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+ </a>
+ </td>
+</table>
+<p><br /></p>
+
+<h3 id="example-5-flatmaptuple-for-key-value-pairs">Example 5: FlatMapTuple
for key-value pairs</h3>
+
+<p>If your <code class="highlighter-rouge">PCollection</code> consists of
<code class="highlighter-rouge">(key, value)</code> pairs,
+you can use <code class="highlighter-rouge">FlatMapTuple</code> to unpack them
into different function arguments.</p>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span
class="kn">import</span> <span class="nn">apache_beam</span> <span
class="kn">as</span> <span class="nn">beam</span>
+
+<span class="k">def</span> <span class="nf">format_plant</span><span
class="p">(</span><span class="n">icon</span><span class="p">,</span> <span
class="n">plant</span><span class="p">):</span>
+ <span class="k">if</span> <span class="n">icon</span><span class="p">:</span>
+ <span class="k">yield</span> <span class="s">'{}{}'</span><span
class="o">.</span><span class="n">format</span><span class="p">(</span><span
class="n">icon</span><span class="p">,</span> <span class="n">plant</span><span
class="p">)</span>
+
+<span class="k">with</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Pipeline</span><span class="p">()</span>
<span class="k">as</span> <span class="n">pipeline</span><span
class="p">:</span>
+ <span class="n">plants</span> <span class="o">=</span> <span
class="p">(</span>
+ <span class="n">pipeline</span>
+ <span class="o">|</span> <span class="s">'Gardening plants'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">Create</span><span class="p">([</span>
+ <span class="p">(</span><span class="s">'🍓'</span><span
class="p">,</span> <span class="s">'Strawberry'</span><span class="p">),</span>
+ <span class="p">(</span><span class="s">'🥕'</span><span
class="p">,</span> <span class="s">'Carrot'</span><span class="p">),</span>
+ <span class="p">(</span><span class="s">'🍆'</span><span
class="p">,</span> <span class="s">'Eggplant'</span><span class="p">),</span>
+ <span class="p">(</span><span class="s">'🍅'</span><span
class="p">,</span> <span class="s">'Tomato'</span><span class="p">),</span>
+ <span class="p">(</span><span class="s">'🥔'</span><span
class="p">,</span> <span class="s">'Potato'</span><span class="p">),</span>
+ <span class="p">(</span><span class="bp">None</span><span
class="p">,</span> <span class="s">'Invalid'</span><span class="p">),</span>
+ <span class="p">])</span>
+ <span class="o">|</span> <span class="s">'Format'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">FlatMapTuple</span><span
class="p">(</span><span class="n">format_plant</span><span class="p">)</span>
+ <span class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Map</span><span class="p">(</span><span
class="k">print</span><span class="p">)</span>
+ <span class="p">)</span>
+</code></pre>
+</div>
+
+<p>Output <code class="highlighter-rouge">PCollection</code> after <code
class="highlighter-rouge">FlatMapTuple</code>:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+]
+</code></pre>
+</div>
+
+<table>
+ <td>
+ <a class="button" target="_blank"
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py";>
+ <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png";
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+ </a>
+ </td>
+</table>
+<p><br /></p>
+
+<h3 id="example-6-flatmap-with-multiple-arguments">Example 6: FlatMap with
multiple arguments</h3>
+
+<p>You can pass functions with multiple arguments to <code
class="highlighter-rouge">FlatMap</code>.
+They are passed as additional positional arguments or keyword arguments to the
function.</p>
+
+<p>In this example, <code class="highlighter-rouge">split_words</code> takes
<code class="highlighter-rouge">text</code> and <code
class="highlighter-rouge">delimiter</code> as arguments.</p>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span
class="kn">import</span> <span class="nn">apache_beam</span> <span
class="kn">as</span> <span class="nn">beam</span>
+
+<span class="k">def</span> <span class="nf">split_words</span><span
class="p">(</span><span class="n">text</span><span class="p">,</span> <span
class="n">delimiter</span><span class="o">=</span><span
class="bp">None</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">text</span><span
class="o">.</span><span class="n">split</span><span class="p">(</span><span
class="n">delimiter</span><span class="p">)</span>
+
+<span class="k">with</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Pipeline</span><span class="p">()</span>
<span class="k">as</span> <span class="n">pipeline</span><span
class="p">:</span>
+ <span class="n">plants</span> <span class="o">=</span> <span
class="p">(</span>
+ <span class="n">pipeline</span>
+ <span class="o">|</span> <span class="s">'Gardening plants'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">Create</span><span class="p">([</span>
+ <span class="s">'🍓Strawberry,🥕Carrot,🍆Eggplant'</span><span
class="p">,</span>
+ <span class="s">'🍅Tomato,🥔Potato'</span><span class="p">,</span>
+ <span class="p">])</span>
+ <span class="o">|</span> <span class="s">'Split words'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span
class="n">split_words</span><span class="p">,</span> <span
class="n">delimiter</span><span class="o">=</span><span
class="s">','</span><span class="p">)</span>
+ <span class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Map</span><span class="p">(</span><span
class="k">print</span><span class="p">)</span>
+ <span class="p">)</span>
+</code></pre>
+</div>
+
+<p>Output <code class="highlighter-rouge">PCollection</code> after <code
class="highlighter-rouge">FlatMap</code>:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+]
+</code></pre>
+</div>
+
+<table>
+ <td>
+ <a class="button" target="_blank"
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py";>
+ <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png";
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+ </a>
+ </td>
+</table>
+<p><br /></p>
+
+<h3 id="example-7-flatmap-with-side-inputs-as-singletons">Example 7: FlatMap
with side inputs as singletons</h3>
+
+<p>If the <code class="highlighter-rouge">PCollection</code> has a single
value, such as the average from another computation,
+passing the <code class="highlighter-rouge">PCollection</code> as a
<em>singleton</em> accesses that value.</p>
+
+<p>In this example, we pass a <code
class="highlighter-rouge">PCollection</code> the value <code
class="highlighter-rouge">','</code> as a singleton.
+We then use that value as the delimiter for the <code
class="highlighter-rouge">str.split</code> method.</p>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span
class="kn">import</span> <span class="nn">apache_beam</span> <span
class="kn">as</span> <span class="nn">beam</span>
+
+<span class="k">with</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Pipeline</span><span class="p">()</span>
<span class="k">as</span> <span class="n">pipeline</span><span
class="p">:</span>
+ <span class="n">delimiter</span> <span class="o">=</span> <span
class="n">pipeline</span> <span class="o">|</span> <span class="s">'Create
delimiter'</span> <span class="o">>></span> <span
class="n">beam</span><span class="o">.</span><span class="n">Create</span><span
class="p">([</span><span class="s">','</span><span class="p">])</span>
+
+ <span class="n">plants</span> <span class="o">=</span> <span
class="p">(</span>
+ <span class="n">pipeline</span>
+ <span class="o">|</span> <span class="s">'Gardening plants'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">Create</span><span class="p">([</span>
+ <span class="s">'🍓Strawberry,🥕Carrot,🍆Eggplant'</span><span
class="p">,</span>
+ <span class="s">'🍅Tomato,🥔Potato'</span><span class="p">,</span>
+ <span class="p">])</span>
+ <span class="o">|</span> <span class="s">'Split words'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">FlatMap</span><span class="p">(</span>
+ <span class="k">lambda</span> <span class="n">text</span><span
class="p">,</span> <span class="n">delimiter</span><span class="p">:</span>
<span class="n">text</span><span class="o">.</span><span
class="n">split</span><span class="p">(</span><span
class="n">delimiter</span><span class="p">),</span>
+ <span class="n">delimiter</span><span class="o">=</span><span
class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span
class="o">.</span><span class="n">AsSingleton</span><span
class="p">(</span><span class="n">delimiter</span><span class="p">),</span>
+ <span class="p">)</span>
+ <span class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Map</span><span class="p">(</span><span
class="k">print</span><span class="p">)</span>
+ <span class="p">)</span>
+</code></pre>
+</div>
+
+<p>Output <code class="highlighter-rouge">PCollection</code> after <code
class="highlighter-rouge">FlatMap</code>:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+]
+</code></pre>
+</div>
+
+<table>
+ <td>
+ <a class="button" target="_blank"
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py";>
+ <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png";
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+ </a>
+ </td>
+</table>
+<p><br /></p>
+
+<h3 id="example-8-flatmap-with-side-inputs-as-iterators">Example 8: FlatMap
with side inputs as iterators</h3>
+
+<p>If the <code class="highlighter-rouge">PCollection</code> has multiple
values, pass the <code class="highlighter-rouge">PCollection</code> as an
<em>iterator</em>.
+This accesses elements lazily as they are needed,
+so it is possible to iterate over large <code
class="highlighter-rouge">PCollection</code>s that won’t fit into memory.</p>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span
class="kn">import</span> <span class="nn">apache_beam</span> <span
class="kn">as</span> <span class="nn">beam</span>
+
+<span class="k">def</span> <span
class="nf">normalize_and_validate_durations</span><span class="p">(</span><span
class="n">plant</span><span class="p">,</span> <span
class="n">valid_durations</span><span class="p">):</span>
+ <span class="n">plant</span><span class="p">[</span><span
class="s">'duration'</span><span class="p">]</span> <span class="o">=</span>
<span class="n">plant</span><span class="p">[</span><span
class="s">'duration'</span><span class="p">]</span><span
class="o">.</span><span class="n">lower</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">plant</span><span
class="p">[</span><span class="s">'duration'</span><span class="p">]</span>
<span class="ow">in</span> <span class="n">valid_durations</span><span
class="p">:</span>
+ <span class="k">yield</span> <span class="n">plant</span>
+
+<span class="k">with</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Pipeline</span><span class="p">()</span>
<span class="k">as</span> <span class="n">pipeline</span><span
class="p">:</span>
+ <span class="n">valid_durations</span> <span class="o">=</span> <span
class="n">pipeline</span> <span class="o">|</span> <span class="s">'Valid
durations'</span> <span class="o">>></span> <span
class="n">beam</span><span class="o">.</span><span class="n">Create</span><span
class="p">([</span>
+ <span class="s">'annual'</span><span class="p">,</span>
+ <span class="s">'biennial'</span><span class="p">,</span>
+ <span class="s">'perennial'</span><span class="p">,</span>
+ <span class="p">])</span>
+
+ <span class="n">valid_plants</span> <span class="o">=</span> <span
class="p">(</span>
+ <span class="n">pipeline</span>
+ <span class="o">|</span> <span class="s">'Gardening plants'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">Create</span><span class="p">([</span>
+ <span class="p">{</span><span class="s">'icon'</span><span
class="p">:</span> <span class="s">'🍓'</span><span class="p">,</span> <span
class="s">'name'</span><span class="p">:</span> <span
class="s">'Strawberry'</span><span class="p">,</span> <span
class="s">'duration'</span><span class="p">:</span> <span
class="s">'Perennial'</span><span class="p">},</span>
+ <span class="p">{</span><span class="s">'icon'</span><span
class="p">:</span> <span class="s">'🥕'</span><span class="p">,</span> <span
class="s">'name'</span><span class="p">:</span> <span
class="s">'Carrot'</span><span class="p">,</span> <span
class="s">'duration'</span><span class="p">:</span> <span
class="s">'BIENNIAL'</span><span class="p">},</span>
+ <span class="p">{</span><span class="s">'icon'</span><span
class="p">:</span> <span class="s">'🍆'</span><span class="p">,</span> <span
class="s">'name'</span><span class="p">:</span> <span
class="s">'Eggplant'</span><span class="p">,</span> <span
class="s">'duration'</span><span class="p">:</span> <span
class="s">'perennial'</span><span class="p">},</span>
+ <span class="p">{</span><span class="s">'icon'</span><span
class="p">:</span> <span class="s">'🍅'</span><span class="p">,</span> <span
class="s">'name'</span><span class="p">:</span> <span
class="s">'Tomato'</span><span class="p">,</span> <span
class="s">'duration'</span><span class="p">:</span> <span
class="s">'annual'</span><span class="p">},</span>
+ <span class="p">{</span><span class="s">'icon'</span><span
class="p">:</span> <span class="s">'🥔'</span><span class="p">,</span> <span
class="s">'name'</span><span class="p">:</span> <span
class="s">'Potato'</span><span class="p">,</span> <span
class="s">'duration'</span><span class="p">:</span> <span
class="s">'unknown'</span><span class="p">},</span>
+ <span class="p">])</span>
+ <span class="o">|</span> <span class="s">'Normalize and validate
durations'</span> <span class="o">>></span> <span
class="n">beam</span><span class="o">.</span><span
class="n">FlatMap</span><span class="p">(</span>
+ <span class="n">normalize_and_validate_durations</span><span
class="p">,</span>
+ <span class="n">valid_durations</span><span class="o">=</span><span
class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span
class="o">.</span><span class="n">AsIter</span><span class="p">(</span><span
class="n">valid_durations</span><span class="p">),</span>
+ <span class="p">)</span>
+ <span class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Map</span><span class="p">(</span><span
class="k">print</span><span class="p">)</span>
+ <span class="p">)</span>
+</code></pre>
+</div>
+
+<p>Output <code class="highlighter-rouge">PCollection</code> after <code
class="highlighter-rouge">FlatMap</code>:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>valid_plants = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+]
+</code></pre>
+</div>
+
+<table>
+ <td>
+ <a class="button" target="_blank"
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py";>
+ <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png";
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+ </a>
+ </td>
+</table>
+<p><br /></p>
+
+<blockquote>
+ <p><strong>Note</strong>: You can pass the <code
class="highlighter-rouge">PCollection</code> as a <em>list</em> with <code
class="highlighter-rouge">beam.pvalue.AsList(pcollection)</code>,
+but this requires that all the elements fit into memory.</p>
+</blockquote>
+
+<h3 id="example-9-flatmap-with-side-inputs-as-dictionaries">Example 9: FlatMap
with side inputs as dictionaries</h3>
+
+<p>If a <code class="highlighter-rouge">PCollection</code> is small enough to
fit into memory, then that <code class="highlighter-rouge">PCollection</code>
can be passed as a <em>dictionary</em>.
+Each element must be a <code class="highlighter-rouge">(key, value)</code>
pair.
+Note that all the elements of the <code
class="highlighter-rouge">PCollection</code> must fit into memory for this.
+If the <code class="highlighter-rouge">PCollection</code> won’t fit into
memory, use <code
class="highlighter-rouge">beam.pvalue.AsIter(pcollection)</code> instead.</p>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span
class="kn">import</span> <span class="nn">apache_beam</span> <span
class="kn">as</span> <span class="nn">beam</span>
+
+<span class="k">def</span> <span
class="nf">replace_duration_if_valid</span><span class="p">(</span><span
class="n">plant</span><span class="p">,</span> <span
class="n">durations</span><span class="p">):</span>
+ <span class="k">if</span> <span class="n">plant</span><span
class="p">[</span><span class="s">'duration'</span><span class="p">]</span>
<span class="ow">in</span> <span class="n">durations</span><span
class="p">:</span>
+ <span class="n">plant</span><span class="p">[</span><span
class="s">'duration'</span><span class="p">]</span> <span class="o">=</span>
<span class="n">durations</span><span class="p">[</span><span
class="n">plant</span><span class="p">[</span><span
class="s">'duration'</span><span class="p">]]</span>
+ <span class="k">yield</span> <span class="n">plant</span>
+
+<span class="k">with</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Pipeline</span><span class="p">()</span>
<span class="k">as</span> <span class="n">pipeline</span><span
class="p">:</span>
+ <span class="n">durations</span> <span class="o">=</span> <span
class="n">pipeline</span> <span class="o">|</span> <span class="s">'Durations
dict'</span> <span class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">Create</span><span class="p">([</span>
+ <span class="p">(</span><span class="mi">0</span><span
class="p">,</span> <span class="s">'annual'</span><span class="p">),</span>
+ <span class="p">(</span><span class="mi">1</span><span
class="p">,</span> <span class="s">'biennial'</span><span class="p">),</span>
+ <span class="p">(</span><span class="mi">2</span><span
class="p">,</span> <span class="s">'perennial'</span><span class="p">),</span>
+ <span class="p">])</span>
+
+ <span class="n">valid_plants</span> <span class="o">=</span> <span
class="p">(</span>
+ <span class="n">pipeline</span>
+ <span class="o">|</span> <span class="s">'Gardening plants'</span> <span
class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">Create</span><span class="p">([</span>
+ <span class="p">{</span><span class="s">'icon'</span><span
class="p">:</span> <span class="s">'🍓'</span><span class="p">,</span> <span
class="s">'name'</span><span class="p">:</span> <span
class="s">'Strawberry'</span><span class="p">,</span> <span
class="s">'duration'</span><span class="p">:</span> <span
class="mi">2</span><span class="p">},</span>
+ <span class="p">{</span><span class="s">'icon'</span><span
class="p">:</span> <span class="s">'🥕'</span><span class="p">,</span> <span
class="s">'name'</span><span class="p">:</span> <span
class="s">'Carrot'</span><span class="p">,</span> <span
class="s">'duration'</span><span class="p">:</span> <span
class="mi">1</span><span class="p">},</span>
+ <span class="p">{</span><span class="s">'icon'</span><span
class="p">:</span> <span class="s">'🍆'</span><span class="p">,</span> <span
class="s">'name'</span><span class="p">:</span> <span
class="s">'Eggplant'</span><span class="p">,</span> <span
class="s">'duration'</span><span class="p">:</span> <span
class="mi">2</span><span class="p">},</span>
+ <span class="p">{</span><span class="s">'icon'</span><span
class="p">:</span> <span class="s">'🍅'</span><span class="p">,</span> <span
class="s">'name'</span><span class="p">:</span> <span
class="s">'Tomato'</span><span class="p">,</span> <span
class="s">'duration'</span><span class="p">:</span> <span
class="mi">0</span><span class="p">},</span>
+ <span class="p">{</span><span class="s">'icon'</span><span
class="p">:</span> <span class="s">'🥔'</span><span class="p">,</span> <span
class="s">'name'</span><span class="p">:</span> <span
class="s">'Potato'</span><span class="p">,</span> <span
class="s">'duration'</span><span class="p">:</span> <span
class="o">-</span><span class="mi">1</span><span class="p">},</span>
+ <span class="p">])</span>
+ <span class="o">|</span> <span class="s">'Replace duration if
valid'</span> <span class="o">>></span> <span class="n">beam</span><span
class="o">.</span><span class="n">FlatMap</span><span class="p">(</span>
+ <span class="n">replace_duration_if_valid</span><span
class="p">,</span>
+ <span class="n">durations</span><span class="o">=</span><span
class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span
class="o">.</span><span class="n">AsDict</span><span class="p">(</span><span
class="n">durations</span><span class="p">),</span>
+ <span class="p">)</span>
+ <span class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">Map</span><span class="p">(</span><span
class="k">print</span><span class="p">)</span>
+ <span class="p">)</span>
+</code></pre>
+</div>
+
+<p>Output <code class="highlighter-rouge">PCollection</code> after <code
class="highlighter-rouge">FlatMap</code>:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>valid_plants = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+]
+</code></pre>
+</div>
+
+<table>
+ <td>
+ <a class="button" target="_blank"
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py";>
+ <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png";
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+ </a>
+ </td>
+</table>
+<p><br /></p>
<h2 id="related-transforms">Related transforms</h2>
+
<ul>
<li><a href="/documentation/transforms/python/elementwise/filter">Filter</a>
is useful if the function is just
deciding whether to output an element or not.</li>
@@ -482,6 +955,16 @@ operation, and includes other abilities such as multiple
output collections and
<li><a href="/documentation/transforms/python/elementwise/map">Map</a>
behaves the same, but produces exactly one output for each input.</li>
</ul>
+<table>
+ <td>
+ <a class="button" target="_blank"
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap";>
+ <img src="https://beam.apache.org/images/logos/sdks/python.png";
width="20px" height="20px" alt="Pydoc" />
+ Pydoc
+ </a>
+ </td>
+</table>
+<p><br /></p>
+
</div>
</div>
<!--
![https://beam.apache.org/images/logos/sdks/python.png"](https://beam.apache.org/images/logos/sdks/python.png"){kind=link}
![https://www.tensorflow.org/images/GitHub-Mark-32px.png"](https://www.tensorflow.org/images/GitHub-Mark-32px.png"){kind=link}