Repository: incubator-beam-site Updated Branches: refs/heads/asf-site d6f8ca78e -> 92fc27503
Addition of PCollection blog post; blog fix Adding links to 5/4 meeting docs Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/697f4388 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/697f4388 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/697f4388 Branch: refs/heads/asf-site Commit: 697f4388c82bd63e2e6f32b1fb4c6f910e1146e6 Parents: d6f8ca7 Author: James Malone <[email protected]> Authored: Fri May 13 16:39:10 2016 -0700 Committer: James Malone <[email protected]> Committed: Wed May 18 21:01:16 2016 -0700 ---------------------------------------------------------------------- _data/authors.yml | 4 + _data/meetings.yml | 9 +- _includes/authors-list.md | 2 +- ...016-05-13-where-is-my-pcollection-dot-map.md | 89 ++++++++ .../2016/03/17/capability-matrix.html | 2 +- .../2016/04/03/presentation-materials.html | 2 +- .../05/13/where-is-my-pcollection-dot-map.html | 212 +++++++++++++++++++ .../sdk/2016/02/25/python-sdk-now-public.html | 2 +- .../website/2016/02/22/beam-has-a-logo.html | 2 +- content/blog/index.html | 16 ++ content/capability-matrix/index.html | 2 +- content/contribution-guide/index.html | 4 +- content/feed.xml | 110 +++++++++- .../images/blog/simple-wordcount-pipeline.png | Bin 0 -> 322878 bytes content/index.html | 2 + content/public-meetings/index.html | 8 +- images/blog/simple-wordcount-pipeline.png | Bin 0 -> 322878 bytes 17 files changed, 447 insertions(+), 19 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/_data/authors.yml ---------------------------------------------------------------------- diff --git a/_data/authors.yml b/_data/authors.yml index b71ad8a..ce732c7 100644 --- a/_data/authors.yml +++ b/_data/authors.yml @@ -10,3 +10,7 @@ takidau: name: Tyler Akidau email: [email protected] twitter: takidau +robertwb: + name: Robert Bradshaw + email: [email protected] + twitter: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/_data/meetings.yml ---------------------------------------------------------------------- diff --git a/_data/meetings.yml b/_data/meetings.yml index c12111c..f2efe9f 100644 --- a/_data/meetings.yml +++ b/_data/meetings.yml @@ -16,6 +16,11 @@ events: location: Virtual type: Technical Deep Dive materials: - notes: Virtual attendance options will<br>be announced prior to the meeting. + - title: Presentation - Beam Community Meeting + link: "https://drive.google.com/open?id=17i7SHViboWtLEZw27iabdMisPl987WWxvapJaXg_dEE"; -last_updated: 2016/04/14 + - title: Notes - Beam Community Meeting + link: "https://drive.google.com/open?id=1szhEE_pfhEtrQye61jXAidUcMW7oebZCRc2InUe3ou0"; + notes: + +last_updated: 2016/05/16 http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/_includes/authors-list.md ---------------------------------------------------------------------- diff --git a/_includes/authors-list.md b/_includes/authors-list.md index 1207445..fb2808e 100644 --- a/_includes/authors-list.md +++ b/_includes/authors-list.md @@ -1 +1 @@ -{% assign count = authors | size %}{% for name in authors %}{% if forloop.first == false and count > 2 %},{% endif %}{% if forloop.last and count > 1 %} &{% endif %}{% assign author = site.data.authors[name] %} {{ author.name }} [<a href="https://twitter.com/{{ author.twitter }}">@{{ author.twitter }}</a>]{% endfor %} +{% assign count = authors | size %}{% for name in authors %}{% if forloop.first == false and count > 2 %},{% endif %}{% if forloop.last and count > 1 %} &{% endif %}{% assign author = site.data.authors[name] %} {{ author.name }} {% if author.twitter %}[<a href="https://twitter.com/{{ author.twitter }}">@{{ author.twitter }}</a>]{% endif %}{% endfor %} http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/_posts/2016-05-13-where-is-my-pcollection-dot-map.md ---------------------------------------------------------------------- diff --git a/_posts/2016-05-13-where-is-my-pcollection-dot-map.md b/_posts/2016-05-13-where-is-my-pcollection-dot-map.md new file mode 100644 index 0000000..3193ea1 --- /dev/null +++ b/_posts/2016-05-13-where-is-my-pcollection-dot-map.md @@ -0,0 +1,89 @@ +--- +layout: post +title: "Where's my PCollection.map()?" +date: 2016-05-13 11:00:00 -0700 +excerpt_separator: <!--more--> +categories: beam model pcollection +authors: + - robertwb +--- +Have you ever wondered why Beam has PTransforms for everything instead of having methods on PCollection? Take a look at the history that led to this (and other) design decisions. + +<!--more--> + +Though Beam is relatively new, its design draws heavily on many years of experience with real-world pipelines. One of the primary inspirations is FlumeJava, which is Google's internal successor to MapReduce first introduced in 2009. + +The original FlumeJava API has methods like `count` and `parallelDo` on the `PCollection`s. Though slightly more succinct, this approach has many disadvantages to extensibility. Every new user to FlumeJava wanted to add transforms, and adding them as methods to `PCollection` simply doesn't scale well. In contrast, a PCollection in Beam has a single `apply` method which takes any PTransform as an argument. + +<table class="table"> + <tr> + <th>FlumeJava</th> + <th>Beam</th> + </tr> + <tr> + <td><pre> +PCollection<T> input = ⦠+PCollection<O> output = input.count() + .parallelDo(...); + </pre></td> + <td><pre> +PCollection<T> input = ⦠+PCollection<O> output = input.apply(Count.perElement()) + .apply(ParDo.of(...)); + </pre></td> + </tr> +</table> + +This is a more scalable approach for several reasons. + +## Where to draw the line? +Adding methods to `PCollection` forces a line to be drawn between operations that are "useful" enough to merit this special treatment and those that are not. It is easy to make the case for flat map, group by key, and combine per key. But what about filter? Count? Approximate count? Approximate quantiles? Most frequent? WriteToMyFavoriteSource? Going too far down this path leads to a single enormous class that contains nearly everything one could want to do. (FlumeJava's PCollection class is over 5000 lines long with around 70 distinct operations, and it could have been *much* larger had we accepted every proposal.) Furthermore, since Java doesnât allow adding methods to a class, there is a sharp divide syntactically between those operations that are added to `PCollection` and those that arenât. A traditional way to share code is with a library of functions, but functions (in traditional languages like Java at least) are written prefix-style, which doesn't mix well with the flue nt builder style (e.g. `input.operation1().operation2().operation3()` vs. `operation3(operation1(input).operation2())`). + +Instead in Beam we've chosen a style that places all transforms--whether they be primitive operations, composite operations bundled in the SDK, or part of an external library--on equal footing. This also facilitates alternative implementations (which may even take different options) that are easily interchangeable. + +<table class="table"> + <tr> + <th>FlumeJava</th> + <th>Beam</th> + </tr> + <tr> + <td><pre> +PCollection<O> output = + ExternalLibrary.doStuff( + MyLibrary.transform(input, myArgs) + .parallelDo(...), + externalLibArgs); + </pre></td> + <td><pre> +PCollection<O> output = input + .apply(MyLibrary.transform(myArgs)) + .apply(ParDo.of(...)) + .apply(ExternalLibrary.doStuff(externalLibArgs)); + + </pre></td> + </tr> +</table> + +## Configurability +While it makes for a more fluent style to let values (PCollections) be the objects passed around and manipulated (i.e. the handles to the deferred execution graph), it is the operations themselves that need to be significantly more composable, configurable, and extendable. Using methods doesn't scale well here, especially in a language without default or keyword arguments. For example, a ParDo operation can have any number of side inputs and side outputs, or a write operation may have configurations dealing with encoding and compression. One option is to separate these out into separate overloads or even methods, but that exacerbates the problems above. (FlumeJava evolved over a dozen overloads of the `parallelDo` method!) Another option is to pass a configuration object that can be built up using more fluent idioms like the builder pattern, but at that point one might as well make the configuration object the operation itself, which is what Beam does. + +## Type Safety +Many operations can only be applied to collections whose elements are of a specific type. For example, the GroupByKey operation should only apply to PCollection<KV<K, V>>s. In Java at least, it's not possible to restrict methods based on the element type parameter alone. In FlumeJava, this led us to add a PTable<K, V> subclassing PCollection<KV<K, V>> to contain all the operations specific to PCollections of key-value pairs. This leads to the same question of which T's are special enough to merit being captured by PCollection subclasses. It is not very extensible for third parties and often requires manual downcasts/conversions (which can't be safely chained in Java) and special operations that produce these PCollection specializations. + +This is particularly inconvenient for transforms that produce outputs whose element types are the same as (or related to) their input's element types, requiring extra support to generate the right subclasses (e.g. a filter on a PTable<K, V> should produce another PTable<K, V> rather than just a raw PCollection of key-value pairs). + +Using PTransforms allows us to sidestep this entire issue. We can place arbitrary constraints on the context in which a transform may be used based on the type of its inputs; for instance GroupByKey is statically typed to only apply to a PCollection<KV<K, V>>. The way this happens is generalizable to arbitrary shapes, without needing to introduce specialized types like PTable. + +## Reusability and Structure +Though PTransforms are generally constructed at the site at which they're used, by pulling them out as separate objects one is able to store them and pass them around. + +As pipelines grow and evolve, it is useful to structure your pipeline into modular, often reusable components, and PTransforms allow one to do this nicely in a data-processing pipeline. In addition, modular PTransforms also expose to the system what the logical structure of your code (e.g. for monitoring). Of the three different representations of the WordCount pipeline below, only the structured view captures the high-level intent of the pipeline. Letting even the simple operations be PTransforms means there's less of an abrupt edge to packaging things up into composite operations. + +<img class="center-block" src="{{ "/images/blog/simple-wordcount-pipeline.png" | prepend: site.baseurl }}" alt="Three different visualizations of a simple WordCount pipeline" width="500"> + +<div class="text-center"> +<i>Three different visualizations of a simple WordCount pipeline which computes the number of occurrences of every work in a set of text files. The flag view gives the full DAG of all operations performed. The execution view groups operations according to how they're executed, e.g. after performing runner-specific optimizations like function composition. The structured view nests operations according to their grouping in PTransforms.</i> +</div> + +## Summary +Although it's tempting to add methods to PCollections, such an approach is not scalable, extensible, or sufficiently expressive. Putting a single apply() method on PCollection and all the logic into the operation itself lets us have the best of both worlds, and avoids hard cliffs of complexity by having a single consistent style across simple and complex pipelines, and between predefined and user-defined operations. http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/content/beam/capability/2016/03/17/capability-matrix.html ---------------------------------------------------------------------- diff --git a/content/beam/capability/2016/03/17/capability-matrix.html b/content/beam/capability/2016/03/17/capability-matrix.html index d256c17..3fd82e3 100644 --- a/content/beam/capability/2016/03/17/capability-matrix.html +++ b/content/beam/capability/2016/03/17/capability-matrix.html @@ -99,7 +99,7 @@ <header class="post-header"> <h1 class="post-title" itemprop="name headline">Clarifying & Formalizing Runner Capabilities</h1> - <p class="post-meta"><time datetime="2016-03-17T19:00:00+01:00" itemprop="datePublished">Mar 17, 2016</time> ⢠Frances Perry [<a href="https://twitter.com/francesjperry";>@francesjperry</a>] & Tyler Akidau [<a href="https://twitter.com/takidau";>@takidau</a>] + <p class="post-meta"><time datetime="2016-03-17T11:00:00-07:00" itemprop="datePublished">Mar 17, 2016</time> ⢠Frances Perry [<a href="https://twitter.com/francesjperry";>@francesjperry</a>] & Tyler Akidau [<a href="https://twitter.com/takidau";>@takidau</a>] </p> </header> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/content/beam/capability/2016/04/03/presentation-materials.html ---------------------------------------------------------------------- diff --git a/content/beam/capability/2016/04/03/presentation-materials.html b/content/beam/capability/2016/04/03/presentation-materials.html index cc06cb3..9b27c7d 100644 --- a/content/beam/capability/2016/04/03/presentation-materials.html +++ b/content/beam/capability/2016/04/03/presentation-materials.html @@ -99,7 +99,7 @@ <header class="post-header"> <h1 class="post-title" itemprop="name headline">Apache Beam Presentation Materials</h1> - <p class="post-meta"><time datetime="2016-04-03T20:00:00+02:00" itemprop="datePublished">Apr 3, 2016</time> ⢠Frances Perry [<a href="https://twitter.com/francesjperry";>@francesjperry</a>] & Tyler Akidau [<a href="https://twitter.com/takidau";>@takidau</a>] + <p class="post-meta"><time datetime="2016-04-03T11:00:00-07:00" itemprop="datePublished">Apr 3, 2016</time> ⢠Frances Perry [<a href="https://twitter.com/francesjperry";>@francesjperry</a>] & Tyler Akidau [<a href="https://twitter.com/takidau";>@takidau</a>] </p> </header> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/content/beam/model/pcollection/2016/05/13/where-is-my-pcollection-dot-map.html ---------------------------------------------------------------------- diff --git a/content/beam/model/pcollection/2016/05/13/where-is-my-pcollection-dot-map.html b/content/beam/model/pcollection/2016/05/13/where-is-my-pcollection-dot-map.html new file mode 100644 index 0000000..0017365 --- /dev/null +++ b/content/beam/model/pcollection/2016/05/13/where-is-my-pcollection-dot-map.html @@ -0,0 +1,212 @@ +<!DOCTYPE html> +<html lang="en"> + + <head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + + <title>Where's my PCollection.map()?</title> + <meta name="description" content="Have you ever wondered why Beam has PTransforms for everything instead of having methods on PCollection? Take a look at the history that led to this (and oth..."> + + <link rel="stylesheet" href="/styles/site.css"> + <link rel="stylesheet" href="/css/theme.css"> + <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";></script> + <script src="/js/bootstrap.min.js"></script> + <link rel="canonical" href="http://beam.incubator.apache.org/beam/model/pcollection/2016/05/13/where-is-my-pcollection-dot-map.html";> + <link rel="alternate" type="application/rss+xml" title="Apache Beam (incubating)" href="http://beam.incubator.apache.org/feed.xml";> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-73650088-1', 'auto'); + ga('send', 'pageview'); + + </script> + <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico"> +</head> + + + <body role="document"> + + <nav class="navbar navbar-default navbar-fixed-top"> + <div class="container"> + <div class="navbar-header"> + <a href="/" class="navbar-brand" > + <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png"> + </a> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + </div> + <div id="navbar" class="navbar-collapse collapse"> + <ul class="nav navbar-nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li class="dropdown-header">Guides</li> + <li><a href="/getting_started/">Getting Started</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Technical Documentation</li> + <li><a href="/capability-matrix/">Capability Matrix</a></li> + <li><a href="https://goo.gl/ps8twC";>Technical Docs</a></li> + <li><a href="https://goo.gl/nk5OM0";>Technical Vision</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Community <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li class="dropdown-header">Community</li> + <li><a href="/mailing_lists/">Mailing Lists</a></li> + <li><a href="/team/">Apache Beam Team</a></li> + <li><a href="/public-meetings/">Public Meetings</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Contribute</li> + <li><a href="/contribution-guide/">Contribution Guide</a></li> + <li><a href="/source_repository/">Source Repository</a></li> + <li><a href="/issue_tracking/">Issue Tracking</a></li> + </ul> + </li> + <li><a href="/blog">Blog</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Project <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/presentation-materials/">Presentation Materials</a></li> + <li><a href="/material/">Logos and design</a></li> + <li><a href="http://apache.org/licenses/LICENSE-2.0.html";>License</a></li> + </ul> + </li> + </ul> + </div><!--/.nav-collapse --> + </div> +</nav> + + +<link rel="stylesheet" href=""> + + + <div class="container" role="main"> + + <div class="container"> + + +<article class="post" itemscope itemtype="http://schema.org/BlogPosting";> + + <header class="post-header"> + <h1 class="post-title" itemprop="name headline">Where's my PCollection.map()?</h1> + <p class="post-meta"><time datetime="2016-05-13T11:00:00-07:00" itemprop="datePublished">May 13, 2016</time> ⢠Robert Bradshaw +</p> + </header> + + <div class="post-content" itemprop="articleBody"> + <p>Have you ever wondered why Beam has PTransforms for everything instead of having methods on PCollection? Take a look at the history that led to this (and other) design decisions.</p> + +<!--more--> + +<p>Though Beam is relatively new, its design draws heavily on many years of experience with real-world pipelines. One of the primary inspirations is FlumeJava, which is Googleâs internal successor to MapReduce first introduced in 2009.</p> + +<p>The original FlumeJava API has methods like <code class="highlighter-rouge">count</code> and <code class="highlighter-rouge">parallelDo</code> on the <code class="highlighter-rouge">PCollection</code>s. Though slightly more succinct, this approach has many disadvantages to extensibility. Every new user to FlumeJava wanted to add transforms, and adding them as methods to <code class="highlighter-rouge">PCollection</code> simply doesnât scale well. In contrast, a PCollection in Beam has a single <code class="highlighter-rouge">apply</code> method which takes any PTransform as an argument.</p> + +<table class="table"> + <tr> + <th>FlumeJava</th> + <th>Beam</th> + </tr> + <tr> + <td><pre> +PCollection<T> input = ⦠+PCollection<O> output = input.count() + .parallelDo(...); + </pre></td> + <td><pre> +PCollection<T> input = ⦠+PCollection<O> output = input.apply(Count.perElement()) + .apply(ParDo.of(...)); + </pre></td> + </tr> +</table> + +<p>This is a more scalable approach for several reasons.</p> + +<h2 id="where-to-draw-the-line">Where to draw the line?</h2> +<p>Adding methods to <code class="highlighter-rouge">PCollection</code> forces a line to be drawn between operations that are âusefulâ enough to merit this special treatment and those that are not. It is easy to make the case for flat map, group by key, and combine per key. But what about filter? Count? Approximate count? Approximate quantiles? Most frequent? WriteToMyFavoriteSource? Going too far down this path leads to a single enormous class that contains nearly everything one could want to do. (FlumeJavaâs PCollection class is over 5000 lines long with around 70 distinct operations, and it could have been <em>much</em> larger had we accepted every proposal.) Furthermore, since Java doesnât allow adding methods to a class, there is a sharp divide syntactically between those operations that are added to <code class="highlighter-rouge">PCollection</code> and those that arenât. A traditional way to share code is with a library of functions, but functions (in traditional la nguages like Java at least) are written prefix-style, which doesnât mix well with the fluent builder style (e.g. <code class="highlighter-rouge">input.operation1().operation2().operation3()</code> vs. <code class="highlighter-rouge">operation3(operation1(input).operation2())</code>).</p> + +<p>Instead in Beam weâve chosen a style that places all transformsâwhether they be primitive operations, composite operations bundled in the SDK, or part of an external libraryâon equal footing. This also facilitates alternative implementations (which may even take different options) that are easily interchangeable.</p> + +<table class="table"> + <tr> + <th>FlumeJava</th> + <th>Beam</th> + </tr> + <tr> + <td><pre> +PCollection<O> output = + ExternalLibrary.doStuff( + MyLibrary.transform(input, myArgs) + .parallelDo(...), + externalLibArgs); + </pre></td> + <td><pre> +PCollection<O> output = input + .apply(MyLibrary.transform(myArgs)) + .apply(ParDo.of(...)) + .apply(ExternalLibrary.doStuff(externalLibArgs)); + + </pre></td> + </tr> +</table> + +<h2 id="configurability">Configurability</h2> +<p>While it makes for a more fluent style to let values (PCollections) be the objects passed around and manipulated (i.e. the handles to the deferred execution graph), it is the operations themselves that need to be significantly more composable, configurable, and extendable. Using methods doesnât scale well here, especially in a language without default or keyword arguments. For example, a ParDo operation can have any number of side inputs and side outputs, or a write operation may have configurations dealing with encoding and compression. One option is to separate these out into separate overloads or even methods, but that exacerbates the problems above. (FlumeJava evolved over a dozen overloads of the <code class="highlighter-rouge">parallelDo</code> method!) Another option is to pass a configuration object that can be built up using more fluent idioms like the builder pattern, but at that point one might as well make the configuration object the operation itself, which is what Beam does.</p> + +<h2 id="type-safety">Type Safety</h2> +<p>Many operations can only be applied to collections whose elements are of a specific type. For example, the GroupByKey operation should only apply to PCollection<KV<K, V»s. In Java at least, itâs not possible to restrict methods based on the element type parameter alone. In FlumeJava, this led us to add a PTable<K, V> subclassing PCollection<KV<K, V» to contain all the operations specific to PCollections of key-value pairs. This leads to the same question of which Tâs are special enough to merit being captured by PCollection subclasses. It is not very extensible for third parties and often requires manual downcasts/conversions (which canât be safely chained in Java) and special operations that produce these PCollection specializations.</p> + +<p>This is particularly inconvenient for transforms that produce outputs whose element types are the same as (or related to) their inputâs element types, requiring extra support to generate the right subclasses (e.g. a filter on a PTable<K, V> should produce another PTable<K, V> rather than just a raw PCollection of key-value pairs).</p> + +<p>Using PTransforms allows us to sidestep this entire issue. We can place arbitrary constraints on the context in which a transform may be used based on the type of its inputs; for instance GroupByKey is statically typed to only apply to a PCollection<KV<K, V». The way this happens is generalizable to arbitrary shapes, without needing to introduce specialized types like PTable.</p> + +<h2 id="reusability-and-structure">Reusability and Structure</h2> +<p>Though PTransforms are generally constructed at the site at which theyâre used, by pulling them out as separate objects one is able to store them and pass them around.</p> + +<p>As pipelines grow and evolve, it is useful to structure your pipeline into modular, often reusable components, and PTransforms allow one to do this nicely in a data-processing pipeline. In addition, modular PTransforms also expose to the system what the logical structure of your code (e.g. for monitoring). Of the three different representations of the WordCount pipeline below, only the structured view captures the high-level intent of the pipeline. Letting even the simple operations be PTransforms means thereâs less of an abrupt edge to packaging things up into composite operations.</p> + +<p><img class="center-block" src="/images/blog/simple-wordcount-pipeline.png" alt="Three different visualizations of a simple WordCount pipeline" width="500" /></p> + +<div class="text-center"> +<i>Three different visualizations of a simple WordCount pipeline which computes the number of occurrences of every work in a set of text files. The flag view gives the full DAG of all operations performed. The execution view groups operations according to how they're executed, e.g. after performing runner-specific optimizations like function composition. The structured view nests operations according to their grouping in PTransforms.</i> +</div> + +<h2 id="summary">Summary</h2> +<p>Although itâs tempting to add methods to PCollections, such an approach is not scalable, extensible, or sufficiently expressive. Putting a single apply() method on PCollection and all the logic into the operation itself lets us have the best of both worlds, and avoids hard cliffs of complexity by having a single consistent style across simple and complex pipelines, and between predefined and user-defined operations.</p> + + </div> + +</article> + + </div> + + + <hr> + <div class="row"> + <div class="col-xs-12"> + <footer> + <p class="text-center">© Copyright 2016 + <a href="http://www.apache.org";>The Apache Software Foundation.</a> All Rights Reserved.</p> + <p class="text-center"><a href="/privacy_policy">Privacy Policy</a> | + <a href="/feed.xml">RSS Feed</a></p> + </footer> + </div> + </div> + <!-- container div end --> +</div> + + + </body> + +</html> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html ---------------------------------------------------------------------- diff --git a/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html b/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html index 5d3bd02..62ffa48 100644 --- a/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html +++ b/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html @@ -99,7 +99,7 @@ <header class="post-header"> <h1 class="post-title" itemprop="name headline">Dataflow Python SDK is now public!</h1> - <p class="post-meta"><time datetime="2016-02-25T22:00:00+01:00" itemprop="datePublished">Feb 25, 2016</time> ⢠James Malone [<a href="https://twitter.com/chimerasaurus";>@chimerasaurus</a>] + <p class="post-meta"><time datetime="2016-02-25T13:00:00-08:00" itemprop="datePublished">Feb 25, 2016</time> ⢠James Malone [<a href="https://twitter.com/chimerasaurus";>@chimerasaurus</a>] </p> </header> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/content/beam/update/website/2016/02/22/beam-has-a-logo.html ---------------------------------------------------------------------- diff --git a/content/beam/update/website/2016/02/22/beam-has-a-logo.html b/content/beam/update/website/2016/02/22/beam-has-a-logo.html index edeb841..bd996dd 100644 --- a/content/beam/update/website/2016/02/22/beam-has-a-logo.html +++ b/content/beam/update/website/2016/02/22/beam-has-a-logo.html @@ -99,7 +99,7 @@ <header class="post-header"> <h1 class="post-title" itemprop="name headline">Apache Beam has a logo!</h1> - <p class="post-meta"><time datetime="2016-02-22T19:21:48+01:00" itemprop="datePublished">Feb 22, 2016</time> ⢠James Malone [<a href="https://twitter.com/chimerasaurus";>@chimerasaurus</a>] + <p class="post-meta"><time datetime="2016-02-22T10:21:48-08:00" itemprop="datePublished">Feb 22, 2016</time> ⢠James Malone [<a href="https://twitter.com/chimerasaurus";>@chimerasaurus</a>] </p> </header> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/content/blog/index.html ---------------------------------------------------------------------- diff --git a/content/blog/index.html b/content/blog/index.html index 9672b54..41862bd 100644 --- a/content/blog/index.html +++ b/content/blog/index.html @@ -104,6 +104,22 @@ <p>This is the blog for the Apache Beam project. This blog contains news and updates for the project.</p> +<h3 id="a-classpost-link-hrefbeammodelpcollection20160513where-is-my-pcollection-dot-maphtmlwheres-my-pcollectionmapa"><a class="post-link" href="/beam/model/pcollection/2016/05/13/where-is-my-pcollection-dot-map.html">Whereâs my PCollection.map()?</a></h3> +<p><i>May 13, 2016 ⢠Robert Bradshaw +</i></p> + +<p>Have you ever wondered why Beam has PTransforms for everything instead of having methods on PCollection? Take a look at the history that led to this (and other) design decisions.</p> + +<!-- Render a "read more" button if the post is longer than the excerpt --> + +<p> +<a class="btn btn-default btn-sm" href="/beam/model/pcollection/2016/05/13/where-is-my-pcollection-dot-map.html#read-more" role="button"> +Read more <span class="glyphicon glyphicon-menu-right" aria-hidden="true"></span> +</a> +</p> + +<hr /> + <h3 id="a-classpost-link-hrefbeamcapability20160403presentation-materialshtmlapache-beam-presentation-materialsa"><a class="post-link" href="/beam/capability/2016/04/03/presentation-materials.html">Apache Beam Presentation Materials</a></h3> <p><i>Apr 3, 2016 ⢠Frances Perry [<a href="https://twitter.com/francesjperry";>@francesjperry</a>] & Tyler Akidau [<a href="https://twitter.com/takidau";>@takidau</a>] </i></p> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/content/capability-matrix/index.html ---------------------------------------------------------------------- diff --git a/content/capability-matrix/index.html b/content/capability-matrix/index.html index 5886bf4..0940360 100644 --- a/content/capability-matrix/index.html +++ b/content/capability-matrix/index.html @@ -95,7 +95,7 @@ <div class="container"> <h1 id="apache-beam-capability-matrix">Apache Beam Capability Matrix</h1> -<p><span style="font-size:11px;float:none">Last updated: 2016-05-05 23:15 CEST</span></p> +<p><span style="font-size:11px;float:none">Last updated: 2016-05-16 09:11 PDT</span></p> <p>Apache Beam (incubating) provides a portable API layer for building sophisticated data-parallel processing engines that may be executed across a diversity of exeuction engines, or <i>runners</i>. The core concepts of this layer are based upon the Beam Model (formerly referred to as the <a href="http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf";>Dataflow Model</a>), and implemented to varying degrees in each Beam runner. To help clarify the capabilities of individual runners, weâve created the capability matrix below.</p> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/content/contribution-guide/index.html ---------------------------------------------------------------------- diff --git a/content/contribution-guide/index.html b/content/contribution-guide/index.html index 1d71e8b..65ee67d 100644 --- a/content/contribution-guide/index.html +++ b/content/contribution-guide/index.html @@ -356,8 +356,8 @@ github https://github.com/apache/incubator-beam.git (push) <p>Fetch references from all remote repositories, and checkout the specific pull request branch.</p> <pre> -</code>$ git fetch --all -$ git checkout -b finish-pr-<b><pull-request-#></b> github/pr/<b><pull-request-#></b></code></pre> +$ git fetch --all +$ git checkout -b finish-pr-<b><pull-request-#></b> github/pr/<b><pull-request-#></b></pre> <p>At this point, you can commit any final touches to the pull request. For example, you should:</p> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/content/feed.xml ---------------------------------------------------------------------- diff --git a/content/feed.xml b/content/feed.xml index 909a7c7..841bac2 100644 --- a/content/feed.xml +++ b/content/feed.xml @@ -6,9 +6,105 @@ </description> <link>http://beam.incubator.apache.org/</link> <atom:link href="http://beam.incubator.apache.org/feed.xml"; rel="self" type="application/rss+xml"/> - <pubDate>Thu, 05 May 2016 23:15:35 +0200</pubDate> - <lastBuildDate>Thu, 05 May 2016 23:15:35 +0200</lastBuildDate> - <generator>Jekyll v3.1.3</generator> + <pubDate>Mon, 16 May 2016 09:11:00 -0700</pubDate> + <lastBuildDate>Mon, 16 May 2016 09:11:00 -0700</lastBuildDate> + <generator>Jekyll v3.1.2</generator> + + <item> + <title>Where's my PCollection.map()?</title> + <description><p>Have you ever wondered why Beam has PTransforms for everything instead of having methods on PCollection? Take a look at the history that led to this (and other) design decisions.</p> + +<!--more--> + +<p>Though Beam is relatively new, its design draws heavily on many years of experience with real-world pipelines. One of the primary inspirations is FlumeJava, which is Googleâs internal successor to MapReduce first introduced in 2009.</p> + +<p>The original FlumeJava API has methods like <code class="highlighter-rouge">count</code> and <code class="highlighter-rouge">parallelDo</code> on the <code class="highlighter-rouge">PCollection</code>s. Though slightly more succinct, this approach has many disadvantages to extensibility. Every new user to FlumeJava wanted to add transforms, and adding them as methods to <code class="highlighter-rouge">PCollection</code> simply doesnât scale well. In contrast, a PCollection in Beam has a single <code class="highlighter-rouge">apply</code> method which takes any PTransform as an argument.</p> + +<table class="table"> + <tr> + <th>FlumeJava</th> + <th>Beam</th> + </tr> + <tr> + <td><pre> +PCollection&lt;T&gt; input = ⦠+PCollection&lt;O&gt; output = input.count() + .parallelDo(...); + </pre></td> + <td><pre> +PCollection&lt;T&gt; input = ⦠+PCollection&lt;O&gt; output = input.apply(Count.perElement()) + .apply(ParDo.of(...)); + </pre></td> + </tr> +</table> + +<p>This is a more scalable approach for several reasons.</p> + +<h2 id="where-to-draw-the-line">Where to draw the line?</h2> +<p>Adding methods to <code class="highlighter-rouge">PCollection</code> forces a line to be drawn between operations that are âusefulâ enough to merit this special treatment and those that are not. It is easy to make the case for flat map, group by key, and combine per key. But what about filter? Count? Approximate count? Approximate quantiles? Most frequent? WriteToMyFavoriteSource? Going too far down this path leads to a single enormous class that contains nearly everything one could want to do. (FlumeJavaâs PCollection class is over 5000 lines long with around 70 distinct operations, and it could have been <em>much</em> larger had we accepted every proposal.) Furthermore, since Java doesnât allow adding methods to a class, there is a sharp divide syntactically between those operations that are added to <code class="highlighter-rouge">PCollection</code> and those that arenât. A traditional way to share code is with a library of functions, but functions (in traditional languages like Java at least) are written prefix-style, which doesnât mix well with the fluent builder style (e.g. <code class="highlighter-rouge">input.operation1().operation2().operation3()</code> vs. <code class="highlighter-rouge">operation3(operation1(input).operation2())</code>).</p> + +<p>Instead in Beam weâve chosen a style that places all transformsâwhether they be primitive operations, composite operations bundled in the SDK, or part of an external libraryâon equal footing. This also facilitates alternative implementations (which may even take different options) that are easily interchangeable.</p> + +<table class="table"> + <tr> + <th>FlumeJava</th> + <th>Beam</th> + </tr> + <tr> + <td><pre> +PCollection&lt;O&gt; output = + ExternalLibrary.doStuff( + MyLibrary.transform(input, myArgs) + .parallelDo(...), + externalLibArgs); + </pre></td> + <td><pre> +PCollection&lt;O&gt; output = input + .apply(MyLibrary.transform(myArgs)) + .apply(ParDo.of(...)) + .apply(ExternalLibrary.doStuff(externalLibArgs)); + &nbsp; + </pre></td> + </tr> +</table> + +<h2 id="configurability">Configurability</h2> +<p>While it makes for a more fluent style to let values (PCollections) be the objects passed around and manipulated (i.e. the handles to the deferred execution graph), it is the operations themselves that need to be significantly more composable, configurable, and extendable. Using methods doesnât scale well here, especially in a language without default or keyword arguments. For example, a ParDo operation can have any number of side inputs and side outputs, or a write operation may have configurations dealing with encoding and compression. One option is to separate these out into separate overloads or even methods, but that exacerbates the problems above. (FlumeJava evolved over a dozen overloads of the <code class="highlighter-rouge">parallelDo</code> method!) Another option is to pass a configuration object that can be built up using more fluent idioms like the builder pattern, but at that point one might as well make the configuration object the ope ration itself, which is what Beam does.</p> + +<h2 id="type-safety">Type Safety</h2> +<p>Many operations can only be applied to collections whose elements are of a specific type. For example, the GroupByKey operation should only apply to PCollection&lt;KV&lt;K, V»s. In Java at least, itâs not possible to restrict methods based on the element type parameter alone. In FlumeJava, this led us to add a PTable&lt;K, V&gt; subclassing PCollection&lt;KV&lt;K, V» to contain all the operations specific to PCollections of key-value pairs. This leads to the same question of which Tâs are special enough to merit being captured by PCollection subclasses. It is not very extensible for third parties and often requires manual downcasts/conversions (which canât be safely chained in Java) and special operations that produce these PCollection specializations.</p> + +<p>This is particularly inconvenient for transforms that produce outputs whose element types are the same as (or related to) their inputâs element types, requiring extra support to generate the right subclasses (e.g. a filter on a PTable&lt;K, V&gt; should produce another PTable&lt;K, V&gt; rather than just a raw PCollection of key-value pairs).</p> + +<p>Using PTransforms allows us to sidestep this entire issue. We can place arbitrary constraints on the context in which a transform may be used based on the type of its inputs; for instance GroupByKey is statically typed to only apply to a PCollection&lt;KV&lt;K, V». The way this happens is generalizable to arbitrary shapes, without needing to introduce specialized types like PTable.</p> + +<h2 id="reusability-and-structure">Reusability and Structure</h2> +<p>Though PTransforms are generally constructed at the site at which theyâre used, by pulling them out as separate objects one is able to store them and pass them around.</p> + +<p>As pipelines grow and evolve, it is useful to structure your pipeline into modular, often reusable components, and PTransforms allow one to do this nicely in a data-processing pipeline. In addition, modular PTransforms also expose to the system what the logical structure of your code (e.g. for monitoring). Of the three different representations of the WordCount pipeline below, only the structured view captures the high-level intent of the pipeline. Letting even the simple operations be PTransforms means thereâs less of an abrupt edge to packaging things up into composite operations.</p> + +<p><img class="center-block" src="/images/blog/simple-wordcount-pipeline.png" alt="Three different visualizations of a simple WordCount pipeline" width="500" /></p> + +<div class="text-center"> +<i>Three different visualizations of a simple WordCount pipeline which computes the number of occurrences of every work in a set of text files. The flag view gives the full DAG of all operations performed. The execution view groups operations according to how they're executed, e.g. after performing runner-specific optimizations like function composition. The structured view nests operations according to their grouping in PTransforms.</i> +</div> + +<h2 id="summary">Summary</h2> +<p>Although itâs tempting to add methods to PCollections, such an approach is not scalable, extensible, or sufficiently expressive. Putting a single apply() method on PCollection and all the logic into the operation itself lets us have the best of both worlds, and avoids hard cliffs of complexity by having a single consistent style across simple and complex pipelines, and between predefined and user-defined operations.</p> +</description> + <pubDate>Fri, 13 May 2016 11:00:00 -0700</pubDate> + <link>http://beam.incubator.apache.org/beam/model/pcollection/2016/05/13/where-is-my-pcollection-dot-map.html</link> + <guid isPermaLink="true">http://beam.incubator.apache.org/beam/model/pcollection/2016/05/13/where-is-my-pcollection-dot-map.html</guid> + + + <category>beam</category> + + <category>model</category> + + <category>pcollection</category> + + </item> <item> <title>Apache Beam Presentation Materials</title> @@ -34,7 +130,7 @@ <p>As Apache Beam grows, so will this repository of presentation materials. We are excited to add new materials as the Apache Beam ecosystem grows with new runners, SDKs, and so on. If you are interested in contributing content or have a request, please see the <a href="/presentation-materials/">Apache Beam presentation materials</a> page or email the <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#064;&#098;&#101;&#097;&#109;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;"><code class="highlighter-rouge">[email protected]</code></a> mailing list with your ideas or questions.</p> </description> - <pubDate>Sun, 03 Apr 2016 20:00:00 +0200</pubDate> + <pubDate>Sun, 03 Apr 2016 11:00:00 -0700</pubDate> <link>http://beam.incubator.apache.org/beam/capability/2016/04/03/presentation-materials.html</link> <guid isPermaLink="true">http://beam.incubator.apache.org/beam/capability/2016/04/03/presentation-materials.html</guid> @@ -831,7 +927,7 @@ </div> </description> - <pubDate>Thu, 17 Mar 2016 19:00:00 +0100</pubDate> + <pubDate>Thu, 17 Mar 2016 11:00:00 -0700</pubDate> <link>http://beam.incubator.apache.org/beam/capability/2016/03/17/capability-matrix.html</link> <guid isPermaLink="true">http://beam.incubator.apache.org/beam/capability/2016/03/17/capability-matrix.html</guid> @@ -857,7 +953,7 @@ included the <a href="https://github.com/GoogleCloudPlatform/DataflowJav <p>Apache Beam Team</p> </description> - <pubDate>Thu, 25 Feb 2016 22:00:00 +0100</pubDate> + <pubDate>Thu, 25 Feb 2016 13:00:00 -0800</pubDate> <link>http://beam.incubator.apache.org/beam/python/sdk/2016/02/25/python-sdk-now-public.html</link> <guid isPermaLink="true">http://beam.incubator.apache.org/beam/python/sdk/2016/02/25/python-sdk-now-public.html</guid> @@ -891,7 +987,7 @@ will also make various permutations and resolutions of this logo available in th coming weeks. For any questions or comments, send an email to the <code class="highlighter-rouge">dev@</code> email list for Apache Beam.</p> </description> - <pubDate>Mon, 22 Feb 2016 19:21:48 +0100</pubDate> + <pubDate>Mon, 22 Feb 2016 10:21:48 -0800</pubDate> <link>http://beam.incubator.apache.org/beam/update/website/2016/02/22/beam-has-a-logo.html</link> <guid isPermaLink="true">http://beam.incubator.apache.org/beam/update/website/2016/02/22/beam-has-a-logo.html</guid> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/content/images/blog/simple-wordcount-pipeline.png ---------------------------------------------------------------------- diff --git a/content/images/blog/simple-wordcount-pipeline.png b/content/images/blog/simple-wordcount-pipeline.png new file mode 100644 index 0000000..a15cb7e Binary files /dev/null and b/content/images/blog/simple-wordcount-pipeline.png differ http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/content/index.html ---------------------------------------------------------------------- diff --git a/content/index.html b/content/index.html index 7e9f1a8..02100b1 100644 --- a/content/index.html +++ b/content/index.html @@ -177,6 +177,8 @@ <h3>Blog</h3> <div class="list-group"> + <a class="list-group-item" href="/beam/model/pcollection/2016/05/13/where-is-my-pcollection-dot-map.html">May 13, 2016 - Where's my PCollection.map()?</a> + <a class="list-group-item" href="/beam/capability/2016/04/03/presentation-materials.html">Apr 3, 2016 - Apache Beam Presentation Materials</a> <a class="list-group-item" href="/beam/capability/2016/03/17/capability-matrix.html">Mar 17, 2016 - Clarifying & Formalizing Runner Capabilities</a> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/content/public-meetings/index.html ---------------------------------------------------------------------- diff --git a/content/public-meetings/index.html b/content/public-meetings/index.html index 9011c31..2f6216c 100644 --- a/content/public-meetings/index.html +++ b/content/public-meetings/index.html @@ -139,12 +139,16 @@ <td>Technical Deep Dive</td> <td> + <div><a href="https://drive.google.com/open?id=17i7SHViboWtLEZw27iabdMisPl987WWxvapJaXg_dEE";>Presentation - Beam Community Meeting</a></div> + + <div><a href="https://drive.google.com/open?id=1szhEE_pfhEtrQye61jXAidUcMW7oebZCRc2InUe3ou0";>Notes - Beam Community Meeting</a></div> + </td> - <td>Virtual attendance options will<br />be announced prior to the meeting.</td> + <td></td> </tr> </table> -<p><em>This list was last updated on 2016/04/14.</em></p> +<p><em>This list was last updated on 2016/05/16.</em></p> <p>All Apache Beam meetings are open to the public and we encourage anyone to attend. From time to time space in our event location may be limited, so preference will be given to PPMC members and others on a first-come, first-serve basis.</p> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/images/blog/simple-wordcount-pipeline.png ---------------------------------------------------------------------- diff --git a/images/blog/simple-wordcount-pipeline.png b/images/blog/simple-wordcount-pipeline.png new file mode 100644 index 0000000..a15cb7e Binary files /dev/null and b/images/blog/simple-wordcount-pipeline.png differ
