http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/6fce44a2/content/capability-matrix/index.html ---------------------------------------------------------------------- diff --git a/content/capability-matrix/index.html b/content/capability-matrix/index.html index ab1c64c..06992fd 100644 --- a/content/capability-matrix/index.html +++ b/content/capability-matrix/index.html @@ -50,10 +50,14 @@ <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a> <ul class="dropdown-menu"> + <li class="dropdown-header">Releases</li> + <li><a href="/releases/">Release information</a></li> + <li role="separator" class="divider"></li> <li class="dropdown-header">Guides</li> <li><a href="/getting_started/">Getting Started</a></li> <li role="separator" class="divider"></li> <li class="dropdown-header">Technical Documentation</li> + <li><a href="/javadoc/0.1.0-incubating/">Java API Reference</a></li> <li><a href="/capability-matrix/">Capability Matrix</a></li> <li><a href="https://goo.gl/ps8twC";>Technical Docs</a></li> <li><a href="https://goo.gl/nk5OM0";>Technical Vision</a></li> @@ -95,11 +99,7 @@ <div class="container"> <h1 id="apache-beam-capability-matrix">Apache Beam Capability Matrix</h1> -<<<<<<< 23feafc0bb80d835bfd9a05a1d98b0c997aafc84 -<p><span style="font-size:11px;float:none">Last updated: 2016-05-27 09:58 PDT</span></p> -======= -<p><span style="font-size:11px;float:none">Last updated: 2016-05-27 09:51 PDT</span></p> ->>>>>>> Update date for blog post +<p><span style="font-size:11px;float:none">Last updated: 2016-06-14 18:36 PDT</span></p> <p>Apache Beam (incubating) provides a portable API layer for building sophisticated data-parallel processing engines that may be executed across a diversity of exeuction engines, or <i>runners</i>. The core concepts of this layer are based upon the Beam Model (formerly referred to as the <a href="http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf";>Dataflow Model</a>), and implemented to varying degrees in each Beam runner. To help clarify the capabilities of individual runners, weâve created the capability matrix below.</p> @@ -157,23 +157,19 @@ - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> </tr> @@ -182,23 +178,19 @@ - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><b><center>~</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"></td> </tr> @@ -207,23 +199,19 @@ - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> </tr> @@ -232,23 +220,19 @@ - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> </tr> @@ -257,23 +241,19 @@ - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><b><center>~</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><b><center>~</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><b><center>~</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"></td> </tr> @@ -282,23 +262,19 @@ - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><b><center>~ (<a href="https://issues.apache.org/jira/browse/BEAM-102";>BEAM-102</a>)</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><b><center>~</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"></td> </tr> @@ -307,23 +283,19 @@ - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"></td> </tr> @@ -332,23 +304,19 @@ - <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><b><center>~</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><b><center>~</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><b><center>~</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><b><center>~</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"></td> </tr> @@ -357,23 +325,19 @@ - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#ca1"><b><center>✕ (<a href="https://issues.apache.org/jira/browse/BEAM-25";>BEAM-25</a>)</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#ca1"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#ca1"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#ca1"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#ca1"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#ca1"></td> </tr> @@ -402,23 +366,19 @@ - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> </tr> @@ -427,23 +387,19 @@ - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#8cf;border-color:#37d"><b><center>~</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8cf;border-color:#37d"></td> </tr> @@ -452,23 +408,19 @@ - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#8cf;border-color:#37d"><b><center>~</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8cf;border-color:#37d"></td> </tr> @@ -477,23 +429,19 @@ - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"></td> </tr> @@ -502,23 +450,19 @@ - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"></td> </tr> @@ -527,23 +471,19 @@ - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"></td> </tr> @@ -552,23 +492,19 @@ - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"></td> </tr> @@ -597,23 +533,19 @@ - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -622,23 +554,19 @@ - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -647,23 +575,19 @@ - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> </tr> @@ -672,23 +596,19 @@ - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -697,23 +617,19 @@ - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕ (<a href="https://issues.apache.org/jira/browse/BEAM-101";>BEAM-101</a>)</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -722,23 +638,19 @@ - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -747,23 +659,19 @@ - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -772,23 +680,19 @@ - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕ (<a href="https://issues.apache.org/jira/browse/BEAM-27";>BEAM-27</a>)</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -817,23 +721,19 @@ - <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"></td> - <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"></td> - <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"></td> - <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"></td> </tr> @@ -842,23 +742,19 @@ - <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"></td> - <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"></td> - <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><b><center>✓</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"></td> </tr> @@ -867,23 +763,19 @@ - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"><b><center>✕ (<a href="https://issues.apache.org/jira/browse/BEAM-91";>BEAM-91</a>)</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"></td> - <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"><b><center>✕</center></b> -</td> + <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"></td> </tr> @@ -926,23 +818,19 @@ - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: element-wise processing</center></b><br />Element-wise transformation parameterized by a chunk of user code. Elements are processed in bundles, with initialization and termination hooks. Bundle size is chosen by the runner and cannot be controlled by user code. ParDo processes a main input PCollection one element at a time, but provides side input access to additional PCollections. -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br />Batch mode uses large bundle sizes. Streaming uses smaller bundle sizes. -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br />ParDo itself, as per-element transformation with UDFs, is fully supported by Flink for both batch and streaming. -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br />ParDo applies per-element transformations as Spark FlatMapFunction. -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> </tr> @@ -951,23 +839,19 @@ - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: key grouping</center></b><br />Grouping of key-value pairs per key, window, and pane. (See also other tabs.) -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br />Uses Flink's keyBy for key grouping. When grouping by window in streaming (creating the panes) the Flink runner uses the Beam code. This guarantees support for all windowing and triggering mechanisms. -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><b><center>Partially: group by window in batch only</center></b><br />Uses Spark's groupByKey for grouping. Grouping by window is currently only supported in batch. -</td> + <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"></td> </tr> @@ -976,23 +860,19 @@ - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: collection concatenation</center></b><br />Concatenates multiple homogenously typed collections together. -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> </tr> @@ -1001,23 +881,19 @@ - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: associative & commutative aggregation</center></b><br />Application of an associative, commutative operation over all values ("globally") or over all values associated with each key ("per key"). Can be implemented using ParDo, but often more efficient implementations exist. -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: efficient execution</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br />Uses a combiner for pre-aggregation for batch and streaming. -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br />Supports GroupedValues, Globally and PerKey. -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> </tr> @@ -1026,23 +902,19 @@ - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: user-defined transformation subgraphs</center></b><br />Allows easy extensibility for library writers. In the near future, we expect there to be more information provided at this level -- customized metadata hooks for monitoring, additional runtime/environment hooks, etc. -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><b><center>Partially: supported via inlining</center></b><br />Currently composite transformations are inlined during execution. The structure is later recreated from the names, but other transform level information (if added to the model) will be lost. -</td> + <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><b><center>Partially: supported via inlining</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><b><center>Partially: supported via inlining</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"></td> </tr> @@ -1051,23 +923,19 @@ - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: additional elements available during DoFn execution</center></b><br />Side inputs are additional <tt>PCollections</tt> whose contents are computed during pipeline execution and then made accessible to DoFn code. The exact shape of the side input depends both on the <tt>PCollectionView</tt> used to describe the access pattern (interable, map, singleton) and the window of the element from the main input that is currently being processed. -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: some size restrictions in streaming</center></b><br />Batch implemented supports a distributed implementation, but streaming mode may force some size restrictions. Neither mode is able to push lookups directly up into key-based sources. -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><b><center>Partially: no supported in streaming<br />(<a href="https://issues.apache.org/jira/browse/BEAM-102";>BEAM-102</a>)</center></b><br />Supported in batch. Side inputs for streaming are currently WiP. -</td> + <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><b><center>Partially: not supported in streaming</center></b><br />Side input is actually a broadcast variable in Spark so it can't be updated during the life of a job. Spark-runner implementation of side input is more of an immutable, static, side input. -</td> + <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"></td> </tr> @@ -1076,23 +944,19 @@ - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: user-defined sources</center></b><br />Allows users to provide additional input sources. Supports both bounded and unbounded data. Includes hooks necessary to provide efficient parallelization (size estimation, progress information, dynamic splitting, etc). -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"><b><center>Yes: fully supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ec3;border-color:#ca1"></td> </tr> @@ -1101,23 +965,19 @@ - <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><b><center>Partially: user-provided metrics</center></b><br />Allow transforms to aggregate simple metrics across bundles in a <tt>DoFn</tt>. Semantically equivalent to using a side output, but support partial results as the transform executes. Will likely want to augment <tt>Aggregators</tt> to be more useful for processing unbounded data by making them windowed. -</td> + <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><b><center>Partially: may miscount in streaming mode</center></b><br />Current model is fully supported in batch mode. In streaming mode, <tt>Aggregators</tt> may under or overcount when bundles are retried. -</td> + <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><b><center>Partially: may undercount in streaming</center></b><br />Current model is fully supported in batch. In streaming mode, <tt>Aggregators</tt> may undercount. -</td> + <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><b><center>Partially: streaming requires more testing</center></b><br />Uses Spark's <tt>AccumulatorParam</tt> mechanism -</td> + <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"></td> </tr> @@ -1126,23 +986,19 @@ - <td width="25%" class="cap" style="background-color:#ddd;border-color:#ca1"><b><center>No: storage per key, per window<br />(<a href="https://issues.apache.org/jira/browse/BEAM-25";>BEAM-25</a>)</center></b><br />Allows fine-grained access to per-key, per-window persistent state. Necessary for certain use cases (e.g. high-volume windows which store large amounts of data, but typically only access small portions of it; complex state machines; etc.) that are not easily or efficiently addressed via <tt>Combine</tt> or <tt>GroupByKey</tt>+<tt>ParDo</tt>. -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#ca1"><b><center>No: pending model support</center></b><br />Dataflow already supports keyed state internally, so adding support for this should be easy once the Beam model exposes it. -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#ca1"><b><center>No: pending model support</center></b><br />Flink already supports keyed state, so adding support for this should be easy once the Beam model exposes it. -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#ca1"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#ca1"><b><center>No: pending model support</center></b><br />Spark supports keyed state with mapWithState() so support shuold be straight forward. -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#ca1"></td> </tr> @@ -1171,23 +1027,19 @@ - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: all time</center></b><br />The default window which covers all of time. (Basically how traditional batch cases fit in the model.) -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: default</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> </tr> @@ -1196,23 +1048,19 @@ - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: periodic, non-overlapping</center></b><br />Fixed-size, timestamp-based windows. (Hourly, Daily, etc) -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: built-in</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#8cf;border-color:#37d"><b><center>Partially: currently only supported in batch</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#8cf;border-color:#37d"></td> </tr> @@ -1221,23 +1069,19 @@ - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: periodic, overlapping</center></b><br />Possibly overlapping fixed-size timestamp-based windows (Every minute, use the last ten minutes of data.) -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: built-in</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#8cf;border-color:#37d"><b><center>Partially: currently only supported in batch</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#8cf;border-color:#37d"></td> </tr> @@ -1246,23 +1090,19 @@ - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: activity-based</center></b><br />Based on bursts of activity separated by a gap size. Different per key. -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: built-in</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: supported</center></b><br />The Runner uses Beam's Windowing and Triggering logic and code. -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#37d"><b><center>No: pending Spark engine support</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#37d"></td> </tr> @@ -1271,23 +1111,19 @@ - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: user-defined windows</center></b><br />All windows must implement <tt>BoundedWindow</tt>, which specifies a max timestamp. Each <tt>WindowFn</tt> assigns elements to an associated window. -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: supported</center></b><br />The Runner uses Beam's Windowing and Triggering logic and code. -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#37d"><b><center>No: pending Spark engine support</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#37d"></td> </tr> @@ -1296,23 +1132,19 @@ - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: user-defined merging windows</center></b><br />A custom <tt>WindowFn</tt> additionally specifies whether and how to merge windows. -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: supported</center></b><br />The Runner uses Beam's Windowing and Triggering logic and code. -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#37d"><b><center>No: pending Spark engine support</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#37d"></td> </tr> @@ -1321,23 +1153,19 @@ - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: output timestamp for window panes</center></b><br />For a grouping transform, such as GBK or Combine, an OutputTimeFn specifies (1) how to combine input timestamps within a window and (2) how to merge aggregated timestamps when windows merge. -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"><b><center>Yes: supported</center></b><br />The Runner uses Beam's Windowing and Triggering logic and code. -</td> + <td width="25%" class="cap" style="background-color:#59f;border-color:#37d"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#37d"><b><center>No: pending Spark engine support</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#37d"></td> </tr> @@ -1366,23 +1194,19 @@ - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: user customizable</center></b><br />Triggering may be specified by the user (instead of simply driven by hardcoded defaults). -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: fully supported</center></b><br />Fully supported in streaming mode. In batch mode, intermediate trigger firings are effectively meaningless. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: fully supported</center></b><br />The Runner uses Beam's Windowing and Triggering logic and code. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -1391,23 +1215,19 @@ - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: relative to event time</center></b><br />Triggers that fire in response to event-time completeness signals, such as watermarks progressing. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: yes in streaming, fixed granularity in batch</center></b><br />Fully supported in streaming mode. In batch mode, currently watermark progress jumps from the beginning of time to the end of time once the input has been fully consumed, thus no additional triggering granularity is available. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: fully supported</center></b><br />The Runner uses Beam's Windowing and Triggering logic and code. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -1416,23 +1236,19 @@ - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: relative to processing time</center></b><br />Triggers that fire in response to processing-time advancing. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: yes in streaming, fixed granularity in batch</center></b><br />Fully supported in streaming mode. In batch mode, from the perspective of triggers, processing time currently jumps from the beginning of time to the end of time once the input has been fully consumed, thus no additional triggering granularity is available. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: fully supported</center></b><br />The Runner uses Beam's Windowing and Triggering logic and code. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: This is Spark streaming's native model</center></b><br />Spark processes streams in micro-batches. The micro-batch size is actually a pre-set, fixed, time interval. Currently, the runner takes the first window size in the pipeline and sets it's size as the batch interval. Any following window operations will be considered processing time windows and will affect triggering. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> </tr> @@ -1441,23 +1257,19 @@ - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: every N elements</center></b><br />Triggers that fire after seeing at least N elements. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: fully supported</center></b><br />Fully supported in streaming mode. In batch mode, elements are processed in the largest bundles possible, so count-based triggers are effectively meaningless. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: fully supported</center></b><br />The Runner uses Beam's Windowing and Triggering logic and code. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -1466,23 +1278,19 @@ - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No: in response to data<br />(<a href="https://issues.apache.org/jira/browse/BEAM-101";>BEAM-101</a>)</center></b><br />Triggers that fire in response to attributes of the data being processed. -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No: pending model support</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No: pending model support</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No: pending model support</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -1491,23 +1299,19 @@ - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: compositions of one or more sub-triggers</center></b><br />Triggers which compose other triggers in more complex structures, such as logical AND, logical OR, early/on-time/late, etc. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: fully supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: fully supported</center></b><br />The Runner uses Beam's Windowing and Triggering logic and code. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -1516,23 +1320,19 @@ - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: event-time bound on window lifetimes</center></b><br />A way to bound the useful lifetime of a window (in event time), after which any unemitted results may be materialized, the window contents may be garbage collected, and any addtional late data that arrive for the window may be discarded. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: fully supported</center></b><br />Fully supported in streaming mode. In batch mode no data is ever late. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"><b><center>Yes: fully supported</center></b><br />The Runner uses Beam's Windowing and Triggering logic and code. -</td> + <td width="25%" class="cap" style="background-color:#8c6;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -1541,23 +1341,19 @@ - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No: delayed processing callbacks<br />(<a href="https://issues.apache.org/jira/browse/BEAM-27";>BEAM-27</a>)</center></b><br />A fine-grained mechanism for performing work at some point in the future, in either the event-time or processing-time domain. Useful for orchestrating delayed events, timeouts, etc in complex state per-key, per-window state machines. -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No: pending model support</center></b><br />Dataflow already supports timers internally, so adding support for this should be easy once the Beam model exposes it. -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No: pending model support</center></b><br />Flink already supports timers internally, so adding support for this should be easy once the Beam model exposes it. -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"><b><center>No: pending model support</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#6a4"></td> </tr> @@ -1586,23 +1382,19 @@ - <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"><b><center>Yes: panes discard elements when fired</center></b><br />Elements are discarded from accumulated state as their pane is fired. -</td> + <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"></td> - <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"><b><center>Yes: fully supported</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"></td> - <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"><b><center>Yes: fully supported</center></b><br />The Runner uses Beam's Windowing and Triggering logic and code. -</td> + <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"></td> - <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"><b><center>Yes: fully supported</center></b><br />Spark streaming natively discards elements after firing. -</td> + <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"></td> </tr> @@ -1611,23 +1403,19 @@ - <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"><b><center>Yes: panes accumulate elements across firings</center></b><br />Elements are accumulated in state across multiple pane firings for the same window. -</td> + <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"></td> - <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"><b><center>Yes: fully supported</center></b><br />Requires that the accumulated pane fits in memory, after being passed through the combiner (if relevant) -</td> + <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"></td> - <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"><b><center>Yes: fully supported</center></b><br />The Runner uses Beam's Windowing and Triggering logic and code. -</td> + <td width="25%" class="cap" style="background-color:#d77;border-color:#b55"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#b55"><b><center>No</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#b55"></td> </tr> @@ -1636,23 +1424,19 @@ - <td width="25%" class="cap" style="background-color:#ddd;border-color:#b55"><b><center>No: accumulation plus retraction of old panes<br />(<a href="https://issues.apache.org/jira/browse/BEAM-91";>BEAM-91</a>)</center></b><br />Elements are accumulated across multiple pane firings and old emitted values are retracted. Also known as "backsies" ;-D -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#b55"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#b55"><b><center>No: pending model support</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#b55"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#b55"><b><center>No: pending model support</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#b55"></td> - <td width="25%" class="cap" style="background-color:#ddd;border-color:#b55"><b><center>No: pending model support</center></b><br /> -</td> + <td width="25%" class="cap" style="background-color:#ddd;border-color:#b55"></td> </tr>
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/6fce44a2/content/coming-soon.html ---------------------------------------------------------------------- diff --git a/content/coming-soon.html b/content/coming-soon.html index 876fab4..8174ac8 100644 --- a/content/coming-soon.html +++ b/content/coming-soon.html @@ -50,10 +50,14 @@ <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a> <ul class="dropdown-menu"> + <li class="dropdown-header">Releases</li> + <li><a href="/releases/">Release information</a></li> + <li role="separator" class="divider"></li> <li class="dropdown-header">Guides</li> <li><a href="/getting_started/">Getting Started</a></li> <li role="separator" class="divider"></li> <li class="dropdown-header">Technical Documentation</li> + <li><a href="/javadoc/0.1.0-incubating/">Java API Reference</a></li> <li><a href="/capability-matrix/">Capability Matrix</a></li> <li><a href="https://goo.gl/ps8twC";>Technical Docs</a></li> <li><a href="https://goo.gl/nk5OM0";>Technical Vision</a></li> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/6fce44a2/content/contribution-guide/index.html ---------------------------------------------------------------------- diff --git a/content/contribution-guide/index.html b/content/contribution-guide/index.html index 1d71e8b..c4e22a7 100644 --- a/content/contribution-guide/index.html +++ b/content/contribution-guide/index.html @@ -50,10 +50,14 @@ <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a> <ul class="dropdown-menu"> + <li class="dropdown-header">Releases</li> + <li><a href="/releases/">Release information</a></li> + <li role="separator" class="divider"></li> <li class="dropdown-header">Guides</li> <li><a href="/getting_started/">Getting Started</a></li> <li role="separator" class="divider"></li> <li class="dropdown-header">Technical Documentation</li> + <li><a href="/javadoc/0.1.0-incubating/">Java API Reference</a></li> <li><a href="/capability-matrix/">Capability Matrix</a></li> <li><a href="https://goo.gl/ps8twC";>Technical Docs</a></li> <li><a href="https://goo.gl/nk5OM0";>Technical Vision</a></li> @@ -170,9 +174,9 @@ <h2 id="engage">Engage</h2> <h3 id="mailing-lists">Mailing list(s)</h3> -<p>We discuss design and implementation issues on [email protected] mailing list, which is archived <a href="http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/";>here</a>. Join by emailing <a href="mailto:dev-subscribe@beam.incubator.apache.org"><code class="highlighter-rouge">[email protected]</code></a>.</p> +<p>We discuss design and implementation issues on [email protected] mailing list, which is archived <a href="http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/";>here</a>. Join by emailing <a href="mailto:dev-subscribe@beam.incubator.apache.org"><code>[email protected]</code></a>.</p> -<p>If interested, you can also join <a href="http://mail-archives.apache.org/mod_mbox/incubator-beam-user/";><code class="highlighter-rouge">[email protected]</code></a> and <a href="http://mail-archives.apache.org/mod_mbox/incubator-beam-commits/";><code class="highlighter-rouge">[email protected]</code></a> too.</p> +<p>If interested, you can also join <a href="http://mail-archives.apache.org/mod_mbox/incubator-beam-user/";><code>[email protected]</code></a> and <a href="http://mail-archives.apache.org/mod_mbox/incubator-beam-commits/";><code>[email protected]</code></a> too.</p> <h3 id="apache-jira">Apache JIRA</h3> <p>We use <a href="https://issues.apache.org/jira/browse/BEAM";>Apache JIRA</a> as an issue tracking and project management tool, as well as a way to communicate among a very diverse and distributed set of contributors. To be able to gather feedback, avoid frustration, and avoid duplicated efforts all Beam-related work should be tracked there.</p> @@ -219,10 +223,9 @@ <p>Clone Beamâs read-only GitHub mirror.</p> -<div class="highlighter-rouge"><pre class="highlight"><code>$ git clone https://github.com/apache/incubator-beam.git +<pre><code>$ git clone https://github.com/apache/incubator-beam.git $ cd incubator-beam </code></pre> -</div> <p>Add your forked repository as an additional Git remote, where youâll push your changes.</p> @@ -231,7 +234,7 @@ $ cd incubator-beam <p>You are now ready to start developing!</p> <h3 id="create-a-branch-in-your-fork">Create a branch in your fork</h3> -<p>Youâll work on your contribution in a branch in your own (forked) repository. Create a local branch, initialized with the state of the branch you expect your changes to be merged into. Keep in mind that we use several branches, including <code class="highlighter-rouge">master</code>, feature-specific, and release-specific branches. If you are unsure, initialize with the state of the <code class="highlighter-rouge">master</code> branch.</p> +<p>Youâll work on your contribution in a branch in your own (forked) repository. Create a local branch, initialized with the state of the branch you expect your changes to be merged into. Keep in mind that we use several branches, including <code>master</code>, feature-specific, and release-specific branches. If you are unsure, initialize with the state of the <code>master</code> branch.</p> <pre><code>$ git fetch --all $ git checkout -b <b><my-branch></b> origin/master</code></pre> @@ -241,11 +244,10 @@ $ git checkout -b <b><my-branch></b> origin/master</code></pre> <h3 id="syncing-and-pushing-your-branch">Syncing and pushing your branch</h3> <p>Periodically while you work, and certainly before submitting a pull request, you should update your branch with the most recent changes to the target branch.</p> -<div class="highlighter-rouge"><pre class="highlight"><code>$ git pull --rebase +<pre><code>$ git pull --rebase </code></pre> -</div> -<p>Remember to always use <code class="highlighter-rouge">--rebase</code> parameter to avoid extraneous merge commits.</p> +<p>Remember to always use <code>--rebase</code> parameter to avoid extraneous merge commits.</p> <p>To push your local, committed changes to your (forked) repository on GitHub, run:</p> @@ -256,9 +258,8 @@ $ git checkout -b <b><my-branch></b> origin/master</code></pre> <p>For contributions to the Java code, run unit tests locally via Maven. Alternatively, you can use Travis-CI.</p> -<div class="highlighter-rouge"><pre class="highlight"><code>$ mvn clean verify +<pre><code>$ mvn clean verify </code></pre> -</div> <h2 id="review">Review</h2> <p>Once the initial code is complete and the tests pass, itâs time to start the code review process. We review and discuss all code, no matter who authors it. Itâs a great way to build community, since you can learn from other developers, and they become familiar with your contribution. It also builds a strong project by encouraging a high quality bar and keeping code consistent throughout the project.</p> @@ -266,9 +267,8 @@ $ git checkout -b <b><my-branch></b> origin/master</code></pre> <h3 id="create-a-pull-request">Create a pull request</h3> <p>Organize your commits to make your reviewerâs job easier. Use the following command to re-order, squash, edit, or change description of individual commits.</p> -<div class="highlighter-rouge"><pre class="highlight"><code>$ git rebase -i origin/master +<pre><code>$ git rebase -i origin/master </code></pre> -</div> <p>Navigate to the <a href="https://github.com/apache/incubator-beam";>Beam GitHub mirror</a> to create a pull request. The title of the pull request should be strictly in the following format:</p> @@ -315,22 +315,19 @@ $ git push <b><GitHub_user></b> --delete <b><my-branch></b></code></ <h3 id="one-time-setup-1">One-time Setup</h3> <p>Add the Apache Git remote in your local clone, by running:</p> -<div class="highlighter-rouge"><pre class="highlight"><code>$ git remote add apache https://git-wip-us.apache.org/repos/asf/incubator-beam.git +<pre><code>$ git remote add apache https://git-wip-us.apache.org/repos/asf/incubator-beam.git </code></pre> -</div> -<p>We recommend renaming the <code class="highlighter-rouge">origin</code> remote to <code class="highlighter-rouge">github</code>, to avoid confusion when dealing with this many remotes.</p> +<p>We recommend renaming the <code>origin</code> remote to <code>github</code>, to avoid confusion when dealing with this many remotes.</p> -<div class="highlighter-rouge"><pre class="highlight"><code>$ git remote rename origin github +<pre><code>$ git remote rename origin github </code></pre> -</div> -<p>For the <code class="highlighter-rouge">github</code> remote, add an additional fetch reference, which will cause every pull request to be made available as a remote branch in your workspace.</p> +<p>For the <code>github</code> remote, add an additional fetch reference, which will cause every pull request to be made available as a remote branch in your workspace.</p> -<div class="highlighter-rouge"><pre class="highlight"><code>$ git config --local --add remote.github.fetch \ +<pre><code>$ git config --local --add remote.github.fetch \ '+refs/pull/*/head:refs/remotes/github/pr/*' </code></pre> -</div> <p>You can confirm your configuration by running the following command.</p> @@ -367,11 +364,10 @@ $ git checkout -b finish-pr-<b><pull-request-#></b> github/pr/<b><pull- <li>Reorganize commits that are part of the pull request, such as squash them into fewer commits that make sense for a historical perspective.</li> </ul> -<p>You will often need the following command, assuming youâll be merging changes into the <code class="highlighter-rouge">master</code> branch:</p> +<p>You will often need the following command, assuming youâll be merging changes into the <code>master</code> branch:</p> -<div class="highlighter-rouge"><pre class="highlight"><code>$ git rebase -i apache/master +<pre><code>$ git rebase -i apache/master </code></pre> -</div> <p>Please make sure to retain authorship of original commits to give proper credit to the contributor. You are welcome to change their commits slightly (e.g., fix a typo) and squash them, but more substantive changes should be a separate commit and review.</p> @@ -384,13 +380,12 @@ $ git merge --no-ff \ -m $'[BEAM-<b><JIRA-issue-#></b>] <b><Title></b>\n\nThis closes #<b><pull-request-#></b>' \ finish-pr-<b><pull-request-#></b></code></pre> -<p>Always use <code class="highlighter-rouge">--no-ff</code> option and the specific commit message âThis closes #<b><pull request #></b>ââ â it ensures proper marking in the tooling. It would be nice to include additional information in the merge commit message, such as the title and summary of the pull request.</p> +<p>Always use <code>--no-ff</code> option and the specific commit message âThis closes #<b><pull request #></b>ââ â it ensures proper marking in the tooling. It would be nice to include additional information in the merge commit message, such as the title and summary of the pull request.</p> -<p>At this point, you want to ensure everything is right. Test it with <code class="highlighter-rouge">mvn verify</code>. Run <code class="highlighter-rouge">gitk</code> or <code class="highlighter-rouge">git log --graph,</code> etc. When you are happy with how it looks, push it. This is the point of no return â proceed with caution.</p> +<p>At this point, you want to ensure everything is right. Test it with <code>mvn verify</code>. Run <code>gitk</code> or <code>git log --graph,</code> etc. When you are happy with how it looks, push it. This is the point of no return â proceed with caution.</p> -<div class="highlighter-rouge"><pre class="highlight"><code>$ git push apache HEAD:master +<pre><code>$ git push apache HEAD:master </code></pre> -</div> <p>Done. You can delete the local <code>finish-pr-<b><pull-request-#></b></code> branch if you like.</p> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/6fce44a2/content/docs/index.html ---------------------------------------------------------------------- diff --git a/content/docs/index.html b/content/docs/index.html index 623d372..ba64568 100644 --- a/content/docs/index.html +++ b/content/docs/index.html @@ -50,10 +50,14 @@ <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a> <ul class="dropdown-menu"> + <li class="dropdown-header">Releases</li> + <li><a href="/releases/">Release information</a></li> + <li role="separator" class="divider"></li> <li class="dropdown-header">Guides</li> <li><a href="/getting_started/">Getting Started</a></li> <li role="separator" class="divider"></li> <li class="dropdown-header">Technical Documentation</li> + <li><a href="/javadoc/0.1.0-incubating/">Java API Reference</a></li> <li><a href="/capability-matrix/">Capability Matrix</a></li> <li><a href="https://goo.gl/ps8twC";>Technical Docs</a></li> <li><a href="https://goo.gl/nk5OM0";>Technical Vision</a></li>
