Regenerate website
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/aa01f116 Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/aa01f116 Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/aa01f116 Branch: refs/heads/asf-site Commit: aa01f1161bbbfc0060ab76ebad6bfcee7826e963 Parents: 8ed3f24 Author: Ahmet Altay <al...@google.com> Authored: Tue May 9 21:24:35 2017 -0700 Committer: Ahmet Altay <al...@google.com> Committed: Tue May 9 21:24:35 2017 -0700 ---------------------------------------------------------------------- content/documentation/runners/flink/index.html | 2 +- .../sdks/java-extensions/index.html | 237 +++++++++++++++++++ content/documentation/sdks/java/index.html | 9 + 3 files changed, 247 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/beam-site/blob/aa01f116/content/documentation/runners/flink/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/runners/flink/index.html b/content/documentation/runners/flink/index.html index 64ae583..deb04c8 100644 --- a/content/documentation/runners/flink/index.html +++ b/content/documentation/runners/flink/index.html @@ -287,7 +287,7 @@ </tr> </table> -<p>See the reference documentation for the <span class="language-java"><a href="/documentation/sdks/javadoc/0.6.0/index.html?org/apache/beam/runners/flink/FlinkPipelineOptions.html">FlinkPipelineOptions</a></span><span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/pipeline_options.py";>PipelineOptions</a></span> interface (and its subinterfaces) for the complete list of pipeline configuration options.</p> +<p>See the reference documentation for the <span class="language-java"><a href="/documentation/sdks/javadoc/0.6.0/index.html?org/apache/beam/runners/flink/FlinkPipelineOptions.html">FlinkPipelineOptions</a></span><span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py";>PipelineOptions</a></span> interface (and its subinterfaces) for the complete list of pipeline configuration options.</p> <h2 id="additional-information-and-caveats">Additional information and caveats</h2> http://git-wip-us.apache.org/repos/asf/beam-site/blob/aa01f116/content/documentation/sdks/java-extensions/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/sdks/java-extensions/index.html b/content/documentation/sdks/java-extensions/index.html new file mode 100644 index 0000000..ee93238 --- /dev/null +++ b/content/documentation/sdks/java-extensions/index.html @@ -0,0 +1,237 @@ +<!DOCTYPE html> +<html lang="en"> + + <head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + + <title>Beam Java SDK Extensions</title> + <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes. +"> + + <link rel="stylesheet" href="/styles/site.css"> + <link rel="stylesheet" href="/css/theme.css"> + <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";></script> + <script src="/js/bootstrap.min.js"></script> + <script src="/js/language-switch.js"></script> + <link rel="canonical" href="https://beam.apache.org/documentation/sdks/java-extensions/"; data-proofer-ignore> + <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml";> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-73650088-1', 'auto'); + ga('send', 'pageview'); + + </script> + <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico"> +</head> + + + <body role="document"> + + <nav class="navbar navbar-default navbar-fixed-top"> + <div class="container"> + <div class="navbar-header"> + <a href="/" class="navbar-brand" > + <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png"> + </a> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + </div> + <div id="navbar" class="navbar-collapse collapse"> + <ul class="nav navbar-nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Get Started <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/get-started/beam-overview/">Beam Overview</a></li> + <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li> + <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Example Walkthroughs</li> + <li><a href="/get-started/wordcount-example/">WordCount</a></li> + <li><a href="/get-started/mobile-gaming-example/">Mobile Gaming</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Resources</li> + <li><a href="/get-started/downloads">Downloads</a></li> + <li><a href="/get-started/support">Support</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/documentation">Using the Documentation</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Beam Concepts</li> + <li><a href="/documentation/programming-guide/">Programming Guide</a></li> + <li><a href="/documentation/resources/">Additional Resources</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Pipeline Fundamentals</li> + <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li> + <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li> + <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li> + <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">SDKs</li> + <li><a href="/documentation/sdks/java/">Java SDK</a></li> + <li><a href="/documentation/sdks/javadoc/0.6.0/" target="_blank">Java SDK API Reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + <li><a href="/documentation/sdks/python/">Python SDK</a></li> + <li><a href="/documentation/sdks/pydoc/0.6.0/" target="_blank">Python SDK API Reference <img src="/images/external-link-icon.png" + width="14" height="14" + alt="External link."></a> + </li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Runners</li> + <li><a href="/documentation/runners/capability-matrix/">Capability Matrix</a></li> + <li><a href="/documentation/runners/direct/">Direct Runner</a></li> + <li><a href="/documentation/runners/apex/">Apache Apex Runner</a></li> + <li><a href="/documentation/runners/flink/">Apache Flink Runner</a></li> + <li><a href="/documentation/runners/spark/">Apache Spark Runner</a></li> + <li><a href="/documentation/runners/dataflow/">Cloud Dataflow Runner</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Contribute <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/contribute">Get Started Contributing</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Guides</li> + <li><a href="/contribute/contribution-guide/">Contribution Guide</a></li> + <li><a href="/contribute/testing/">Testing Guide</a></li> + <li><a href="/contribute/release-guide/">Release Guide</a></li> + <li><a href="/contribute/ptransform-style-guide/">PTransform Style Guide</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Technical References</li> + <li><a href="/contribute/design-principles/">Design Principles</a></li> + <li><a href="/contribute/work-in-progress/">Ongoing Projects</a></li> + <li><a href="/contribute/source-repository/">Source Repository</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Promotion</li> + <li><a href="/contribute/presentation-materials/">Presentation Materials</a></li> + <li><a href="/contribute/logos/">Logos and Design</a></li> + <li role="separator" class="divider"></li> + <li><a href="/contribute/maturity-model/">Maturity Model</a></li> + <li><a href="/contribute/team/">Team</a></li> + </ul> + </li> + + <li><a href="/blog">Blog</a></li> + </ul> + <ul class="nav navbar-nav navbar-right"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png"; alt="Apache Logo" style="height:24px;">Apache Software Foundation<span class="caret"></span></a> + <ul class="dropdown-menu dropdown-menu-right"> + <li><a href="http://www.apache.org/";>ASF Homepage</a></li> + <li><a href="http://www.apache.org/licenses/";>License</a></li> + <li><a href="http://www.apache.org/security/";>Security</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="https://www.apache.org/foundation/policies/conduct";>Code of Conduct</a></li> + </ul> + </li> + </ul> + </div><!--/.nav-collapse --> + </div> +</nav> + + +<link rel="stylesheet" href=""> + + + <div class="container" role="main"> + + <div class="row"> + <h1 id="apache-beam-java-sdk-extensions">Apache Beam Java SDK Extensions</h1> + +<h2 id="a-namejoin-libraryajoin-library"><a name="join-library"></a>Join-library</h2> + +<p>Join-library provides inner join, outer left join, and outer right join functions. The aim +is to simplify the most common cases of join to a simple function call.</p> + +<p>The functions are generic and support joins of any Beam-supported types. +Input to the join functions are <code class="highlighter-rouge">PCollections</code> of <code class="highlighter-rouge">Key</code> / <code class="highlighter-rouge">Value</code>s. Both +the left and right <code class="highlighter-rouge">PCollection</code>s need the same type for the key. All the join +functions return a <code class="highlighter-rouge">Key</code> / <code class="highlighter-rouge">Value</code> where <code class="highlighter-rouge">Key</code> is the join key and value is +a <code class="highlighter-rouge">Key</code> / <code class="highlighter-rouge">Value</code> where the key is the left value and right is the value.</p> + +<p>For outer joins, the user must provide a value that represents <code class="highlighter-rouge">null</code> because <code class="highlighter-rouge">null</code> +cannot be serialized.</p> + +<p>Example usage:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>PCollection<KV<String, String>> leftPcollection = ... +PCollection<KV<String, Long>> rightPcollection = ... + +PCollection<KV<String, KV<String, Long>>> joinedPcollection = + Join.innerJoin(leftPcollection, rightPcollection); +</code></pre> +</div> + +<h2 id="a-namesorterasorter"><a name="sorter"></a>Sorter</h2> + +<p>This module provides the <code class="highlighter-rouge">SortValues</code> transform, which takes a <code class="highlighter-rouge">PCollection<KV<K, Iterable<KV<K2, V>>>></code> and produces a <code class="highlighter-rouge">PCollection<KV<K, Iterable<KV<K2, V>>>></code> where, for each primary key <code class="highlighter-rouge">K</code> the paired <code class="highlighter-rouge">Iterable<KV<K2, V>></code> has been sorted by the byte encoding of secondary key (<code class="highlighter-rouge">K2</code>). It is an efficient and scalable sorter for iterables, even if they are large (do not fit in memory).</p> + +<h3 id="caveats">Caveats</h3> + +<ul> + <li>This transform performs value-only sorting; the iterable accompanying each key is sorted, but <em>there is no relationship between different keys</em>, as Beam does not support any defined relationship between different elements in a <code class="highlighter-rouge">PCollection</code>.</li> + <li>Each <code class="highlighter-rouge">Iterable<KV<K2, V>></code> is sorted on a single worker using local memory and disk. This means that <code class="highlighter-rouge">SortValues</code> may be a performance and/or scalability bottleneck when used in different pipelines. For example, users are discouraged from using <code class="highlighter-rouge">SortValues</code> on a <code class="highlighter-rouge">PCollection</code> of a single element to globally sort a large <code class="highlighter-rouge">PCollection</code>. A (rough) estimate of the number of bytes of disk space utilized if sorting spills to disk is <code class="highlighter-rouge">numRecords * (numSecondaryKeyBytesPerRecord + numValueBytesPerRecord + 16) * 3</code>.</li> +</ul> + +<h3 id="options">Options</h3> + +<ul> + <li>The user can customize the temporary location used if sorting requires spilling to disk and the maximum amount of memory to use by creating a custom instance of <code class="highlighter-rouge">BufferedExternalSorter.Options</code> to pass into <code class="highlighter-rouge">SortValues.create</code>.</li> +</ul> + +<h3 id="example-usage-of-sortvalues">Example usage of <code class="highlighter-rouge">SortValues</code></h3> + +<div class="highlighter-rouge"><pre class="highlight"><code>PCollection<KV<String, KV<String, Integer>>> input = ... + +// Group by primary key, bringing <SecondaryKey, Value> pairs for the same key together. +PCollection<KV<String, Iterable<KV<String, Integer>>>> grouped = + input.apply(GroupByKey.<String, KV<String, Integer>>create()); + +// For every primary key, sort the iterable of <SecondaryKey, Value> pairs by secondary key. +PCollection<KV<String, Iterable<KV<String, Integer>>>> groupedAndSorted = + grouped.apply( + SortValues.<String, String, Integer>create(new BufferedExternalSorter.Options())); +</code></pre> +</div> + + </div> + + + <hr> + <div class="row"> + <div class="col-xs-12"> + <footer> + <p class="text-center"> + © Copyright + <a href="http://www.apache.org";>The Apache Software Foundation</a>, + 2017. All Rights Reserved. + </p> + <p class="text-center"> + <a href="/privacy_policy">Privacy Policy</a> | + <a href="/feed.xml">RSS Feed</a> + </p> + </footer> + </div> + </div> + <!-- container div end --> +</div> + + + </body> + +</html> http://git-wip-us.apache.org/repos/asf/beam-site/blob/aa01f116/content/documentation/sdks/java/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/sdks/java/index.html b/content/documentation/sdks/java/index.html index d44c20f..08d5819 100644 --- a/content/documentation/sdks/java/index.html +++ b/content/documentation/sdks/java/index.html @@ -168,6 +168,15 @@ <h2 id="pipeline-io">Pipeline I/O</h2> <p>See the <a href="/documentation/io/built-in/">Beam-provided I/O Transforms</a> page for a list of the currently available I/O transforms.</p> +<h2 id="extensions">Extensions</h2> + +<p>The Java SDK has the following extensions:</p> + +<ul> + <li><a href="/documentation/sdks/java-extensions/#join-library">join-library</a> provides inner join, outer left join, and outer right join functions.</li> + <li><a href="/documentation/sdks/java-extensions/#sorter">sorter</a> is an efficient and scalable sorter for large iterables.</li> +</ul> + </div>
![https://www.apache.org/foundation/press/kit/feather_small.png"](https://www.apache.org/foundation/press/kit/feather_small.png"){kind=link}