Repository: incubator-beam-site Updated Branches: refs/heads/asf-site bb5cccf56 -> 0ae2bcdc9
Regenerate html Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/5d3e72dc Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/5d3e72dc Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/5d3e72dc Branch: refs/heads/asf-site Commit: 5d3e72dcba19de8c81eb1ce4a26466b8b1045499 Parents: 73e5d3f Author: Frances Perry <[email protected]> Authored: Sat Nov 5 22:16:24 2016 -0700 Committer: Frances Perry <[email protected]> Committed: Sat Nov 5 22:20:28 2016 -0700 ---------------------------------------------------------------------- content/documentation/index.html | 39 +++- .../runners/capability-matrix/index.html | 2 +- content/documentation/runners/index.html | 172 --------------- content/documentation/sdks/index.html | 166 -------------- content/feed.xml | 4 +- content/get-started/quickstart/index.html | 219 ++++++++++++++++--- content/learn/runners/index.html | 10 - content/learn/sdks/index.html | 10 - 8 files changed, 225 insertions(+), 397 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5d3e72dc/content/documentation/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/index.html b/content/documentation/index.html index 8266a10..45ae28c 100644 --- a/content/documentation/index.html +++ b/content/documentation/index.html @@ -139,31 +139,48 @@ <div class="container" role="main"> <div class="row"> - <h1 id="learn-about-the-apache-beam-model">Learn about the Apache Beam Model</h1> + <h1 id="apache-beam-documentation">Apache Beam Documentation</h1> <p>Get in-depth conceptual information and reference material for the Beam Model, SDKs and Runners:</p> -<h4 id="beam-programming-guidelearnprogramming-guide"><a href="/learn/programming-guide/">Beam Programming Guide</a></h4> +<h2 id="concepts">Concepts</h2> + <p>Learn about the Beam Programming Model and the concepts common to all Beam SDKs and Runners.</p> -<h4 id="beam-sdks">Beam SDKs</h4> +<ul> + <li>The <a href="/documentation/programming-guide/">Programming Guide</a> introduces all the key Beam concepts.</li> + <li>Visit <a href="/documentation/resources/">Additional Resources</a> for some of our favorite articles and talks about Beam.</li> +</ul> + +<h2 id="sdks">SDKs</h2> + <p>Find status and reference information on all of the available Beam SDKs.</p> <ul> - <li><a href="/learn/sdks/java/">Java SDK</a></li> + <li><a href="/documentation/sdks/java/">Java SDK</a></li> + <li><em>[Under Development]</em> <a href="/contribute/work-in-progress/#feature-branches">Python SDK</a></li> </ul> -<h4 id="runners">Runners</h4> -<p>Learn about the <a href="/learn/runners/capability-matrix/">Capability Matrix</a> and find status and reference information on all of the available Beam Runners:</p> +<h2 id="runners">Runners</h2> + +<p>A Beam Runner runs a Beam pipeline on a specific (often distributed) data processing system.</p> + +<h3 id="available-runners">Available Runners</h3> <ul> - <li><a href="/learn/runners/direct/">Direct Runner</a></li> - <li><a href="/learn/runners/flink/">Apache Flink</a></li> - <li><a href="/learn/runners/spark/">Apache Spark</a></li> - <li><a href="/learn/runners/dataflow/">Cloud Dataflow</a></li> + <li><a href="/documentation/runners/direct/">DirectRunner</a>: Runs locally on your machine â great for developing, testing, and debugging.</li> + <li><a href="/documentation/runners/flink/">FlinkRunner</a>: Runs on <a href="http://flink.apache.org";>Apache Flink</a>.</li> + <li><a href="/documentation/runners/spark/">SparkRunner</a>: Runs on <a href="http://spark.apache.org";>Apache Spark</a>.</li> + <li><a href="/documentation/runners/dataflow/">DataflowRunner</a>: Runs on <a href="https://cloud.google.com/dataflow";>Google Cloud Dataflow</a>, a fully managed service within <a href="https://cloud.google.com/";>Google Cloud Platform</a>.</li> + <li><em>[Under Development]</em> <a href="/contribute/work-in-progress/#feature-branches">ApexRunner</a>: Runs on <a href="http://apex.apache.org";>Apache Apex</a>.</li> + <li><em>[Under Development]</em> <a href="/contribute/work-in-progress/#feature-branches">GearpumpRunner</a>: Runs on <a href="http://gearpump.apache.org";>Apache Gearpump (incubating)</a>.</li> </ul> -<h4 id="additional-resourceslearnresources"><a href="/learn/resources/">Additional Resources</a></h4> +<h3 id="choosing-a-runner">Choosing a Runner</h3> + +<p>Beam is designed to enable pipelines to be portable across different runners. However, given every runner has different capabilities, they also have different abilities to implement the core concepts in the Beam model. The <a href="/documentation/runners/capability-matrix">Capability Matrix</a> provides a detailed comparison of runner functionality.</p> + +<p>Once you have chosen which runner to use, see that runnerâs page for more information about any initial runner-specific setup as well as any required or optional <code class="highlighter-rouge">PipelineOptions</code> for configuring itâs execution. You may also want to refer back to the <a href="/get-started/quickstart">Quickstart</a> for instructions on executing the sample WordCount pipeline.</p> </div> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5d3e72dc/content/documentation/runners/capability-matrix/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/runners/capability-matrix/index.html b/content/documentation/runners/capability-matrix/index.html index 8db9915..ca1aa72 100644 --- a/content/documentation/runners/capability-matrix/index.html +++ b/content/documentation/runners/capability-matrix/index.html @@ -140,7 +140,7 @@ <div class="row"> <h1 id="beam-capability-matrix">Beam Capability Matrix</h1> -<p><span style="font-size:11px;float:none">Last updated: 2016-11-04 17:50 PDT</span></p> +<p><span style="font-size:11px;float:none">Last updated: 2016-11-05 22:12 PDT</span></p> <p>Apache Beam (incubating) provides a portable API layer for building sophisticated data-parallel processing engines that may be executed across a diversity of exeuction engines, or <i>runners</i>. The core concepts of this layer are based upon the Beam Model (formerly referred to as the <a href="http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf";>Dataflow Model</a>), and implemented to varying degrees in each Beam runner. To help clarify the capabilities of individual runners, weâve created the capability matrix below.</p> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5d3e72dc/content/documentation/runners/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/runners/index.html b/content/documentation/runners/index.html deleted file mode 100644 index 8f24be7..0000000 --- a/content/documentation/runners/index.html +++ /dev/null @@ -1,172 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> - - <head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - - <title>Beam Runners</title> - <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes. -"> - - <link rel="stylesheet" href="/styles/site.css"> - <link rel="stylesheet" href="/css/theme.css"> - <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";></script> - <script src="/js/bootstrap.min.js"></script> - <script src="/js/language-switch.js"></script> - <link rel="canonical" href="http://beam.incubator.apache.org/documentation/runners/"; data-proofer-ignore> - <link rel="alternate" type="application/rss+xml" title="Apache Beam (incubating)" href="http://beam.incubator.apache.org/feed.xml";> - <script> - (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ - (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), - m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) - })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); - - ga('create', 'UA-73650088-1', 'auto'); - ga('send', 'pageview'); - - </script> - <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico"> -</head> - - - <body role="document"> - - <nav class="navbar navbar-default navbar-fixed-top"> - <div class="container"> - <div class="navbar-header"> - <a href="/" class="navbar-brand" > - <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png"> - </a> - <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> - <span class="sr-only">Toggle navigation</span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - </div> - <div id="navbar" class="navbar-collapse collapse"> - <ul class="nav navbar-nav"> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Get Started <span class="caret"></span></a> - <ul class="dropdown-menu"> - <li><a href="/get-started/beam-overview/">Beam Overview</a></li> - <li><a href="/get-started/quickstart/">Quickstart</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Example Walkthroughs</li> - <li><a href="/get-started/wordcount-example/">WordCount</a></li> - <li><a href="/get-started/mobile-gaming-example/">Mobile Gaming</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Resources</li> - <li><a href="/get-started/downloads">Downloads</a></li> - <li><a href="/get-started/releases">Release Notes</a></li> - <li><a href="/get-started/support">Support</a></li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a> - <ul class="dropdown-menu"> - <li><a href="/documentation">Using the Documentation</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Beam Concepts</li> - <li><a href="/documentation/programming-guide/">Programming Guide</a></li> - <li><a href="/documentation/resources/">Additional Resources</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Pipeline Fundamentals</li> - <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li> - <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li> - <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">SDKs</li> - <li><a href="/documentation/sdks/java/">Java SDK</a></li> - <li><a href="/documentation/sdks/javadoc/">Java SDK API Reference</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Runners</li> - <li><a href="/documentation/runners/capability-matrix/">Capability Matrix</a></li> - <li><a href="/documentation/runners/direct/">Direct Runner</a></li> - <li><a href="/documentation/runners/flink/">Apache Flink Runner</a></li> - <li><a href="/documentation/runners/spark/">Apache Spark Runner</a></li> - <li><a href="/documentation/runners/dataflow/">Cloud Dataflow Runner</a></li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Contribute <span class="caret"></span></a> - <ul class="dropdown-menu"> - <li><a href="/contribute">Get Started Contributing</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Guides</li> - <li><a href="/contribute/contribution-guide/">Contribution Guide</a></li> - <li><a href="/contribute/testing/">Testing Guide</a></li> - <li><a href="/contribute/release-guide/">Release Guide</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Technical References</li> - <li><a href="/contribute/design-principles/">Design Principles</a></li> - <li><a href="/contribute/work-in-progress/">Ongoing Projects</a></li> - <li><a href="/contribute/source-repository/">Source Repository</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Promotion</li> - <li><a href="/contribute/presentation-materials/">Presentation Materials</a></li> - <li><a href="/contribute/logos/">Logos and Design</a></li> - <li role="separator" class="divider"></li> - <li><a href="/contribute/team/">Team</a></li> - </ul> - </li> - - <li><a href="/blog">Blog</a></li> - </ul> - <ul class="nav navbar-nav navbar-right"> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png"; alt="Apache Logo" style="height:24px;">Apache Software Foundation<span class="caret"></span></a> - <ul class="dropdown-menu dropdown-menu-right"> - <li><a href="http://www.apache.org/";>ASF Homepage</a></li> - <li><a href="http://www.apache.org/licenses/";>License</a></li> - <li><a href="http://www.apache.org/security/";>Security</a></li> - <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> - <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> - <li><a href="https://www.apache.org/foundation/policies/conduct";>Code of Conduct</a></li> - </ul> - </li> - </ul> - </div><!--/.nav-collapse --> - </div> -</nav> - - -<link rel="stylesheet" href=""> - - - <div class="container" role="main"> - - <div class="row"> - <h1 id="apache-beam-runners">Apache Beam Runners</h1> - -<h4 id="direct-runnerlearnrunnersdirect"><a href="/learn/runners/direct/">Direct Runner</a></h4> - -<h4 id="apache-flink-runnerlearnrunnersflink"><a href="/learn/runners/flink/">Apache Flink Runner</a></h4> - -<h4 id="apache-spark-runnerlearnrunnersspark"><a href="/learn/runners/spark/">Apache Spark Runner</a></h4> - -<h4 id="cloud-dataflow-runnerlearnrunnersdataflow"><a href="/learn/runners/dataflow/">Cloud Dataflow Runner</a></h4> - - </div> - - - <hr> - <div class="row"> - <div class="col-xs-12"> - <footer> - <p class="text-center">© Copyright 2016 - <a href="http://www.apache.org";>The Apache Software Foundation.</a> All Rights Reserved.</p> - <p class="text-center"><a href="/privacy_policy">Privacy Policy</a> | - <a href="/feed.xml">RSS Feed</a></p> - </footer> - </div> - </div> - <!-- container div end --> -</div> - - - </body> - -</html> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5d3e72dc/content/documentation/sdks/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/sdks/index.html b/content/documentation/sdks/index.html deleted file mode 100644 index ce7ecf7..0000000 --- a/content/documentation/sdks/index.html +++ /dev/null @@ -1,166 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> - - <head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - - <title>Beam SDKs</title> - <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes. -"> - - <link rel="stylesheet" href="/styles/site.css"> - <link rel="stylesheet" href="/css/theme.css"> - <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";></script> - <script src="/js/bootstrap.min.js"></script> - <script src="/js/language-switch.js"></script> - <link rel="canonical" href="http://beam.incubator.apache.org/documentation/sdks/"; data-proofer-ignore> - <link rel="alternate" type="application/rss+xml" title="Apache Beam (incubating)" href="http://beam.incubator.apache.org/feed.xml";> - <script> - (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ - (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), - m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) - })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); - - ga('create', 'UA-73650088-1', 'auto'); - ga('send', 'pageview'); - - </script> - <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico"> -</head> - - - <body role="document"> - - <nav class="navbar navbar-default navbar-fixed-top"> - <div class="container"> - <div class="navbar-header"> - <a href="/" class="navbar-brand" > - <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png"> - </a> - <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> - <span class="sr-only">Toggle navigation</span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - </div> - <div id="navbar" class="navbar-collapse collapse"> - <ul class="nav navbar-nav"> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Get Started <span class="caret"></span></a> - <ul class="dropdown-menu"> - <li><a href="/get-started/beam-overview/">Beam Overview</a></li> - <li><a href="/get-started/quickstart/">Quickstart</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Example Walkthroughs</li> - <li><a href="/get-started/wordcount-example/">WordCount</a></li> - <li><a href="/get-started/mobile-gaming-example/">Mobile Gaming</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Resources</li> - <li><a href="/get-started/downloads">Downloads</a></li> - <li><a href="/get-started/releases">Release Notes</a></li> - <li><a href="/get-started/support">Support</a></li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a> - <ul class="dropdown-menu"> - <li><a href="/documentation">Using the Documentation</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Beam Concepts</li> - <li><a href="/documentation/programming-guide/">Programming Guide</a></li> - <li><a href="/documentation/resources/">Additional Resources</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Pipeline Fundamentals</li> - <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li> - <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li> - <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">SDKs</li> - <li><a href="/documentation/sdks/java/">Java SDK</a></li> - <li><a href="/documentation/sdks/javadoc/">Java SDK API Reference</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Runners</li> - <li><a href="/documentation/runners/capability-matrix/">Capability Matrix</a></li> - <li><a href="/documentation/runners/direct/">Direct Runner</a></li> - <li><a href="/documentation/runners/flink/">Apache Flink Runner</a></li> - <li><a href="/documentation/runners/spark/">Apache Spark Runner</a></li> - <li><a href="/documentation/runners/dataflow/">Cloud Dataflow Runner</a></li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Contribute <span class="caret"></span></a> - <ul class="dropdown-menu"> - <li><a href="/contribute">Get Started Contributing</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Guides</li> - <li><a href="/contribute/contribution-guide/">Contribution Guide</a></li> - <li><a href="/contribute/testing/">Testing Guide</a></li> - <li><a href="/contribute/release-guide/">Release Guide</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Technical References</li> - <li><a href="/contribute/design-principles/">Design Principles</a></li> - <li><a href="/contribute/work-in-progress/">Ongoing Projects</a></li> - <li><a href="/contribute/source-repository/">Source Repository</a></li> - <li role="separator" class="divider"></li> - <li class="dropdown-header">Promotion</li> - <li><a href="/contribute/presentation-materials/">Presentation Materials</a></li> - <li><a href="/contribute/logos/">Logos and Design</a></li> - <li role="separator" class="divider"></li> - <li><a href="/contribute/team/">Team</a></li> - </ul> - </li> - - <li><a href="/blog">Blog</a></li> - </ul> - <ul class="nav navbar-nav navbar-right"> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png"; alt="Apache Logo" style="height:24px;">Apache Software Foundation<span class="caret"></span></a> - <ul class="dropdown-menu dropdown-menu-right"> - <li><a href="http://www.apache.org/";>ASF Homepage</a></li> - <li><a href="http://www.apache.org/licenses/";>License</a></li> - <li><a href="http://www.apache.org/security/";>Security</a></li> - <li><a href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li> - <li><a href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> - <li><a href="https://www.apache.org/foundation/policies/conduct";>Code of Conduct</a></li> - </ul> - </li> - </ul> - </div><!--/.nav-collapse --> - </div> -</nav> - - -<link rel="stylesheet" href=""> - - - <div class="container" role="main"> - - <div class="row"> - <h1 id="apache-beam-sdks">Apache Beam SDKs</h1> - -<h4 id="java-sdklearnsdksjava"><a href="/learn/sdks/java/">Java SDK</a></h4> - - </div> - - - <hr> - <div class="row"> - <div class="col-xs-12"> - <footer> - <p class="text-center">© Copyright 2016 - <a href="http://www.apache.org";>The Apache Software Foundation.</a> All Rights Reserved.</p> - <p class="text-center"><a href="/privacy_policy">Privacy Policy</a> | - <a href="/feed.xml">RSS Feed</a></p> - </footer> - </div> - </div> - <!-- container div end --> -</div> - - - </body> - -</html> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5d3e72dc/content/feed.xml ---------------------------------------------------------------------- diff --git a/content/feed.xml b/content/feed.xml index 4ab11f7..e29720b 100644 --- a/content/feed.xml +++ b/content/feed.xml @@ -6,8 +6,8 @@ </description> <link>http://beam.incubator.apache.org/</link> <atom:link href="http://beam.incubator.apache.org/feed.xml"; rel="self" type="application/rss+xml"/> - <pubDate>Fri, 04 Nov 2016 17:50:55 -0700</pubDate> - <lastBuildDate>Fri, 04 Nov 2016 17:50:55 -0700</lastBuildDate> + <pubDate>Sat, 05 Nov 2016 22:12:20 -0700</pubDate> + <lastBuildDate>Sat, 05 Nov 2016 22:12:20 -0700</lastBuildDate> <generator>Jekyll v3.2.0</generator> <item> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5d3e72dc/content/get-started/quickstart/index.html ---------------------------------------------------------------------- diff --git a/content/get-started/quickstart/index.html b/content/get-started/quickstart/index.html index b1541d5..f2c6b9f 100644 --- a/content/get-started/quickstart/index.html +++ b/content/get-started/quickstart/index.html @@ -139,41 +139,210 @@ <div class="container" role="main"> <div class="row"> - <h1 id="apache-beam-quickstart">Apache Beam Quickstart</h1> + <h1 id="apache-beam-java-sdk-quickstart">Apache Beam Java SDK Quickstart</h1> -<p>The Apache Beam project is in the process of bootstrapping. This includes the creation of project resources, the refactoring of the initial code submission, and the formulation of project documentation, planning, and design documents. Until the project is fully initialized, this page contains useful resources to learn more about the model and tools which comprise Apache Beam.</p> +<p>This Quickstart will walk you through executing your first Beam pipeline to run <a href="/get-started/wordcount-example">WordCount</a>, written using Beamâs <a href="/documentation/sdks/java">Java SDK</a>, on a <a href="/documentation#runners">runner</a> of your choice.</p> -<h2 id="articles--slides">Articles & slides</h2> -<ul> - <li><a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101";>The world beyond batch: Streaming 101</a></li> - <li><a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102";>The world beyong batch: Streaming 102</a></li> - <li><a href="https://cloud.google.com/dataflow/blog/dataflow-beam-and-spark-comparison";>Dataflow/Beam & Spark: A Programming Model Comparison</a></li> - <li><a href="http://googlecloudplatform.blogspot.com/2016/01/Dataflow-and-open-source-proposal-to-join-the-Apache-Incubator.html";>Dataflow and open source - proposal to join the Apache Incubator</a></li> +<ul id="markdown-toc"> + <li><a href="#set-up-your-development-environment" id="markdown-toc-set-up-your-development-environment">Set up your Development Environment</a></li> + <li><a href="#get-the-wordcount-code" id="markdown-toc-get-the-wordcount-code">Get the WordCount Code</a></li> + <li><a href="#run-wordcount" id="markdown-toc-run-wordcount">Run WordCount</a></li> + <li><a href="#inspect-the-results" id="markdown-toc-inspect-the-results">Inspect the results</a></li> + <li><a href="#next-steps" id="markdown-toc-next-steps">Next Steps</a></li> </ul> -<h2 id="current-code">Current code</h2> -<p>The following GitHub repositories contain code which will be incorporated into Apache Beam.</p> +<h2 id="set-up-your-development-environment">Set up your Development Environment</h2> -<ul> - <li><a href="https://github.com/GoogleCloudPlatform/DataflowJavaSDK";>Dataflow Java SDK</a></li> - <li><a href="https://github.com/dataArtisans/flink-dataflow";>Flink Dataflow runner</a></li> - <li><a href="https://github.com/cloudera/spark-dataflow";>Spark Dataflow runner</a></li> -</ul> +<ol> + <li> + <p>Download and install the <a href="http://www.oracle.com/technetwork/java/javase/downloads/index.html";>Java Development Kit (JDK)</a> version 1.7 or later. Verify that the <a href="https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html";>JAVA_HOME</a> environment variable is set and points to your JDK installation.</p> + </li> + <li> + <p>Download and install <a href="http://maven.apache.org/download.cgi";>Apache Maven</a> by following Mavenâs <a href="http://maven.apache.org/install.html";>installation guide</a> for your specific operating system.</p> + </li> +</ol> -<p>These code repositories will be refactored and managed together (along with other code and new contributions) into a single repository.</p> +<h2 id="get-the-wordcount-code">Get the WordCount Code</h2> -<h2 id="documentation">Documentation</h2> -<ul> - <li><a href="https://goo.gl/KJrEl7";>Apache Beam incubation proposal</a></li> - <li><em>Apache Beam technical vision</em> - <ul> - <li><a href="https://goo.gl/5qZt3d";>Detailed</a></li> - <li><a href="https://goo.gl/nk5OM0";>Summary</a></li> - </ul> +<p>The easiest way to get a copy of the WordCount pipeline is to use the following command to generate a simple Maven project that contains Beamâs WordCount examples and builds against the most recent Beam release:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>$ mvn archetype:generate \ + -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \ + -DarchetypeVersion=LATEST \ + -DarchetypeGroupId=org.apache.beam \ + -DgroupId=org.example \ + -DartifactId=word-count-beam \ + -Dversion="0.1" \ + -DinteractiveMode=false \ + -Dpackage=org.apache.beam.examples +</code></pre> +</div> + +<p>This will create a directory <code class="highlighter-rouge">word-count-beam</code> that contains a simple <code class="highlighter-rouge">pom.xml</code> and a series of example pipelines that count words in text files.</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>$ cd beam-word-count/ + +$ ls +pom.xml src + +$ ls src/main/java/org/apache/beam/examples/ +DebuggingWordCount.java WindowedWordCount.java common +MinimalWordCount.java WordCount.java +</code></pre> +</div> + +<p>For a detailed introduction to the Beam concepts used in these examples, see the <a href="/get-started/wordcount-example">WordCount Example Walkthrough</a>. Here, weâll just focus on executing <code class="highlighter-rouge">WordCount.java</code>.</p> + +<h2 id="run-wordcount">Run WordCount</h2> + +<p>A single Beam pipeline can run on multiple Beam <a href="/documentation#runners">runners</a>, including the <a href="/documentation/runners/spark">SparkRunner</a>, <a href="/documentation/runners/flink">FlinkRunner</a>, or <a href="/documentation/runners/dataflow">DataflowRunner</a>. The <a href="/documentation/runners/direct">DirectRunner</a> is a common runner for getting started, as it runs locally on your machine and requires no specific setup.</p> + +<p>After youâve chosen which runner youâd like to use:</p> + +<ol> + <li>Ensure youâve done any runner-specific setup.</li> + <li>Build your commandline by: + <ol> + <li>Specifying a specific runner with <code class="highlighter-rouge">--runner=<runner></code> (defaults to the <a href="/documentation/runners/direct">DirectRunner</a>)</li> + <li>Adding any runner-specific required options</li> + <li>Choosing input files and an output location are accessible on the chosen runner. (For example, you canât access a local file if you are running the pipeline on an external cluster.)</li> + </ol> + </li> + <li> + <p>Run your first WordCount pipeline.</p> + + <ol> + <li> + <p><a href="/documentation/runners/direct">DirectRunner</a></p> + + <div class="highlighter-rouge"><pre class="highlight"><code>$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ + -Dexec.args="--inputFile=pom.xml --output=counts" +</code></pre> + </div> + </li> + <li> + <p><a href="/documentation/runners/flink">FlinkRunner</a></p> + + <div class="highlighter-rouge"><pre class="highlight"><code>TODO BEAM-899 +</code></pre> + </div> + </li> + <li> + <p><a href="/documentation/runners/spark">SparkRunner</a></p> + + <div class="highlighter-rouge"><pre class="highlight"><code>TODO BEAM-900 +</code></pre> + </div> + </li> + <li> + <p><a href="/documentation/runners/dataflow">DataflowRunner</a></p> + + <div class="highlighter-rouge"><pre class="highlight"><code>$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ + -Dexec.args="--runner=DataflowRunner --gcpTempLocation=gs://<your-gcs-bucket>/tmp \ + --inputFile=gs://apache-beam-samples/shakespeare/* --output=gs://<your-gcs-bucket>/counts" +</code></pre> + </div> + </li> + </ol> + </li> +</ol> + +<h2 id="inspect-the-results">Inspect the results</h2> + +<p>Once the pipeline has completed, you can view the output. Youâll notice that there may be multiple output files prefixed by <code class="highlighter-rouge">count</code>. The exact number of these files is decided by the runner, giving it the flexibility to do efficient, distributed execution.</p> + +<ol> + <li> + <p><a href="/documentation/runners/direct">DirectRunner</a></p> + + <div class="highlighter-rouge"><pre class="highlight"><code>$ ls counts* +</code></pre> + </div> + </li> + <li> + <p><a href="/documentation/runners/flink">FlinkRunner</a></p> + + <div class="highlighter-rouge"><pre class="highlight"><code>TODO BEAM-899 +</code></pre> + </div> + </li> + <li> + <p><a href="/documentation/runners/spark">SparkRunner</a></p> + + <div class="highlighter-rouge"><pre class="highlight"><code>TODO BEAM-900 +</code></pre> + </div> + </li> + <li> + <p><a href="/documentation/runners/dataflow">DataflowRunner</a></p> + + <div class="highlighter-rouge"><pre class="highlight"><code>$ gsutil ls gs://<your-gcs-bucket>/counts* +</code></pre> + </div> + </li> +</ol> + +<p>When you look into the contents of the file, youâll see that they contain unique words and the number of occurrences of each word. The order of elements within the file may differ because the Beam model does not generally guarantee ordering, again to allow runners to optimize for efficiency.</p> + +<ol> + <li> + <p><a href="/documentation/runners/direct">DirectRunner</a></p> + + <div class="highlighter-rouge"><pre class="highlight"><code>$ more counts* +api: 9 +bundled: 1 +old: 4 +Apache: 2 +The: 1 +limitations: 1 +Foundation: 1 +... +</code></pre> + </div> </li> - <li><a href="https://goo.gl/ps8twC";>Apache Beam technical documentation</a></li> + <li> + <p><a href="/documentation/runners/flink">FlinkRunner</a></p> + + <div class="highlighter-rouge"><pre class="highlight"><code>TODO BEAM-899 +</code></pre> + </div> + </li> + <li> + <p><a href="/documentation/runners/spark">SparkRunner</a></p> + + <div class="highlighter-rouge"><pre class="highlight"><code>TODO BEAM-900 +</code></pre> + </div> + </li> + <li> + <p><a href="/documentation/runners/dataflow">DataflowRunner</a></p> + + <div class="highlighter-rouge"><pre class="highlight"><code>$ gsutil cat gs://<your-gcs-bucket>/counts* +feature: 15 +smother'st: 1 +revelry: 1 +bashfulness: 1 +Bashful: 1 +Below: 2 +deserves: 32 +barrenly: 1 +... +</code></pre> + </div> + </li> +</ol> + +<h2 id="next-steps">Next Steps</h2> + +<ul> + <li>Learn more about these WordCount examples in the <a href="/get-started/wordcount-example">WordCount Example Walkthrough</a>.</li> + <li>Dive in to some of our favorite <a href="/documentation/resources">articles and presentations</a>.</li> + <li>Join the Beam <a href="/get-started/support#mailing-lists">users@</a> mailing list.</li> </ul> +<p>Please donât hesitate to <a href="/get-started/support">reach out</a> if you encounter any issues!</p> + + </div> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5d3e72dc/content/learn/runners/index.html ---------------------------------------------------------------------- diff --git a/content/learn/runners/index.html b/content/learn/runners/index.html deleted file mode 100644 index 770b3f4..0000000 --- a/content/learn/runners/index.html +++ /dev/null @@ -1,10 +0,0 @@ -<!DOCTYPE html> -<html lang="en-US"> -<meta charset="utf-8"> -<title>Redirectingâ¦</title> -<link rel="canonical" href="http://beam.incubator.apache.org/documentation/runners/";> -<meta http-equiv="refresh" content="0; url=http://beam.incubator.apache.org/documentation/runners/";> -<h1>Redirectingâ¦</h1> -<a href="http://beam.incubator.apache.org/documentation/runners/";>Click here if you are not redirected.</a> -<script>location="http://beam.incubator.apache.org/documentation/runners/";</script> -</html> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5d3e72dc/content/learn/sdks/index.html ---------------------------------------------------------------------- diff --git a/content/learn/sdks/index.html b/content/learn/sdks/index.html deleted file mode 100644 index 62e8620..0000000 --- a/content/learn/sdks/index.html +++ /dev/null @@ -1,10 +0,0 @@ -<!DOCTYPE html> -<html lang="en-US"> -<meta charset="utf-8"> -<title>Redirectingâ¦</title> -<link rel="canonical" href="http://beam.incubator.apache.org/documentation/sdks/";> -<meta http-equiv="refresh" content="0; url=http://beam.incubator.apache.org/documentation/sdks/";> -<h1>Redirectingâ¦</h1> -<a href="http://beam.incubator.apache.org/documentation/sdks/";>Click here if you are not redirected.</a> -<script>location="http://beam.incubator.apache.org/documentation/sdks/";</script> -</html>
![https://www.apache.org/foundation/press/kit/feather_small.png"](https://www.apache.org/foundation/press/kit/feather_small.png"){kind=link}