Repository: incubator-beam-site Updated Branches: refs/heads/asf-site 3b54e40ce -> c90dbf184
Added the /docs/ directory and a landing page at docs/index.md. Added coming-soon.md as a placeholder for missing content. Modified main landing page with SDK and Runner status tables and descriptions. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/34ae3e35 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/34ae3e35 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/34ae3e35 Branch: refs/heads/asf-site Commit: 34ae3e35c71e63318fe1071e471bf52d78077463 Parents: 3b54e40 Author: Devin Donnelly <[email protected]> Authored: Wed Apr 27 14:18:58 2016 -0700 Committer: Devin Donnelly <[email protected]> Committed: Wed May 4 16:59:22 2016 -0700 ---------------------------------------------------------------------- coming-soon.md | 17 ++++ content/capability-matrix/index.html | 2 +- content/coming-soon.html | 126 +++++++++++++++++++++++++ content/contribution-guide/index.html | 4 +- content/docs/index.html | 144 +++++++++++++++++++++++++++++ content/feed.xml | 6 +- content/index.html | 82 ++++++++++++---- docs/index.md | 29 ++++++ index.md | 70 +++++++++++--- 9 files changed, 442 insertions(+), 38 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/34ae3e35/coming-soon.md ---------------------------------------------------------------------- diff --git a/coming-soon.md b/coming-soon.md new file mode 100644 index 0000000..370af20 --- /dev/null +++ b/coming-soon.md @@ -0,0 +1,17 @@ +--- +layout: default +--- +<p> + <div class="alert alert-info alert-dismissible" role="alert"> + <span class="glyphicon glyphicon-flag" aria-hidden="true"></span> + <button type="button" class="close" data-dismiss="alert" aria-label="Close"><span aria-hidden="true">×</span></button> + The Apache Beam project is in the process of bootstrapping. This includes the creation of project resources, the refactoring of the initial code submission, and the formulation of project documentation, planning, and design documents. For more information about Beam see the <a href="/getting_started/">getting started page</a>. + </div> +</p> + +# Documentation Coming Soon + +You've reached a page that's still in draft, or otherwise being developed! Please bear with us as we improve the documentation for Apache Beam. + +[Go Back](/) to the main Beam site. + http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/34ae3e35/content/capability-matrix/index.html ---------------------------------------------------------------------- diff --git a/content/capability-matrix/index.html b/content/capability-matrix/index.html index 543f899..e4d0c01 100644 --- a/content/capability-matrix/index.html +++ b/content/capability-matrix/index.html @@ -89,7 +89,7 @@ <div class="container"> <h1 id="apache-beam-capability-matrix">Apache Beam Capability Matrix</h1> -<p><span style="font-size:11px;float:none">Last updated: 2016-04-29 14:54 PDT</span></p> +<p><span style="font-size:11px;float:none">Last updated: 2016-05-04 16:48 PDT</span></p> <p>Apache Beam (incubating) provides a portable API layer for building sophisticated data-parallel processing engines that may be executed across a diversity of exeuction engines, or <i>runners</i>. The core concepts of this layer are based upon the Beam Model (formerly referred to as the <a href="http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf";>Dataflow Model</a>), and implemented to varying degrees in each Beam runner. To help clarify the capabilities of individual runners, weâve created the capability matrix below.</p> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/34ae3e35/content/coming-soon.html ---------------------------------------------------------------------- diff --git a/content/coming-soon.html b/content/coming-soon.html new file mode 100644 index 0000000..5491f6b --- /dev/null +++ b/content/coming-soon.html @@ -0,0 +1,126 @@ +<!DOCTYPE html> +<html lang="en"> + + <head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + + <title>Apache Beam (incubating)</title> + <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes. +"> + + <link rel="stylesheet" href="/styles/site.css"> + <link rel="stylesheet" href="/css/theme.css"> + <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";></script> + <script src="/js/bootstrap.min.js"></script> + <link rel="canonical" href="http://beam.incubator.apache.org/coming-soon.html";> + <link rel="alternate" type="application/rss+xml" title="Apache Beam (incubating)" href="http://beam.incubator.apache.org/feed.xml";> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-73650088-1', 'auto'); + ga('send', 'pageview'); + + </script> + <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico"> +</head> + + + <body role="document"> + + <nav class="navbar navbar-default navbar-fixed-top"> + <div class="container"> + <div class="navbar-header"> + <a href="/" class="navbar-brand" > + <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png"> + </a> + </div> + <div id="navbar" class="navbar-collapse collapse"> + <ul class="nav navbar-nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li class="dropdown-header">Guides</li> + <li><a href="/getting_started/">Getting Started</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Technical Documentation</li> + <li><a href="/capability-matrix/">Capability Matrix</a></li> + <li><a href="https://goo.gl/ps8twC";>Technical Docs</a></li> + <li><a href="https://goo.gl/nk5OM0";>Technical Vision</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Community <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li class="dropdown-header">Community</li> + <li><a href="/mailing_lists/">Mailing Lists</a></li> + <li><a href="/team/">Apache Beam Team</a></li> + <li><a href="/public-meetings/">Public Meetings</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Contribute</li> + <li><a href="/contribution-guide/">Contribution Guide</a></li> + <li><a href="/source_repository/">Source Repository</a></li> + <li><a href="/issue_tracking/">Issue Tracking</a></li> + </ul> + </li> + <li><a href="/blog">Blog</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Project <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/presentation-materials/">Presentation Materials</a></li> + <li><a href="/material/">Logos and design</a></li> + <li><a href="http://apache.org/licenses/LICENSE-2.0.html";>License</a></li> + </ul> + </li> + </ul> + </div><!--/.nav-collapse --> + </div> +</nav> + + +<link rel="stylesheet" href=""> + + + <div class="container" role="main"> + + <div class="container"> + <p> + <div class="alert alert-info alert-dismissible" role="alert"> + <span class="glyphicon glyphicon-flag" aria-hidden="true"></span> + <button type="button" class="close" data-dismiss="alert" aria-label="Close"><span aria-hidden="true">×</span></button> + The Apache Beam project is in the process of bootstrapping. This includes the creation of project resources, the refactoring of the initial code submission, and the formulation of project documentation, planning, and design documents. For more information about Beam see the <a href="/getting_started/">getting started page</a>. + </div> +</p> + +<h1 id="documentation-coming-soon">Documentation Coming Soon</h1> + +<p>Youâve reached a page thatâs still in draft, or otherwise being developed! Please bear with us as we improve the documentation for Apache Beam.</p> + +<p><a href="/">Go Back</a> to the main Beam site.</p> + + + </div> + + + <hr> + <div class="row"> + <div class="col-xs-12"> + <footer> + <p class="text-center">© Copyright 2016 + <a href="http://www.apache.org";>The Apache Software Foundation.</a> All Rights Reserved.</p> + <p class="text-center"><a href="/privacy_policy">Privacy Policy</a> | + <a href="/feed.xml">RSS Feed</a></p> + </footer> + </div> + </div> + <!-- container div end --> +</div> + + + </body> + +</html> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/34ae3e35/content/contribution-guide/index.html ---------------------------------------------------------------------- diff --git a/content/contribution-guide/index.html b/content/contribution-guide/index.html index 9deeeb5..23562f8 100644 --- a/content/contribution-guide/index.html +++ b/content/contribution-guide/index.html @@ -350,8 +350,8 @@ github https://github.com/apache/incubator-beam.git (push) <p>Fetch references from all remote repositories, and checkout the specific pull request branch.</p> <pre> -$ git fetch --all -$ git checkout -b finish-pr-<b><pull-request-#></b> github/pr/<b><pull-request-#></b></pre> +</code>$ git fetch --all +$ git checkout -b finish-pr-<b><pull-request-#></b> github/pr/<b><pull-request-#></b></code></pre> <p>At this point, you can commit any final touches to the pull request. For example, you should:</p> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/34ae3e35/content/docs/index.html ---------------------------------------------------------------------- diff --git a/content/docs/index.html b/content/docs/index.html new file mode 100644 index 0000000..db2395b --- /dev/null +++ b/content/docs/index.html @@ -0,0 +1,144 @@ +<!DOCTYPE html> +<html lang="en"> + + <head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + + <title>Apache Beam (incubating)</title> + <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes. +"> + + <link rel="stylesheet" href="/styles/site.css"> + <link rel="stylesheet" href="/css/theme.css"> + <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";></script> + <script src="/js/bootstrap.min.js"></script> + <link rel="canonical" href="http://beam.incubator.apache.org/docs/";> + <link rel="alternate" type="application/rss+xml" title="Apache Beam (incubating)" href="http://beam.incubator.apache.org/feed.xml";> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-73650088-1', 'auto'); + ga('send', 'pageview'); + + </script> + <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico"> +</head> + + + <body role="document"> + + <nav class="navbar navbar-default navbar-fixed-top"> + <div class="container"> + <div class="navbar-header"> + <a href="/" class="navbar-brand" > + <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png"> + </a> + </div> + <div id="navbar" class="navbar-collapse collapse"> + <ul class="nav navbar-nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li class="dropdown-header">Guides</li> + <li><a href="/getting_started/">Getting Started</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Technical Documentation</li> + <li><a href="/capability-matrix/">Capability Matrix</a></li> + <li><a href="https://goo.gl/ps8twC";>Technical Docs</a></li> + <li><a href="https://goo.gl/nk5OM0";>Technical Vision</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Community <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li class="dropdown-header">Community</li> + <li><a href="/mailing_lists/">Mailing Lists</a></li> + <li><a href="/team/">Apache Beam Team</a></li> + <li><a href="/public-meetings/">Public Meetings</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">Contribute</li> + <li><a href="/contribution-guide/">Contribution Guide</a></li> + <li><a href="/source_repository/">Source Repository</a></li> + <li><a href="/issue_tracking/">Issue Tracking</a></li> + </ul> + </li> + <li><a href="/blog">Blog</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Project <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/presentation-materials/">Presentation Materials</a></li> + <li><a href="/material/">Logos and design</a></li> + <li><a href="http://apache.org/licenses/LICENSE-2.0.html";>License</a></li> + </ul> + </li> + </ul> + </div><!--/.nav-collapse --> + </div> +</nav> + + +<link rel="stylesheet" href=""> + + + <div class="container" role="main"> + + <div class="container"> + <p> + <div class="alert alert-info alert-dismissible" role="alert"> + <span class="glyphicon glyphicon-flag" aria-hidden="true"></span> + <button type="button" class="close" data-dismiss="alert" aria-label="Close"><span aria-hidden="true">×</span></button> + The Apache Beam project is in the process of bootstrapping. This includes the creation of project resources, the refactoring of the initial code submission, and the formulation of project documentation, planning, and design documents. For more information about Beam see the <a href="/getting_started/">getting started page</a>. + </div> +</p> + +<h1 id="apache-beam-documentation">Apache Beam Documentation</h1> + +<p>Welcome to the Apache Beam project documentation! The following resources can help you use, integrate with, and contribute to Apache Beam.</p> + +<h2 id="using-beam">Using Beam</h2> + +<p>These resources help you build and run Beam pipelines for your data processing tasks.</p> + +<ul> + <li><a href="/coming-soon.html">Beam Programming Guide</a> - Learn how to use the Beam SDKs to define your data processing workflows.</li> + <li><a href="/coming-soon.html">Beam Example Walkthroughs</a> - Check out detailed walkthroughs of runnable Beam pipelines that use the Beam SDKs to tackle a variety of use cases.</li> +</ul> + +<h2 id="integrating-your-distribured-processing-back-end-with-beam">Integrating your Distribured Processing Back-end with Beam</h2> + +<ul> + <li><a href="/coming-soon.html">Beam Runner Developerâs Guide</a> - Learn how to build a Beam Runner to help Beam pipelines work with your distributed processing back-end.</li> +</ul> + +<h2 id="contributing-to-beam">Contributing to Beam</h2> + +<ul> + <li><a href="/contribution-guide/">Beam Contribution Guide</a> - Learn how to contribute to the various open-source Beam SDKs.</li> +</ul> + + </div> + + + <hr> + <div class="row"> + <div class="col-xs-12"> + <footer> + <p class="text-center">© Copyright 2016 + <a href="http://www.apache.org";>The Apache Software Foundation.</a> All Rights Reserved.</p> + <p class="text-center"><a href="/privacy_policy">Privacy Policy</a> | + <a href="/feed.xml">RSS Feed</a></p> + </footer> + </div> + </div> + <!-- container div end --> +</div> + + + </body> + +</html> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/34ae3e35/content/feed.xml ---------------------------------------------------------------------- diff --git a/content/feed.xml b/content/feed.xml index 3d9ce65..a897748 100644 --- a/content/feed.xml +++ b/content/feed.xml @@ -6,9 +6,9 @@ </description> <link>http://beam.incubator.apache.org/</link> <atom:link href="http://beam.incubator.apache.org/feed.xml"; rel="self" type="application/rss+xml"/> - <pubDate>Fri, 29 Apr 2016 14:54:11 -0700</pubDate> - <lastBuildDate>Fri, 29 Apr 2016 14:54:11 -0700</lastBuildDate> - <generator>Jekyll v3.1.2</generator> + <pubDate>Wed, 04 May 2016 16:48:02 -0700</pubDate> + <lastBuildDate>Wed, 04 May 2016 16:48:02 -0700</lastBuildDate> + <generator>Jekyll v3.1.3</generator> <item> <title>Apache Beam Presentation Materials</title> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/34ae3e35/content/index.html ---------------------------------------------------------------------- diff --git a/content/index.html b/content/index.html index 31bdacc..3c651e8 100644 --- a/content/index.html +++ b/content/index.html @@ -98,29 +98,72 @@ <h1 id="apache-beam-incubating">Apache Beam (incubating)</h1> -<p>Apache Beam is an open source, unified model and set of language-specific SDKs for defining data processing workflows that may then be executed on top of a set of supported runners, currently including <a href="http://flink.apache.org";>Apache Flink</a>, <a href="http://spark.apache.org";>Apache Spark</a>, and <a href="https://cloud.google.com/dataflow";>Google Cloud Dataflow</a>.</p> +<p>Apache Beam is an open source, unified programming model that you can use to create a data processing <strong>pipeline</strong>. You start by building a program that defines the pipeline using one of the open source Beam SDKs. The pipeline is then executed by one of Beamâs supported <strong>distributed processing back-ends</strong>, which include <a href="http://flink.apache.org";>Apache Flink</a>, <a href="http://spark.apache.org";>Apache Spark</a>, and <a href="https://cloud.google.com/dataflow";>Google Cloud Dataflow</a>.</p> + +<p>Beam is particularly useful for <a href="http://en.wikipedia.org/wiki/Embarassingly_parallel";>Embarrassingly Parallel</a> data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system.</p> + +<h2 id="apache-beam-sdks">Apache Beam SDKs</h2> + +<p>The Beam SDKs provide a unified programming model that can represent and transform data sets of any size, whether the input is a finite data set from a batch data source, or an infinite data set from a streaming data source. The Beam SDKs use the same classes to represent both bounded and unbounded data, and the same transforms to operate on that data. You use the Beam SDK of your choice to build a program that defines your data processing pipeline.</p> + +<p>Beam currently supports the following language-specific SDKs:</p> + +<table class="table table-condensed"> +<tr> + <th>Language</th> + <th>SDK Status</th> +</tr> +<tr> + <td>Java</td> + <td>Active Development</td> +</tr> +<tr> + <td>Python</td> + <td>Coming Soon</td> +</tr> +<tr> + <td>Other</td> + <td>TBD</td> +</tr> +</table> + +<h2 id="apache-beam-pipeline-runners">Apache Beam Pipeline Runners</h2> + +<p>The Beam Pipeline Runners translate the data processing pipeline you define with your Beam program into the API compatible with the distributed processing back-end of your choice. When you run your Beam program, youâll need to specify the appropriate runner for the back-end where you want to execute your pipeline.</p> + +<p>Beam currently supports Runners that work with the following distributed processing back-ends:</p> + +<table class="table table-condensed"> +<tr> + <th>Runner</th> + <th>Status</th> +</tr> +<tr> + <td>Google Cloud Dataflow</td> + <td>In Development</td> +</tr> +<tr> + <td>Apache Flink</td> + <td>In Development</td> +</tr> +<tr> + <td>Apache Spark</td> + <td>In Development</td> +</tr> +</table> + +<p><strong>Note:</strong> You can always execute your pipeline locally for testing and debugging purposes.</p> + +<h2 id="getting-started-with-apache-beam">Getting Started with Apache Beam</h2> + +<p>Interested in working with Apache Beam? Great! Hereâs how to get started:</p> -<h2 id="using-apache-beam">Using Apache Beam</h2> -<p>You can use Beam for nearly any kind of data processing task, including both batch and streaming data processing. Beam provides a unified data model that can represent any size data set, including an unbounded or infinite data set from a continuously updating data source such as Kafka.</p> - -<p>In particular, Beam pipelines can represent high-volume computations, where the steps in your job need to process an amount of data that exceeds the memory capacity of a cost-effective cluster. Beam is particularly useful for <a href="http://en.wikipedia.org/wiki/Embarassingly_parallel";>Embarrassingly Parallel</a> data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel.</p> - -<p>You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system.</p> - -<h2 id="programming-model">Programming Model</h2> -<p>Beam provides a simple and <a href="https://cloud.google.com/dataflow/model/programming-model";>elegant programming model</a> to express your data processing jobs. Each job is represented by a data processing pipeline that you create by writing a program with Beam. Each pipeline is an independent entity that reads some input data, performs some transforms on that data to gain useful or actionable intelligence about it, and produces some resulting output data. A pipelineâs transform might include filtering, grouping, comparing, or joining data.</p> - -<p>Beam provides several useful abstractions that allow you to think about your data processing pipeline in a simple, logical way. Beam simplifies the mechanics of large-scale parallel data processing, freeing you from the need to manage orchestration details such as partitioning your data and coordinating individual workers.</p> - -<h2 id="key-concepts">Key Concepts</h2> <ul> - <li><strong>Simple data representation.</strong> Beam uses a specialized collection class, called PCollection, to represent your pipeline data. This class can represent data sets of virtually unlimited size, including bounded and unbounded data collections.</li> - <li><strong>Powerful data transforms.</strong> Beam provides several core data transforms that you can apply to your data. These transforms, called PTransforms, are generic frameworks that apply functions that you provide across an entire data set.</li> - <li><strong>I/O APIs for a variety of data formats.</strong> Beam provides APIs that let your pipeline read and write data to and from a variety of formats and storage technologies. Your pipeline can read text files, Avro files, and more.</li> + <li>If you are interested in using Beam for your data processing tasks, start with the <a href="/docs/">Beam Programming Guide</a> and <a href="/docs/">Beam Examples</a>.</li> + <li>If youâre interested in creating a Beam Pipeline Runner for your distributed processing back-end, start with the <a href="/docs/">Beam Runner Developerâs Guide</a>.</li> + <li>If youâre interested in contributing to the Beam SDKs, start with the <a href="/contribution-guide/">Contribution Guide</a>.</li> </ul> -<p>See the <a href="https://cloud.google.com/dataflow/model/programming-model";>programming model documentation</a> to learn more about how Beam implements these concepts.</p> - <hr /> <div class="row"> @@ -149,6 +192,7 @@ <p>Apache Beam is an <a href="http://www.apache.org";>Apache Software Foundation project</a>, available under the Apache v2 license.</p> + </div> http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/34ae3e35/docs/index.md ---------------------------------------------------------------------- diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..8ca1550 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,29 @@ +--- +layout: default +--- +<p> + <div class="alert alert-info alert-dismissible" role="alert"> + <span class="glyphicon glyphicon-flag" aria-hidden="true"></span> + <button type="button" class="close" data-dismiss="alert" aria-label="Close"><span aria-hidden="true">×</span></button> + The Apache Beam project is in the process of bootstrapping. This includes the creation of project resources, the refactoring of the initial code submission, and the formulation of project documentation, planning, and design documents. For more information about Beam see the <a href="/getting_started/">getting started page</a>. + </div> +</p> + +# Apache Beam Documentation + +Welcome to the Apache Beam project documentation! The following resources can help you use, integrate with, and contribute to Apache Beam. + +## Using Beam + +These resources help you build and run Beam pipelines for your data processing tasks. + +* [Beam Programming Guide](/coming-soon.html) - Learn how to use the Beam SDKs to define your data processing workflows. +* [Beam Example Walkthroughs](/coming-soon.html) - Check out detailed walkthroughs of runnable Beam pipelines that use the Beam SDKs to tackle a variety of use cases. + +## Integrating your Distribured Processing Back-end with Beam + +* [Beam Runner Developer's Guide](/coming-soon.html) - Learn how to build a Beam Runner to help Beam pipelines work with your distributed processing back-end. + +## Contributing to Beam + +* [Beam Contribution Guide](/contribution-guide/) - Learn how to contribute to the various open-source Beam SDKs. http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/34ae3e35/index.md ---------------------------------------------------------------------- diff --git a/index.md b/index.md index 6111146..cee6606 100644 --- a/index.md +++ b/index.md @@ -11,26 +11,69 @@ layout: default # Apache Beam (incubating) -Apache Beam is an open source, unified model and set of language-specific SDKs for defining data processing workflows that may then be executed on top of a set of supported runners, currently including [Apache Flink](http://flink.apache.org), [Apache Spark](http://spark.apache.org), and [Google Cloud Dataflow](https://cloud.google.com/dataflow). +Apache Beam is an open source, unified programming model that you can use to create a data processing **pipeline**. You start by building a program that defines the pipeline using one of the open source Beam SDKs. The pipeline is then executed by one of Beam's supported **distributed processing back-ends**, which include [Apache Flink](http://flink.apache.org), [Apache Spark](http://spark.apache.org), and [Google Cloud Dataflow](https://cloud.google.com/dataflow). -## Using Apache Beam -You can use Beam for nearly any kind of data processing task, including both batch and streaming data processing. Beam provides a unified data model that can represent any size data set, including an unbounded or infinite data set from a continuously updating data source such as Kafka. +Beam is particularly useful for [Embarrassingly Parallel](http://en.wikipedia.org/wiki/Embarassingly_parallel) data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system. -In particular, Beam pipelines can represent high-volume computations, where the steps in your job need to process an amount of data that exceeds the memory capacity of a cost-effective cluster. Beam is particularly useful for [Embarrassingly Parallel](http://en.wikipedia.org/wiki/Embarassingly_parallel) data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. +## Apache Beam SDKs -You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system. +The Beam SDKs provide a unified programming model that can represent and transform data sets of any size, whether the input is a finite data set from a batch data source, or an infinite data set from a streaming data source. The Beam SDKs use the same classes to represent both bounded and unbounded data, and the same transforms to operate on that data. You use the Beam SDK of your choice to build a program that defines your data processing pipeline. -## Programming Model -Beam provides a simple and [elegant programming model](https://cloud.google.com/dataflow/model/programming-model) to express your data processing jobs. Each job is represented by a data processing pipeline that you create by writing a program with Beam. Each pipeline is an independent entity that reads some input data, performs some transforms on that data to gain useful or actionable intelligence about it, and produces some resulting output data. A pipelineâs transform might include filtering, grouping, comparing, or joining data. +Beam currently supports the following language-specific SDKs: -Beam provides several useful abstractions that allow you to think about your data processing pipeline in a simple, logical way. Beam simplifies the mechanics of large-scale parallel data processing, freeing you from the need to manage orchestration details such as partitioning your data and coordinating individual workers. +<table class="table table-condensed"> +<tr> + <th>Language</th> + <th>SDK Status</th> +</tr> +<tr> + <td>Java</td> + <td>Active Development</td> +</tr> +<tr> + <td>Python</td> + <td>Coming Soon</td> +</tr> +<tr> + <td>Other</td> + <td>TBD</td> +</tr> +</table> -## Key Concepts -* **Simple data representation.** Beam uses a specialized collection class, called PCollection, to represent your pipeline data. This class can represent data sets of virtually unlimited size, including bounded and unbounded data collections. -* **Powerful data transforms.** Beam provides several core data transforms that you can apply to your data. These transforms, called PTransforms, are generic frameworks that apply functions that you provide across an entire data set. -* **I/O APIs for a variety of data formats.** Beam provides APIs that let your pipeline read and write data to and from a variety of formats and storage technologies. Your pipeline can read text files, Avro files, and more. +## Apache Beam Pipeline Runners -See the [programming model documentation](https://cloud.google.com/dataflow/model/programming-model) to learn more about how Beam implements these concepts. +The Beam Pipeline Runners translate the data processing pipeline you define with your Beam program into the API compatible with the distributed processing back-end of your choice. When you run your Beam program, you'll need to specify the appropriate runner for the back-end where you want to execute your pipeline. + +Beam currently supports Runners that work with the following distributed processing back-ends: + +<table class="table table-condensed"> +<tr> + <th>Runner</th> + <th>Status</th> +</tr> +<tr> + <td>Google Cloud Dataflow</td> + <td>In Development</td> +</tr> +<tr> + <td>Apache Flink</td> + <td>In Development</td> +</tr> +<tr> + <td>Apache Spark</td> + <td>In Development</td> +</tr> +</table> + +**Note:** You can always execute your pipeline locally for testing and debugging purposes. + +## Getting Started with Apache Beam + +Interested in working with Apache Beam? Great! Here's how to get started: + +* If you are interested in using Beam for your data processing tasks, start with the [Beam Programming Guide](/docs/) and [Beam Examples](/docs/). +* If you're interested in creating a Beam Pipeline Runner for your distributed processing back-end, start with the [Beam Runner Developer's Guide](/docs/). +* If you're interested in contributing to the Beam SDKs, start with the [Contribution Guide](/contribution-guide/). <hr> <div class="row"> @@ -52,3 +95,4 @@ See the [programming model documentation](https://cloud.google.com/dataflow/mode ## Apache Project Apache Beam is an [Apache Software Foundation project](http://www.apache.org), available under the Apache v2 license. +
