Regenerate website


Branch: refs/heads/asf-site
Commit: f439af099412e73da73a288cd212ff8e93221e35
Parents: 7e96f7b
Author: Dan Halperin <>
Authored: Thu Dec 1 14:31:10 2016 -0800
Committer: Dan Halperin <>
Committed: Thu Dec 1 14:31:10 2016 -0800

 content/documentation/runners/flink/index.html | 138 +++++++++++++++++++-
 1 file changed, 137 insertions(+), 1 deletion(-)
diff --git a/content/documentation/runners/flink/index.html 
index 6ccaff7..edd5bcd 100644
--- a/content/documentation/runners/flink/index.html
+++ b/content/documentation/runners/flink/index.html
@@ -146,7 +146,143 @@
       <div class="row">
         <h1 id="using-the-apache-flink-runner">Using the Apache Flink 
-<p>This page is under construction (<a 
+<p>The Apache Flink Runner can be used to execute Beam pipelines using <a 
href="";>Apache Flink</a>. When using the Flink Runner 
you will create a jar file containing your job that can be executed on a 
regular Flink cluster. It’s also possible to execute a Beam pipeline using 
Flink’s local execution mode without setting up a cluster. This is helpful 
for development and debugging of your pipeline.</p>
+<p>The Flink Runner and Flink are suitable for large scale, continuous jobs, 
and provide:</p>
+  <li>A streaming-first runtime that supports both batch processing and data 
streaming programs</li>
+  <li>A runtime that supports very high throughput and low event latency at 
the same time</li>
+  <li>Fault-tolerance with <em>exactly-once</em> processing guarantees</li>
+  <li>Natural back-pressure in streaming programs</li>
+  <li>Custom memory management for efficient and robust switching between 
in-memory and out-of-core data processing algorithms</li>
+  <li>Integration with YARN and other components of the Apache Hadoop 
+<p>The <a href="/documentation/runners/capability-matrix/">Beam Capability 
Matrix</a> documents the supported capabilities of the Flink Runner.</p>
+<h2 id="flink-runner-prerequisites-and-setup">Flink Runner prerequisites and 
+<p>If you want to use the local execution mode with the Flink runner to 
don’t have to complete any setup.</p>
+<p>To use the Flink Runner for executing on a cluster, you have to setup a 
Flink cluster by following the Flink <a 
+<p>To find out which version of Flink you need you can run this command to 
check the version of the Flink dependency that your project is using:</p>
+<div class="highlighter-rouge"><pre class="highlight"><code>$ mvn 
dependency:tree -Pflink-runner |grep flink
+[INFO] |  +- org.apache.flink:flink-streaming-java_2.10:jar:1.1.2:runtime
+<p>Here, we would need Flink 1.1.2.</p>
+<p>For more information, the <a 
Documentation</a> can be helpful.</p>
+<h3 id="specify-your-dependency">Specify your dependency</h3>
+<p>You must specify your dependency on the Flink Runner.</p>
+<div class="language-java highlighter-rouge"><pre 
class="highlight"><code><span class="o">&lt;</span><span 
class="n">dependency</span><span class="o">&gt;</span>
+  <span class="o">&lt;</span><span class="n">groupId</span><span 
class="o">&gt;</span><span class="n">org</span><span class="o">.</span><span 
class="na">apache</span><span class="o">.</span><span 
class="na">beam</span><span class="o">&lt;/</span><span 
class="n">groupId</span><span class="o">&gt;</span>
+  <span class="o">&lt;</span><span class="n">artifactId</span><span 
class="o">&gt;</span><span class="n">beam</span><span class="o">-</span><span 
class="n">runners</span><span class="o">-</span><span 
class="n">flink_2</span><span class="o">.</span><span class="mi">10</span><span 
class="o">&lt;/</span><span class="n">artifactId</span><span 
+  <span class="o">&lt;</span><span class="n">version</span><span 
class="o">&gt;</span><span class="mf">0.3</span><span class="o">.</span><span 
class="mi">0</span><span class="o">-</span><span 
class="n">incubating</span><span class="o">&lt;/</span><span 
class="n">version</span><span class="o">&gt;</span>
+  <span class="o">&lt;</span><span class="n">scope</span><span 
class="o">&gt;</span><span class="n">runtime</span><span 
class="o">&lt;/</span><span class="n">scope</span><span class="o">&gt;</span>
+<span class="o">&lt;/</span><span class="n">dependency</span><span 
+<h2 id="executing-a-pipeline-on-a-flink-cluster">Executing a pipeline on a 
Flink cluster</h2>
+<p>For executing a pipeline on a Flink cluster you need to package your 
program along will all dependencies in a so-called fat jar. How you do this 
depends on your build system but if you follow along the <a 
href="/get-started/quickstart/">Beam Quickstart</a> this is the command that 
you have to run:</p>
+<div class="highlighter-rouge"><pre class="highlight"><code>$ mvn package 
+<p>The Beam Quickstart Maven project is setup to use the Maven Shade plugin to 
create a fat jar and the <code class="highlighter-rouge">-Pflink-runner</code> 
argument makes sure to include the dependency on the Flink Runner.</p>
+<p>For actually running the pipeline you would use this command</p>
+<div class="highlighter-rouge"><pre class="highlight"><code>$ mvn exec:java 
-Dexec.mainClass=org.apache.beam.examples.WordCount \
+    -Pflink-runner \
+    -Dexec.args="--runner=FlinkRunner \
+      --inputFile=/path/to/pom.xml \
+      --output=/path/to/counts \
+      --flinkMaster=&lt;flink master url&gt; \
+      --filesToStage=target/word-count-beam--bundled-0.1.jar"
+<p>If you have a Flink <code class="highlighter-rouge">JobManager</code> 
running on your local machine you can give <code 
class="highlighter-rouge">localhost:6123</code> for
+<code class="highlighter-rouge">flinkMaster</code>.</p>
+<h2 id="pipeline-options-for-the-flink-runner">Pipeline options for the Flink 
+<p>When executing your pipeline with the Flink Runner, you can set these 
pipeline options.</p>
+<table class="table table-bordered">
+  <th>Field</th>
+  <th>Description</th>
+  <th>Default Value</th>
+  <td><code>runner</code></td>
+  <td>The pipeline runner to use. This option allows you to determine the 
pipeline runner at runtime.</td>
+  <td>Set to <code>FlinkRunner</code> to run using Flink.</td>
+  <td><code>streaming</code></td>
+  <td>Whether streaming mode is enabled or disabled; <code>true</code> if 
enabled. Set to <code>true</code> if running pipelines with unbounded 
+  <td><code>false</code></td>
+  <td><code>flinkMaster</code></td>
+  <td>The url of the Flink JobManager on which to execute pipelines. This can 
either be the address of a cluster JobManager, in the form 
<code>"host:port"</code> or one of the special Strings <code>"[local]"</code> 
or <code>"[auto]"</code>. <code>"[local]"</code> will start a local Flink 
Cluster in the JVM while <code>"[auto]"</code> will let the system decide where 
to execute the pipeline based on the environment.</td>
+  <td><code>[auto]</code></td>
+  <td><code>filesToStage</code></td>
+  <td>Jar Files to send to all workers and put on the classpath. Here you have 
to put the fat jar that contains your program along with all dependencies.</td>
+  <td>empty</td>
+  <td><code>parallelism</code></td>
+  <td>The degree of parallelism to be used when distributing operations onto 
+  <td><code>1</code></td>
+  <td><code>checkpointingInterval</code></td>
+  <td>The interval between consecutive checkpoints (i.e. snapshots of the 
current pipeline state used for fault tolerance).</td>
+  <td><code>-1L</code>, i.e. disabled</td>
+  <td><code>numberOfExecutionRetries</code></td>
+  <td>Sets the number of times that failed tasks are re-executed. A value of 
<code>0</code> effectively disables fault tolerance. A value of <code>-1</code> 
indicates that the system default value (as defined in the configuration) 
should be used.</td>
+  <td><code>-1</code></td>
+  <td><code>executionRetryDelay</code></td>
+  <td>Sets the delay between executions. A value of <code>-1</code> indicates 
that the default value should be used.</td>
+  <td><code>-1</code></td>
+  <td><code>stateBackend</code></td>
+  <td>Sets the state backend to use in streaming mode. The default is to read 
this setting from the Flink config.</td>
+  <td><code>empty</code>, i.e. read from Flink config</td>
+<p>See the reference documentation for the  <span class="language-java"><a 
 interface (and its subinterfaces) for the complete list of pipeline 
configuration options.</p>
+<h2 id="additional-information-and-caveats">Additional information and 
+<h3 id="monitoring-your-job">Monitoring your job</h3>
+<p>You can monitor a running Flink job using the Flink JobManager Dashboard. 
By default, this is available at port <code 
class="highlighter-rouge">8081</code> of the JobManager node. If you have a 
Flink installation on your local machine that would be <code 
+<h3 id="streaming-execution">Streaming Execution</h3>
+<p>If your pipeline uses an unbounded data source or sink, the Flink Runner 
will automatically switch to streaming mode. You can enforce streaming mode by 
using the <code class="highlighter-rouge">streaming</code> setting mentioned 

Reply via email to