Modified: flink/site/docs/0.6-incubating/spargel_guide.html URL: http://svn.apache.org/viewvc/flink/site/docs/0.6-incubating/spargel_guide.html?rev=1657551&r1=1657550&r2=1657551&view=diff ============================================================================== --- flink/site/docs/0.6-incubating/spargel_guide.html (original) +++ flink/site/docs/0.6-incubating/spargel_guide.html Thu Feb 5 12:21:38 2015 @@ -5,143 +5,170 @@ <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Apache Flink (incubating): Spargel Graph Processing API</title> - <link rel="stylesheet" href="/css/bootstrap.css"> - <link rel="stylesheet" href="/css/bootstrap-lumen-custom.css"> - <link rel="stylesheet" href="/css/syntax.css"> - <link rel="/css/custom.css"> - <link rel="css/codetabs.css"> - <link href="//maxcdn.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css" rel="stylesheet"> + <link rel="shortcut icon" href="favicon.ico" type="image/x-icon"> + <link rel="icon" href="favicon.ico" type="image/x-icon"> + <link rel="stylesheet" href="css/bootstrap.css"> + <link rel="stylesheet" href="css/bootstrap-lumen-custom.css"> + <link rel="stylesheet" href="css/syntax.css"> + <link rel="stylesheet" href="css/custom.css"> + <link href="css/main/main.css" rel="stylesheet"> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js";></script> - <script src="/js/bootstrap.min.js"></script> + <script src="js/bootstrap.min.js"></script> <script src="js/codetabs.js"></script> </head> <body> <nav class="navbar navbar-default navbar-fixed-top" role="navigation"> <div class="container"> - <div class="navbar-header"> - <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse"> - <span class="sr-only">Toggle navigation</span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - <a class="navbar-brand" href="http://flink.incubator.apache.org";>Apache Flink</a> - </div> - - <div class="collapse navbar-collapse" id="navbar-collapse-1"> - <ul class="nav navbar-nav"> + <div class="row"> + <div class="col-md-1 af-mobile-nav-bar"> + <a href="index.html" title="Home"> + <img class="hidden-xs hidden-sm img-responsive" + src="img/logo.png" alt="Apache Flink Logo"> + </a> + <div class="row visible-xs"> + <div class="col-xs-3"> + <a href="index.html" title="Home"> + <img class="hidden-x hidden-sm img-responsive" + src="img/logo.png" alt="Apache Flink Logo"> + </a> + </div> + <div class="col-xs-5"></div> + <div class="col-xs-4"> + <div class="af-mobile-btn"> + <span class="glyphicon glyphicon-plus"></span> + </div> + </div> + </div> + </div> + <!-- Navigation --> + <div class="col-md-11"> + <div class="collapse navbar-collapse" id="navbar-collapse-1"> + <ul class="nav navbar-nav"> + + <li> + <a href="index.html" class="">Documentation</a> + </li> + + <li> + <a href="api/java/index.html">Javadoc</a> + </li> + + <li> + <a href="api/scala/index.html#org.apache.flink.api.scala.package">Scaladoc</a> + </li> - <li> - <a href="index.html" class="">Documentation</a> - </li> - - <li> - <a href="api/java/index.html">Javadoc</a> - </li> - </ul> + </ul> + </div> + </div> </div> </div> </nav> - <div style="padding-top:70px" class="container"> + + <div style="padding-top:120px" class="container"> <div class="row"> <div class="col-md-3"> <ul> - <li>Quickstart - <ul> - <li><a href="setup_quickstart.html">Install</a></li> - <li><a href="run_example_quickstart.html">Run Example</a></li> - <li><a href="java_api_quickstart.html">Java API</a></li> - <li><a href="scala_api_quickstart.html">Scala API</a></li> - <li><a href="faq.html">FAQ</a></li> - </ul> - </li> - - <li>Setup & Configuration - <ul> - <li><a href="building.html">Build Instructions</a></li> - <li><a href="local_setup.html">Local Setup</a></li> - <li><a href="cluster_setup.html">Cluster Setup</a></li> - <li><a href="yarn_setup.html">YARN Setup</a></li> - <li><a href="config.html">Configuration</a></li> - </ul> - </li> - - <li>Programming Guides - <ul> - <li><a href="java_api_guide.html">Java API</a></li> - <li><a href="java_api_transformations.html">Java API Transformations</a></li> - <li><a href="scala_api_guide.html">Scala API</a></li> - <li><a href="iterations.html">Iterations</a></li> - <li><a href="spargel_guide.html">Spargel Graph API</a></li> - </ul> - </li> - - <li>Examples - <ul> - <li><a href="java_api_examples.html">Java API</a></li> - <li><a href="scala_api_examples.html">Scala API</a></li> - <li><a href="example_connectors.html">Connecting to other systems</a></li> - </ul> - </li> - - <li>Execution - <ul> - <li><a href="local_execution.html">Local/Debugging</a></li> - <li><a href="cluster_execution.html">Cluster</a></li> - <li><a href="cli.html">Command-Line Interface</a></li> - <li><a href="web_client.html">Web Interface</a></li> - </ul> - </li> - - <li>Internals - <ul> - <li><a href="internal_overview.html">Overview</a></li> - <li><a href="internal_general_arch.html">General Architecture</a></li> - <li><a href="internal_add_operator.html">How-to: Adding a new Operator</a></li> - </ul> - </li> + <li><a href="faq.html">FAQ</a></li> + <li>Quickstart + <ul> + <li><a href="setup_quickstart.html">Setup</a></li> + <li><a href="run_example_quickstart.html">Run Example</a></li> + <li><a href="java_api_quickstart.html">Java API</a></li> + <li><a href="scala_api_quickstart.html">Scala API</a></li> + </ul> + </li> + + <li>Setup & Configuration + <ul> + <li><a href="local_setup.html">Local Setup</a></li> + <li><a href="building.html">Build Flink</a></li> + <li><a href="cluster_setup.html">Cluster Setup</a></li> + <li><a href="yarn_setup.html">YARN Setup</a></li> + <li><a href="config.html">Configuration</a></li> + </ul> + </li> + + <li>Programming Guides + <ul> + <li><a href="programming_guide.html">Programming Guide</a></li> + <li><a href="dataset_transformations.html">DataSet Transformations</a></li> + <li><a href="java8_programming_guide.html">Java 8 Programming Guide</a></li> + <li><a href="streaming_guide.html">Streaming Guide</a></li> + <li><a href="iterations.html">Iterations</a></li> + <li><a href="spargel_guide.html">Spargel Graph API</a></li> + <li><a href="hadoop_compatibility.html">Hadoop Compatibility</a></li> + </ul> + </li> + + <li>Examples + <ul> + <li><a href="examples.html">Bundled Examples</a></li> + <li><a href="example_connectors.html">Connecting to other systems</a></li> + </ul> + </li> + + <li>Execution + <ul> + <li><a href="local_execution.html">Local/Debugging</a></li> + <li><a href="cluster_execution.html">Cluster</a></li> + <li><a href="cli.html">Command-Line Interface</a></li> + <li><a href="web_client.html">Web Interface</a></li> + </ul> + </li> + + <li>Internals + <ul> + <li><a href="internal_overview.html">Overview</a></li> + </ul> + </li> </ul> + </div> <div class="col-md-9"> <h1>Spargel Graph Processing API</h1> + + <ul id="markdown-toc"> + <li><a href="#spargel-api">Spargel API</a></li> + <li><a href="#example-propagate-minimum-vertex-id-in-graph">Example: Propagate Minimum Vertex ID in Graph</a></li> +</ul> - <h1 id="spargel">Spargel</h1> - -<p>Spargel is our <a href="http://giraph.apache.org";>Giraph</a> like <strong>graph processing</strong> Java API. It supports basic graph computations, which are run as a sequence of <a href="iterations.html#supersteps">supersteps</a>. Spargel and Giraph both implement the <a href="https://en.wikipedia.org/wiki/Bulk_Synchronous_Parallel";>Bulk Synchronous Parallel (BSP)</a> programming model, propsed by Google's <a href="http://googleresearch.blogspot.de/2009/06/large-scale-graph-computing-at-google.html";>Pregel</a>.</p> +<p>Spargel is our <a href="http://giraph.apache.org";>Giraph</a> like <strong>graph processing</strong> Java API. It supports basic graph computations, which are run as a sequence of <a href="iterations.html#supersteps">supersteps</a>. Spargel and Giraph both implement the <a href="https://en.wikipedia.org/wiki/Bulk_Synchronous_Parallel";>Bulk Synchronous Parallel (BSP)</a> programming model, propsed by Googleâs <a href="http://googleresearch.blogspot.de/2009/06/large-scale-graph-computing-at-google.html";>Pregel</a>.</p> <p>The API provides a <strong>vertex-centric</strong> view on graph processing with two basic operations per superstep:</p> <ol> -<li><strong>Send messages</strong> to other vertices, and</li> -<li><strong>Receive messages</strong> from other vertices and <strong>update own vertex state</strong>.</li> + <li><strong>Send messages</strong> to other vertices, and</li> + <li><strong>Receive messages</strong> from other vertices and <strong>update own vertex state</strong>.</li> </ol> <p>This vertex-centric view makes it easy to express a large class of graph problems efficiently. We will list all <em>relevant interfaces</em> of the <strong>Spargel API</strong> to implement and walk through an <strong>example Spargel program</strong>.</p> <h2 id="spargel-api">Spargel API</h2> -<p>The Spargel API is part of the <em>addons</em> Maven project. All relevant classes are located in the <em>org.apache.flinkspargel.java</em> package.</p> +<p>The Spargel API is part of the <em>addons</em> Maven project. All relevant classes are located in the <em>org.apache.flink.spargel.java</em> package.</p> <p>Add the following dependency to your <code>pom.xml</code> to use the Spargel.</p> -<div class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt"><dependency></span> - <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span> - <span class="nt"><artifactId></span>spargel<span class="nt"></artifactId></span> - <span class="nt"><version></span>0.6.1-incubating<span class="nt"></version></span> -<span class="nt"></dependency></span> -</code></pre></div> + +<div class="highlight"><pre><code class="language-xml"><span class="nt"><dependency></span> + <span class="nt"><groupId></span>org.apache.flink<span class="nt"></groupId></span> + <span class="nt"><artifactId></span>flink-spargel<span class="nt"></artifactId></span> + <span class="nt"><version></span>0.7.0-incubating<span class="nt"></version></span> +<span class="nt"></dependency></span></code></pre></div> + <p>Extend <strong>VertexUpdateFunction<</strong><em>VertexKeyType</em>, <em>VertexValueType</em>, <em>MessageType</em><strong>></strong> to implement your <em>custom vertex update logic</em>.</p> <p>Extend <strong>MessagingFunction<</strong><em>VertexKeyType</em>, <em>VertexValueType</em>, <em>MessageType</em>, <em>EdgeValueType</em><strong>></strong> to implement your <em>custom message logic</em>.</p> <p>Create a <strong>SpargelIteration</strong> operator to include Spargel in your data flow.</p> -<h2 id="example:-propagate-minimum-vertex-id-in-graph">Example: Propagate Minimum Vertex ID in Graph</h2> +<h2 id="example-propagate-minimum-vertex-id-in-graph">Example: Propagate Minimum Vertex ID in Graph</h2> <p>The Spargel operator <strong>SpargelIteration</strong> includes Spargel graph processing into your data flow. As usual, it can be combined with other operators like <em>map</em>, <em>reduce</em>, <em>join</em>, etc.</p> -<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">FileDataSource</span> <span class="n">vertices</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">FileDataSource</span><span class="o">(...);</span> + +<div class="highlight"><pre><code class="language-java"><span class="n">FileDataSource</span> <span class="n">vertices</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">FileDataSource</span><span class="o">(...);</span> <span class="n">FileDataSource</span> <span class="n">edges</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">FileDataSource</span><span class="o">(...);</span> <span class="n">SpargelIteration</span> <span class="n">iteration</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">SpargelIteration</span><span class="o">(</span><span class="k">new</span> <span class="nf">MinMessager</span><span class="o">(),</span> <span class="k">new</span> <span class="nf">MinNeighborUpdater</span><span class="o">());</span> @@ -152,51 +179,53 @@ <span class="n">FileDataSink</span> <span class="n">result</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">FileDataSink</span><span class="o">(...);</span> <span class="n">result</span><span class="o">.</span><span class="na">setInput</span><span class="o">(</span><span class="n">iteration</span><span class="o">.</span><span class="na">getOutput</span><span class="o">());</span> -<span class="k">new</span> <span class="nf">Plan</span><span class="o">(</span><span class="n">result</span><span class="o">);</span> -</code></pre></div> +<span class="k">new</span> <span class="nf">Plan</span><span class="o">(</span><span class="n">result</span><span class="o">);</span></code></pre></div> + <p>Besides the <strong>program logic</strong> of vertex updates in <em>MinNeighborUpdater</em> and messages in <em>MinMessager</em>, you have to specify the <strong>initial vertex</strong> and <strong>edge input</strong>. Every vertex has a <strong>key</strong> and <strong>value</strong>. In each superstep, it <strong>receives messages</strong> from other vertices and updates its state:</p> <ul> -<li><strong>Vertex</strong> input: <strong>(id</strong>: <em>VertexKeyType</em>, <strong>value</strong>: <em>VertexValueType</em><strong>)</strong></li> -<li><strong>Edge</strong> input: <strong>(source</strong>: <em>VertexKeyType</em>, <strong>target</strong>: <em>VertexKeyType</em>[, <strong>value</strong>: <em>EdgeValueType</em>])</li> + <li><strong>Vertex</strong> input: <strong>(id</strong>: <em>VertexKeyType</em>, <strong>value</strong>: <em>VertexValueType</em><strong>)</strong></li> + <li><strong>Edge</strong> input: <strong>(source</strong>: <em>VertexKeyType</em>, <strong>target</strong>: <em>VertexKeyType</em>[, <strong>value</strong>: <em>EdgeValueType</em>])</li> </ul> -<p>For our example, we set the vertex ID as both <em>id and value</em> (initial minimum) and <em>leave out the edge values</em> as we don't need them:</p> +<p>For our example, we set the vertex ID as both <em>id and value</em> (initial minimum) and <em>leave out the edge values</em> as we donât need them:</p> <p class="text-center"> <img alt="Spargel Example Input" width="75%" src="img/spargel_example_input.png" /> </p> <p>In order to <strong>propagate the minimum vertex ID</strong>, we iterate over all received messages (which contain the neighboring IDs) and update our value, if we found a new minimum:</p> -<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MinNeighborUpdater</span> <span class="kd">extends</span> <span class="n">VertexUpdateFunction</span><span class="o"><</span><span class="n">IntValue</span><span class="o">,</span> <span class="n">IntValue</span><span class="o">,</span> <span class="n">IntValue</span><span class="o">></span> <span class="o">{</span> - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">updateVertex</span><span class="o">(</span><span class="n">IntValue</span> <span class="n">id</span><span class="o">,</span> <span class="n">IntValue</span> <span class="n">currentMin</span><span class="o">,</span> <span class="n">Iterator</span><span class="o"><</span><span class="n">IntValue</span><span class="o">></span> <span class="n">messages</span><span class="o">)</span> <span class="o">{</span> - <span class="kt">int</span> <span class="n">min</span> <span class="o">=</span> <span class="n">Integer</span><span class="o">.</span><span class="na">MAX_VALUE</span><span class="o">;</span> - - <span class="c1">// iterate over all received messages</span> - <span class="k">while</span> <span class="o">(</span><span class="n">messages</span><span class="o">.</span><span class="na">hasNext</span><span class="o">())</span> <span class="o">{</span> - <span class="kt">int</span> <span class="n">next</span> <span class="o">=</span> <span class="n">messages</span><span class="o">.</span><span class="na">next</span><span class="o">().</span><span class="na">getValue</span><span class="o">();</span> - <span class="n">min</span> <span class="o">=</span> <span class="n">next</span> <span class="o"><</span> <span class="n">min</span> <span class="o">?</span> <span class="n">next</span> <span class="o">:</span> <span class="n">min</span><span class="o">;</span> - <span class="o">}</span> - - <span class="c1">// update vertex value, if new minimum</span> - <span class="k">if</span> <span class="o">(</span><span class="n">min</span> <span class="o"><</span> <span class="n">currentMin</span><span class="o">.</span><span class="na">getValue</span><span class="o">())</span> <span class="o">{</span> - <span class="n">setNewVertexValue</span><span class="o">(</span><span class="k">new</span> <span class="nf">IntValue</span><span class="o">(</span><span class="n">min</span><span class="o">));</span> - <span class="o">}</span> - <span class="o">}</span> -<span class="o">}</span> -</code></pre></div> +<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MinNeighborUpdater</span> <span class="kd">extends</span> <span class="n">VertexUpdateFunction</span><span class="o"><</span><span class="n">IntValue</span><span class="o">,</span> <span class="n">IntValue</span><span class="o">,</span> <span class="n">IntValue</span><span class="o">></span> <span class="o">{</span> + + <span class="nd">@Override</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">updateVertex</span><span class="o">(</span><span class="n">IntValue</span> <span class="n">id</span><span class="o">,</span> <span class="n">IntValue</span> <span class="n">currentMin</span><span class="o">,</span> <span class="n">Iterator</span><span class="o"><</span><span class="n">IntValue</span><span class="o">></span> <span class="n">messages</span><span class="o">)</span> <span class="o">{</span> + <span class="kt">int</span> <span class="n">min</span> <span class="o">=</span> <span class="n">Integer</span><span class="o">.</span><span class="na">MAX_VALUE</span><span class="o">;</span> + + <span class="c1">// iterate over all received messages</span> + <span class="k">while</span> <span class="o">(</span><span class="n">messages</span><span class="o">.</span><span class="na">hasNext</span><span class="o">())</span> <span class="o">{</span> + <span class="kt">int</span> <span class="n">next</span> <span class="o">=</span> <span class="n">messages</span><span class="o">.</span><span class="na">next</span><span class="o">().</span><span class="na">getValue</span><span class="o">();</span> + <span class="n">min</span> <span class="o">=</span> <span class="n">next</span> <span class="o"><</span> <span class="n">min</span> <span class="o">?</span> <span class="n">next</span> <span class="o">:</span> <span class="n">min</span><span class="o">;</span> + <span class="o">}</span> + + <span class="c1">// update vertex value, if new minimum</span> + <span class="k">if</span> <span class="o">(</span><span class="n">min</span> <span class="o"><</span> <span class="n">currentMin</span><span class="o">.</span><span class="na">getValue</span><span class="o">())</span> <span class="o">{</span> + <span class="n">setNewVertexValue</span><span class="o">(</span><span class="k">new</span> <span class="nf">IntValue</span><span class="o">(</span><span class="n">min</span><span class="o">));</span> + <span class="o">}</span> + <span class="o">}</span> +<span class="o">}</span></code></pre></div> + <p>The <strong>messages in each superstep</strong> consist of the <strong>current minimum ID</strong> seen by the vertex:</p> -<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MinMessager</span> <span class="kd">extends</span> <span class="n">MessagingFunction</span><span class="o"><</span><span class="n">IntValue</span><span class="o">,</span> <span class="n">IntValue</span><span class="o">,</span> <span class="n">IntValue</span><span class="o">,</span> <span class="n">NullValue</span><span class="o">></span> <span class="o">{</span> - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">sendMessages</span><span class="o">(</span><span class="n">IntValue</span> <span class="n">id</span><span class="o">,</span> <span class="n">IntValue</span> <span class="n">currentMin</span><span class="o">)</span> <span class="o">{</span> - <span class="c1">// send current minimum to neighbors</span> - <span class="n">sendMessageToAllNeighbors</span><span class="o">(</span><span class="n">currentMin</span><span class="o">);</span> +<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MinMessager</span> <span class="kd">extends</span> <span class="n">MessagingFunction</span><span class="o"><</span><span class="n">IntValue</span><span class="o">,</span> <span class="n">IntValue</span><span class="o">,</span> <span class="n">IntValue</span><span class="o">,</span> <span class="n">NullValue</span><span class="o">></span> <span class="o">{</span> + + <span class="nd">@Override</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">sendMessages</span><span class="o">(</span><span class="n">IntValue</span> <span class="n">id</span><span class="o">,</span> <span class="n">IntValue</span> <span class="n">currentMin</span><span class="o">)</span> <span class="o">{</span> + <span class="c1">// send current minimum to neighbors</span> + <span class="n">sendMessageToAllNeighbors</span><span class="o">(</span><span class="n">currentMin</span><span class="o">);</span> <span class="o">}</span> -<span class="o">}</span> -</code></pre></div> +<span class="o">}</span></code></pre></div> + <p>The <strong>API-provided method</strong> <code>sendMessageToAllNeighbors(MessageType)</code> sends the message to all neighboring vertices. It is also possible to address specific vertices with <code>sendMessageTo(VertexKeyType, MessageType)</code>.</p> <p>If the value of a vertex does not change during a superstep, it will <strong>not send</strong> any messages in the superstep. This allows to do incremental updates to the <strong>hot (changing) parts</strong> of the graph, while leaving <strong>cold (steady) parts</strong> untouched.</p> @@ -207,7 +236,8 @@ <img alt="Spargel Example" width="75%" src="img/spargel_example.png" /> </p> - + + <!-- Disqus Area --> <div style="padding-top:30px" id="disqus_thread"></div> <script type="text/javascript"> @@ -222,14 +252,12 @@ })(); </script> <noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript> - <a href="http://disqus.com"; class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a> - </div> </div> <div class="footer"> - <p><hr class="divider"></p> + <hr class="divider" /> <p><small>Apache Flink is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is @@ -240,9 +268,10 @@ incubation status is not necessarily a r stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</small></p> -<p><a href="http://incubator.apache.org/";><img src="/img/apache-incubator-logo.png" alt="Incubator Logo"></a></p> +<p><a href="http://incubator.apache.org/";><img src="/img/apache-incubator-logo.png" alt="Incubator Logo" /></a></p> -<p class="text-center"><a href="/privacy-policy.html">Privacy Policy<a></p> +<p class="text-center"><a href="privacy-policy.html">Privacy Policy<a> +</a></a></p> </div> </div>
Modified: flink/site/docs/0.6-incubating/web_client.html URL: http://svn.apache.org/viewvc/flink/site/docs/0.6-incubating/web_client.html?rev=1657551&r1=1657550&r2=1657551&view=diff ============================================================================== --- flink/site/docs/0.6-incubating/web_client.html (original) +++ flink/site/docs/0.6-incubating/web_client.html Thu Feb 5 12:21:38 2015 @@ -5,161 +5,200 @@ <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Apache Flink (incubating): Web Client</title> - <link rel="stylesheet" href="/css/bootstrap.css"> - <link rel="stylesheet" href="/css/bootstrap-lumen-custom.css"> - <link rel="stylesheet" href="/css/syntax.css"> - <link rel="/css/custom.css"> - <link rel="css/codetabs.css"> - <link href="//maxcdn.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css" rel="stylesheet"> + <link rel="shortcut icon" href="favicon.ico" type="image/x-icon"> + <link rel="icon" href="favicon.ico" type="image/x-icon"> + <link rel="stylesheet" href="css/bootstrap.css"> + <link rel="stylesheet" href="css/bootstrap-lumen-custom.css"> + <link rel="stylesheet" href="css/syntax.css"> + <link rel="stylesheet" href="css/custom.css"> + <link href="css/main/main.css" rel="stylesheet"> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js";></script> - <script src="/js/bootstrap.min.js"></script> + <script src="js/bootstrap.min.js"></script> <script src="js/codetabs.js"></script> </head> <body> <nav class="navbar navbar-default navbar-fixed-top" role="navigation"> <div class="container"> - <div class="navbar-header"> - <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse"> - <span class="sr-only">Toggle navigation</span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - <a class="navbar-brand" href="http://flink.incubator.apache.org";>Apache Flink</a> - </div> - - <div class="collapse navbar-collapse" id="navbar-collapse-1"> - <ul class="nav navbar-nav"> + <div class="row"> + <div class="col-md-1 af-mobile-nav-bar"> + <a href="index.html" title="Home"> + <img class="hidden-xs hidden-sm img-responsive" + src="img/logo.png" alt="Apache Flink Logo"> + </a> + <div class="row visible-xs"> + <div class="col-xs-3"> + <a href="index.html" title="Home"> + <img class="hidden-x hidden-sm img-responsive" + src="img/logo.png" alt="Apache Flink Logo"> + </a> + </div> + <div class="col-xs-5"></div> + <div class="col-xs-4"> + <div class="af-mobile-btn"> + <span class="glyphicon glyphicon-plus"></span> + </div> + </div> + </div> + </div> + <!-- Navigation --> + <div class="col-md-11"> + <div class="collapse navbar-collapse" id="navbar-collapse-1"> + <ul class="nav navbar-nav"> + + <li> + <a href="index.html" class="">Documentation</a> + </li> + + <li> + <a href="api/java/index.html">Javadoc</a> + </li> + + <li> + <a href="api/scala/index.html#org.apache.flink.api.scala.package">Scaladoc</a> + </li> - <li> - <a href="index.html" class="">Documentation</a> - </li> - - <li> - <a href="api/java/index.html">Javadoc</a> - </li> - </ul> + </ul> + </div> + </div> </div> </div> </nav> - <div style="padding-top:70px" class="container"> + + <div style="padding-top:120px" class="container"> <div class="row"> <div class="col-md-3"> <ul> - <li>Quickstart - <ul> - <li><a href="setup_quickstart.html">Install</a></li> - <li><a href="run_example_quickstart.html">Run Example</a></li> - <li><a href="java_api_quickstart.html">Java API</a></li> - <li><a href="scala_api_quickstart.html">Scala API</a></li> - <li><a href="faq.html">FAQ</a></li> - </ul> - </li> - - <li>Setup & Configuration - <ul> - <li><a href="building.html">Build Instructions</a></li> - <li><a href="local_setup.html">Local Setup</a></li> - <li><a href="cluster_setup.html">Cluster Setup</a></li> - <li><a href="yarn_setup.html">YARN Setup</a></li> - <li><a href="config.html">Configuration</a></li> - </ul> - </li> - - <li>Programming Guides - <ul> - <li><a href="java_api_guide.html">Java API</a></li> - <li><a href="java_api_transformations.html">Java API Transformations</a></li> - <li><a href="scala_api_guide.html">Scala API</a></li> - <li><a href="iterations.html">Iterations</a></li> - <li><a href="spargel_guide.html">Spargel Graph API</a></li> - </ul> - </li> - - <li>Examples - <ul> - <li><a href="java_api_examples.html">Java API</a></li> - <li><a href="scala_api_examples.html">Scala API</a></li> - <li><a href="example_connectors.html">Connecting to other systems</a></li> - </ul> - </li> - - <li>Execution - <ul> - <li><a href="local_execution.html">Local/Debugging</a></li> - <li><a href="cluster_execution.html">Cluster</a></li> - <li><a href="cli.html">Command-Line Interface</a></li> - <li><a href="web_client.html">Web Interface</a></li> - </ul> - </li> - - <li>Internals - <ul> - <li><a href="internal_overview.html">Overview</a></li> - <li><a href="internal_general_arch.html">General Architecture</a></li> - <li><a href="internal_add_operator.html">How-to: Adding a new Operator</a></li> - </ul> - </li> + <li><a href="faq.html">FAQ</a></li> + <li>Quickstart + <ul> + <li><a href="setup_quickstart.html">Setup</a></li> + <li><a href="run_example_quickstart.html">Run Example</a></li> + <li><a href="java_api_quickstart.html">Java API</a></li> + <li><a href="scala_api_quickstart.html">Scala API</a></li> + </ul> + </li> + + <li>Setup & Configuration + <ul> + <li><a href="local_setup.html">Local Setup</a></li> + <li><a href="building.html">Build Flink</a></li> + <li><a href="cluster_setup.html">Cluster Setup</a></li> + <li><a href="yarn_setup.html">YARN Setup</a></li> + <li><a href="config.html">Configuration</a></li> + </ul> + </li> + + <li>Programming Guides + <ul> + <li><a href="programming_guide.html">Programming Guide</a></li> + <li><a href="dataset_transformations.html">DataSet Transformations</a></li> + <li><a href="java8_programming_guide.html">Java 8 Programming Guide</a></li> + <li><a href="streaming_guide.html">Streaming Guide</a></li> + <li><a href="iterations.html">Iterations</a></li> + <li><a href="spargel_guide.html">Spargel Graph API</a></li> + <li><a href="hadoop_compatibility.html">Hadoop Compatibility</a></li> + </ul> + </li> + + <li>Examples + <ul> + <li><a href="examples.html">Bundled Examples</a></li> + <li><a href="example_connectors.html">Connecting to other systems</a></li> + </ul> + </li> + + <li>Execution + <ul> + <li><a href="local_execution.html">Local/Debugging</a></li> + <li><a href="cluster_execution.html">Cluster</a></li> + <li><a href="cli.html">Command-Line Interface</a></li> + <li><a href="web_client.html">Web Interface</a></li> + </ul> + </li> + + <li>Internals + <ul> + <li><a href="internal_overview.html">Overview</a></li> + </ul> + </li> </ul> + </div> <div class="col-md-9"> <h1>Web Client</h1> + + <ul id="markdown-toc"> + <li><a href="#starting-stopping-and-configuring-the-web-interface">Starting, Stopping, and Configuring the Web Interface</a></li> + <li><a href="#using-the-web-interface">Using the Web Interface</a> <ul> + <li><a href="#job-view">Job View</a></li> + <li><a href="#plan-view">Plan View</a></li> + </ul> + </li> +</ul> - <p>Flink provides a web interface to upload jobs, inspect their execution plans, and execute them. The interface is a great tool to showcase programs, debug execution plans, or demonstrate the system as a whole.</p> +<p>Flink provides a web interface to upload jobs, inspect their execution plans, and execute them. The interface is a great tool to showcase programs, debug execution plans, or demonstrate the system as a whole.</p> -<h1 id="start,-stop,-and-configure-the-web-interface">Start, Stop, and Configure the Web Interface</h1> +<h2 id="starting-stopping-and-configuring-the-web-interface">Starting, Stopping, and Configuring the Web Interface</h2> <p>Start the web interface by executing:</p> -<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/start-webclient.sh + +<div class="highlight"><pre><code>./bin/start-webclient.sh </code></pre></div> + <p>and stop it by calling:</p> -<div class="highlight"><pre><code class="language-text" data-lang="text">./bin/stop-webclient.sh + +<div class="highlight"><pre><code>./bin/stop-webclient.sh </code></pre></div> -<p>The web interface runs on port 8080 by default. To specify a custom port set the <code>webclient.port</code> property in the <em>./conf/flink.yaml</em> configuration file. Jobs are submitted to the JobManager specified by <code>jobmanager.rpc.address</code> and <code>jobmanager.rpc.port</code>. Please consult the <a href="config.html#web_frontend">configuration</a> page for details and further configuration options.</p> -<h1 id="use-the-web-interface">Use the Web Interface</h1> +<p>The web interface runs on port 8080 by default. To specify a custom port set the <code>webclient.port</code> property in the <em>./conf/flink.yaml</em> configuration file. Jobs are submitted to the JobManager specified by <code>jobmanager.rpc.address</code> and <code>jobmanager.rpc.port</code>. Please consult the <a href="config.html#webclient">configuration</a> page for details and further configuration options.</p> + +<h2 id="using-the-web-interface">Using the Web Interface</h2> <p>The web interface provides two views:</p> <ol> -<li> The <strong>job view</strong> to upload, preview, and submit Flink programs.</li> -<li> The <strong>plan view</strong> to analyze the optimized execution plans of Flink programs.</li> + <li>The <strong>job view</strong> to upload, preview, and submit Flink programs.</li> + <li>The <strong>plan view</strong> to analyze the optimized execution plans of Flink programs.</li> </ol> -<h2 id="job-view">Job View</h2> +<h3 id="job-view">Job View</h3> <p>The interface starts serving the job view. </p> <p>You can <strong>upload</strong> a Flink program as a jar file. To <strong>execute</strong> an uploaded program:</p> <ul> -<li>select it from the job list on the left, </li> -<li>enter the program arguments in the <em>"Arguments"</em> field (bottom left), and </li> -<li>click on the <em>"Run Job"</em> button (bottom right).</li> + <li>select it from the job list on the left, </li> + <li>enter the program arguments in the <em>âArgumentsâ</em> field (bottom left), and </li> + <li>click on the <em>âRun Jobâ</em> button (bottom right).</li> </ul> <p>If the <em>âShow optimizer planâ</em> option is enabled (default), the <em>plan view</em> is display next, otherwise the job is directly submitted to the JobManager for execution.</p> -<p>In case the jar's manifest file does not specify the program class, you can specify it before the argument list as:</p> -<div class="highlight"><pre><code class="language-text" data-lang="text">assembler <assemblerClass> <programArgs...> -</code></pre></div> -<h2 id="plan-view">Plan View</h2> +<p>In case the jarâs manifest file does not specify the program class, you can specify it before the argument list as:</p> + +<p><code> +assembler <assemblerClass> <programArgs...> +</code></p> + +<h3 id="plan-view">Plan View</h3> <p>The plan view shows the optimized execution plan of the submitted program in the upper half of the page. The bottom part of the page displays detailed information about the currently selected plan operator including:</p> <ul> -<li>the chosen shipping strategies (local forward, hash partition, range partition, broadcast, ...),</li> -<li>the chosen local strategy (sort, hash join, merge join, ...),</li> -<li>inferred data properties (partitioning, grouping, sorting), and </li> -<li>used optimizer estimates (data size, I/O and network costs, ...).</li> + <li>the chosen shipping strategies (local forward, hash partition, range partition, broadcast, â¦),</li> + <li>the chosen local strategy (sort, hash join, merge join, â¦),</li> + <li>inferred data properties (partitioning, grouping, sorting), and </li> + <li>used optimizer estimates (data size, I/O and network costs, â¦).</li> </ul> -<p>To submit the job for execution, click again on the <em>"Run Job"</em> button in the bottom right.</p> - +<p>To submit the job for execution, click again on the <em>âRun Jobâ</em> button in the bottom right.</p> + + <!-- Disqus Area --> <div style="padding-top:30px" id="disqus_thread"></div> <script type="text/javascript"> @@ -174,14 +213,12 @@ })(); </script> <noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript> - <a href="http://disqus.com"; class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a> - </div> </div> <div class="footer"> - <p><hr class="divider"></p> + <hr class="divider" /> <p><small>Apache Flink is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is @@ -192,9 +229,10 @@ incubation status is not necessarily a r stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</small></p> -<p><a href="http://incubator.apache.org/";><img src="/img/apache-incubator-logo.png" alt="Incubator Logo"></a></p> +<p><a href="http://incubator.apache.org/";><img src="/img/apache-incubator-logo.png" alt="Incubator Logo" /></a></p> -<p class="text-center"><a href="/privacy-policy.html">Privacy Policy<a></p> +<p class="text-center"><a href="privacy-policy.html">Privacy Policy<a> +</a></a></p> </div> </div> Modified: flink/site/docs/0.6-incubating/yarn_setup.html URL: http://svn.apache.org/viewvc/flink/site/docs/0.6-incubating/yarn_setup.html?rev=1657551&r1=1657550&r2=1657551&view=diff ============================================================================== --- flink/site/docs/0.6-incubating/yarn_setup.html (original) +++ flink/site/docs/0.6-incubating/yarn_setup.html Thu Feb 5 12:21:38 2015 @@ -5,243 +5,331 @@ <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Apache Flink (incubating): YARN Setup</title> - <link rel="stylesheet" href="/css/bootstrap.css"> - <link rel="stylesheet" href="/css/bootstrap-lumen-custom.css"> - <link rel="stylesheet" href="/css/syntax.css"> - <link rel="/css/custom.css"> - <link rel="css/codetabs.css"> - <link href="//maxcdn.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css" rel="stylesheet"> + <link rel="shortcut icon" href="favicon.ico" type="image/x-icon"> + <link rel="icon" href="favicon.ico" type="image/x-icon"> + <link rel="stylesheet" href="css/bootstrap.css"> + <link rel="stylesheet" href="css/bootstrap-lumen-custom.css"> + <link rel="stylesheet" href="css/syntax.css"> + <link rel="stylesheet" href="css/custom.css"> + <link href="css/main/main.css" rel="stylesheet"> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js";></script> - <script src="/js/bootstrap.min.js"></script> + <script src="js/bootstrap.min.js"></script> <script src="js/codetabs.js"></script> </head> <body> <nav class="navbar navbar-default navbar-fixed-top" role="navigation"> <div class="container"> - <div class="navbar-header"> - <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse"> - <span class="sr-only">Toggle navigation</span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - <a class="navbar-brand" href="http://flink.incubator.apache.org";>Apache Flink</a> - </div> - - <div class="collapse navbar-collapse" id="navbar-collapse-1"> - <ul class="nav navbar-nav"> + <div class="row"> + <div class="col-md-1 af-mobile-nav-bar"> + <a href="index.html" title="Home"> + <img class="hidden-xs hidden-sm img-responsive" + src="img/logo.png" alt="Apache Flink Logo"> + </a> + <div class="row visible-xs"> + <div class="col-xs-3"> + <a href="index.html" title="Home"> + <img class="hidden-x hidden-sm img-responsive" + src="img/logo.png" alt="Apache Flink Logo"> + </a> + </div> + <div class="col-xs-5"></div> + <div class="col-xs-4"> + <div class="af-mobile-btn"> + <span class="glyphicon glyphicon-plus"></span> + </div> + </div> + </div> + </div> + <!-- Navigation --> + <div class="col-md-11"> + <div class="collapse navbar-collapse" id="navbar-collapse-1"> + <ul class="nav navbar-nav"> + + <li> + <a href="index.html" class="">Documentation</a> + </li> + + <li> + <a href="api/java/index.html">Javadoc</a> + </li> + + <li> + <a href="api/scala/index.html#org.apache.flink.api.scala.package">Scaladoc</a> + </li> - <li> - <a href="index.html" class="">Documentation</a> - </li> - - <li> - <a href="api/java/index.html">Javadoc</a> - </li> - </ul> + </ul> + </div> + </div> </div> </div> </nav> - <div style="padding-top:70px" class="container"> + + <div style="padding-top:120px" class="container"> <div class="row"> <div class="col-md-3"> <ul> - <li>Quickstart - <ul> - <li><a href="setup_quickstart.html">Install</a></li> - <li><a href="run_example_quickstart.html">Run Example</a></li> - <li><a href="java_api_quickstart.html">Java API</a></li> - <li><a href="scala_api_quickstart.html">Scala API</a></li> - <li><a href="faq.html">FAQ</a></li> - </ul> - </li> - - <li>Setup & Configuration - <ul> - <li><a href="building.html">Build Instructions</a></li> - <li><a href="local_setup.html">Local Setup</a></li> - <li><a href="cluster_setup.html">Cluster Setup</a></li> - <li><a href="yarn_setup.html">YARN Setup</a></li> - <li><a href="config.html">Configuration</a></li> - </ul> - </li> - - <li>Programming Guides - <ul> - <li><a href="java_api_guide.html">Java API</a></li> - <li><a href="java_api_transformations.html">Java API Transformations</a></li> - <li><a href="scala_api_guide.html">Scala API</a></li> - <li><a href="iterations.html">Iterations</a></li> - <li><a href="spargel_guide.html">Spargel Graph API</a></li> - </ul> - </li> - - <li>Examples - <ul> - <li><a href="java_api_examples.html">Java API</a></li> - <li><a href="scala_api_examples.html">Scala API</a></li> - <li><a href="example_connectors.html">Connecting to other systems</a></li> - </ul> - </li> - - <li>Execution - <ul> - <li><a href="local_execution.html">Local/Debugging</a></li> - <li><a href="cluster_execution.html">Cluster</a></li> - <li><a href="cli.html">Command-Line Interface</a></li> - <li><a href="web_client.html">Web Interface</a></li> - </ul> - </li> - - <li>Internals - <ul> - <li><a href="internal_overview.html">Overview</a></li> - <li><a href="internal_general_arch.html">General Architecture</a></li> - <li><a href="internal_add_operator.html">How-to: Adding a new Operator</a></li> - </ul> - </li> + <li><a href="faq.html">FAQ</a></li> + <li>Quickstart + <ul> + <li><a href="setup_quickstart.html">Setup</a></li> + <li><a href="run_example_quickstart.html">Run Example</a></li> + <li><a href="java_api_quickstart.html">Java API</a></li> + <li><a href="scala_api_quickstart.html">Scala API</a></li> + </ul> + </li> + + <li>Setup & Configuration + <ul> + <li><a href="local_setup.html">Local Setup</a></li> + <li><a href="building.html">Build Flink</a></li> + <li><a href="cluster_setup.html">Cluster Setup</a></li> + <li><a href="yarn_setup.html">YARN Setup</a></li> + <li><a href="config.html">Configuration</a></li> + </ul> + </li> + + <li>Programming Guides + <ul> + <li><a href="programming_guide.html">Programming Guide</a></li> + <li><a href="dataset_transformations.html">DataSet Transformations</a></li> + <li><a href="java8_programming_guide.html">Java 8 Programming Guide</a></li> + <li><a href="streaming_guide.html">Streaming Guide</a></li> + <li><a href="iterations.html">Iterations</a></li> + <li><a href="spargel_guide.html">Spargel Graph API</a></li> + <li><a href="hadoop_compatibility.html">Hadoop Compatibility</a></li> + </ul> + </li> + + <li>Examples + <ul> + <li><a href="examples.html">Bundled Examples</a></li> + <li><a href="example_connectors.html">Connecting to other systems</a></li> + </ul> + </li> + + <li>Execution + <ul> + <li><a href="local_execution.html">Local/Debugging</a></li> + <li><a href="cluster_execution.html">Cluster</a></li> + <li><a href="cli.html">Command-Line Interface</a></li> + <li><a href="web_client.html">Web Interface</a></li> + </ul> + </li> + + <li>Internals + <ul> + <li><a href="internal_overview.html">Overview</a></li> + </ul> + </li> </ul> + </div> <div class="col-md-9"> <h1>YARN Setup</h1> + + <ul id="markdown-toc"> + <li><a href="#in-a-nutshell">In a Nutshell</a></li> + <li><a href="#introducing-yarn">Introducing YARN</a> <ul> + <li><a href="#start-flink-session">Start Flink Session</a></li> + </ul> + </li> + <li><a href="#submit-job-to-flink">Submit Job to Flink</a></li> + <li><a href="#debugging-a-failed-yarn-session">Debugging a failed YARN session</a> <ul> + <li><a href="#log-files">Log Files</a></li> + <li><a href="#yarn-client-console--webinterfaces">YARN Client console & Webinterfaces</a></li> + </ul> + </li> + <li><a href="#build-yarn-client-for-a-specific-hadoop-version">Build YARN client for a specific Hadoop version</a></li> + <li><a href="#background">Background</a></li> +</ul> - <h1 id="in-a-nutshell">In a Nutshell</h1> +<h2 id="in-a-nutshell">In a Nutshell</h2> -<p>Start YARN session with 4 Taskmanagers (each with 4 GB of Heapspace):</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash">wget http://www.apache.org/dyn/closer.cgi/incubator/flink/flink-0.6.1-incubating/flink-0.6.1-incubating-bin-hadoop2-yarn.tgz -tar xvzf flink-0.6.1-incubating-bin-hadoop2-yarn.tgz -<span class="nb">cd </span>flink-yarn-0.6.1-incubating/ -./bin/yarn-session.sh -n <span class="m">4</span> -jm <span class="m">1024</span> -tm 4096 -</code></pre></div> -<h1 id="introducing-yarn">Introducing YARN</h1> +<p>Start YARN session with 4 Task Managers (each with 4 GB of Heapspace):</p> + +<div class="highlight"><pre><code class="language-bash">wget http://artfiles.org/apache.org/incubator/flink/flink-0.7.0-incubating/flink-0.7.0-incubating-bin-hadoop2-yarn.tgz +tar xvzf flink-0.7.0-incubating-bin-hadoop2-yarn.tgz +<span class="nb">cd </span>flink-yarn-0.7.0-incubating/ +./bin/yarn-session.sh -n <span class="m">4</span> -jm <span class="m">1024</span> -tm 4096</code></pre></div> + +<p>Specify the <code>-s</code> flag for the number of processing slots per Task Manager. We recommend to set the number of slots to the number of processors per machine.</p> + +<h2 id="introducing-yarn">Introducing YARN</h2> <p>Apache <a href="http://hadoop.apache.org/";>Hadoop YARN</a> is a cluster resource management framework. It allows to run various distributed applications on top of a cluster. Flink runs on YARN next to other applications. Users do not have to setup or install anything if there is already a YARN setup.</p> <p><strong>Requirements</strong></p> <ul> -<li>Apache Hadoop 2.2</li> -<li>HDFS</li> + <li>Apache Hadoop 2.2</li> + <li>HDFS (Hadoop Distributed File System)</li> </ul> -<p>If you have troubles using the Flink YARN client, have a look in the <a href="/docs/0.5/general/faq.html">FAQ section</a>.</p> +<p>If you have troubles using the Flink YARN client, have a look in the <a href="faq.html">FAQ section</a>.</p> -<h2 id="start-flink-session">Start Flink Session</h2> +<h3 id="start-flink-session">Start Flink Session</h3> <p>Follow these instructions to learn how to launch a Flink Session within your YARN cluster.</p> <p>A session will start all required Flink services (JobManager and TaskManagers) so that you can submit programs to the cluster. Note that you can run multiple programs per session.</p> -<h3 id="download-flink-for-yarn">Download Flink for YARN</h3> - -<p>Download the YARN tgz package on the <a href="/downloads/#nightly">download page</a>. It contains the required files.</p> +<h4 id="download-flink-for-yarn">Download Flink for YARN</h4> -<p>If you want to build the YARN .tgz file from sources, follow the build instructions. Make sure to use the <code>-Dhadoop.profile=2</code> profile. You can find the file in <code>flink-dist/target/flink-dist--yarn.tar.gz</code> (<em>Note: The version might be different for you</em> ).</p> +<p>Download the YARN tgz package on the <a href="/downloads.html">download page</a>. It contains the required files.</p> <p>Extract the package using:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash">tar xvzf flink-dist-0.6.1-incubating-yarn.tar.gz -<span class="nb">cd </span>flink-yarn-0.6.1-incubating/ -</code></pre></div> -<h3 id="start-a-session">Start a Session</h3> + +<div class="highlight"><pre><code class="language-bash">tar xvzf flink-0.7.0-incubating-bin-hadoop2-yarn.tgz +<span class="nb">cd </span>flink-yarn-0.7.0-incubating/</code></pre></div> + +<p>If you want to build the YARN .tgz file from sources, follow the <a href="building.html">build instructions</a>. Make sure to use the <code>-Dhadoop.profile=2</code> profile. You can find the result of the build in <code>flink-dist/target/flink-0.7.0-incubating-bin/flink-yarn-0.7.0-incubating/</code> (<em>Note: The version might be different for you</em> ).</p> + +<h4 id="start-a-session">Start a Session</h4> <p>Use the following command to start a session</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash">./bin/yarn-session.sh -</code></pre></div> + +<div class="highlight"><pre><code class="language-bash">./bin/yarn-session.sh</code></pre></div> + <p>This command will show you the following overview:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash">Usage: + +<div class="highlight"><pre><code class="language-bash">Usage: Required - -n,--container <arg> Number of Yarn container to allocate <span class="o">(=</span>Number of TaskTrackers<span class="o">)</span> + -n,--container <arg> Number of Yarn container to allocate <span class="o">(=</span>Number of Task Managers<span class="o">)</span> Optional + -D <arg> Dynamic Properties -jm,--jobManagerMemory <arg> Memory <span class="k">for</span> JobManager Container <span class="o">[</span>in MB<span class="o">]</span> -q,--query Display available YARN resources <span class="o">(</span>memory, cores<span class="o">)</span> -qu,--queue <arg> Specify YARN queue. + -s,--slots <arg> Number of slots per TaskManager -tm,--taskManagerMemory <arg> Memory per TaskManager Container <span class="o">[</span>in MB<span class="o">]</span> -tmc,--taskManagerCores <arg> Virtual CPU cores per TaskManager - -v,--verbose Verbose debug mode -</code></pre></div> + -v,--verbose Verbose debug mode</code></pre></div> + <p>Please note that the Client requires the <code>HADOOP_HOME</code> (or <code>YARN_CONF_DIR</code> or <code>HADOOP_CONF_DIR</code>) environment variable to be set to read the YARN and HDFS configuration.</p> -<p><strong>Example:</strong> Issue the following command to allocate 10 TaskTrackers, with 8 GB of memory each:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash">./bin/yarn-session.sh -n <span class="m">10</span> -tm 8192 -</code></pre></div> -<p>The system will use the configuration in <code>conf/flink-config.yaml</code>. Please follow our <a href="config.html">configuration guide</a> if you want to change something. Flink on YARN will overwrite the following configuration parameters <code>jobmanager.rpc.address</code> (because the JobManager is always allocated at different machines) and <code>taskmanager.tmp.dirs</code> (we are using the tmp directories given by YARN).</p> +<p><strong>Example:</strong> Issue the following command to allocate 10 Task Managers, with 8 GB of memory and 32 processing slots each:</p> + +<div class="highlight"><pre><code class="language-bash">./bin/yarn-session.sh -n <span class="m">10</span> -tm <span class="m">8192</span> -s 32</code></pre></div> + +<p>The system will use the configuration in <code>conf/flink-config.yaml</code>. Please follow our <a href="config.html">configuration guide</a> if you want to change something. </p> + +<p>Flink on YARN will overwrite the following configuration parameters <code>jobmanager.rpc.address</code> (because the JobManager is always allocated at different machines), <code>taskmanager.tmp.dirs</code> (we are using the tmp directories given by YARN) and <code>parallelization.degree.default</code> if the number of slots has been specified.</p> -<p>The example invocation starts 11 containers, since there is one additional container for the ApplicationMaster and JobTracker.</p> +<p>If you donât want to change the configuration file to pass configuration parameters, there is the option to pass dynamic properties via the <code>-D</code> flag. So you can pass parameters this way: <code>-Dfs.overwrite-files=true -Dtaskmanager.network.numberOfBuffers=16368</code>.</p> -<p>Once Flink is deployed in your YARN cluster, it will show you the connection details of the JobTracker.</p> +<p>The example invocation starts 11 containers, since there is one additional container for the ApplicationMaster and Job Manager.</p> + +<p>Once Flink is deployed in your YARN cluster, it will show you the connection details of the Job Manager.</p> <p>The client has to remain open to keep the deployment running. We suggest to use <code>screen</code>, which will start a detachable shell:</p> <ol> -<li>Open <code>screen</code>,</li> -<li>Start Flink on YARN,</li> -<li>Use <code>CTRL+a</code>, then press <code>d</code> to detach the screen session,</li> -<li>Use <code>screen -r</code> to resume again.</li> + <li>Open <code>screen</code>,</li> + <li>Start Flink on YARN,</li> + <li>Use <code>CTRL+a</code>, then press <code>d</code> to detach the screen session,</li> + <li>Use <code>screen -r</code> to resume again.</li> </ol> -<h1 id="submit-job-to-flink">Submit Job to Flink</h1> +<h2 id="submit-job-to-flink">Submit Job to Flink</h2> <p>Use the following command to submit a Flink program to the YARN cluster:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash">./bin/flink -</code></pre></div> + +<div class="highlight"><pre><code class="language-bash">./bin/flink</code></pre></div> + <p>Please refer to the documentation of the <a href="cli.html">commandline client</a>.</p> <p>The command will show you a help menu like this:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">[</span>...<span class="o">]</span> -Action <span class="s2">"run"</span> compiles and submits a Flink program. + +<div class="highlight"><pre><code class="language-bash"><span class="o">[</span>...<span class="o">]</span> +Action <span class="s2">"run"</span> compiles and runs a program. + + Syntax: run <span class="o">[</span>OPTIONS<span class="o">]</span> <jar-file> <arguments> <span class="s2">"run"</span> action arguments: - -a,--arguments <programArgs> Program arguments - -c,--class <classname> Program class - -j,--jarfile <jarfile> Flink program JAR file - -m,--jobmanager <host:port> Jobmanager to which the program is submitted - -w,--wait Wait <span class="k">for</span> program to finish -<span class="o">[</span>...<span class="o">]</span> -</code></pre></div> + -c,--class <classname> Class with the program entry point <span class="o">(</span><span class="s2">"main"</span> + method or <span class="s2">"getPlan()"</span> method. Only needed + <span class="k">if</span> the JAR file does not specify the class + in its manifest. + -m,--jobmanager <host:port> Address of the JobManager <span class="o">(</span>master<span class="o">)</span> to + which to connect. Use this flag to connect + to a different JobManager than the one + specified in the configuration. + -p,--parallelism <parallelism> The parallelism with which to run the + program. Optional flag to override the + default value specified in the + configuration</code></pre></div> + <p>Use the <em>run</em> action to submit a job to YARN. The client is able to determine the address of the JobManager. In the rare event of a problem, you can also pass the JobManager address using the <code>-m</code> argument. The JobManager address is visible in the YARN console.</p> <p><strong>Example</strong></p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash">wget -O apache-license-v2.txt http://www.apache.org/licenses/LICENSE-2.0.txt -./bin/flink run -j ./examples/flink-java-examples-0.6.1-incubating-WordCount.jar <span class="se">\</span> - -a <span class="m">1</span> file://<span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/apache-license-v2.txt file://<span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/wordcount-result.txt -</code></pre></div> +<div class="highlight"><pre><code class="language-bash">wget -O apache-license-v2.txt http://www.apache.org/licenses/LICENSE-2.0.txt + +./bin/flink run -j ./examples/flink-java-examples-0.7.0-incubating-WordCount.jar <span class="se">\</span> + -a <span class="m">1</span> file://<span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/apache-license-v2.txt file://<span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/wordcount-result.txt</code></pre></div> + <p>If there is the following error, make sure that all TaskManagers started:</p> -<div class="highlight"><pre><code class="language-bash" data-lang="bash">Exception in thread <span class="s2">"main"</span> org.apache.flinkcompiler.CompilerException: - Available instances could not be determined from job manager: Connection timed out. -</code></pre></div> + +<div class="highlight"><pre><code class="language-bash">Exception in thread <span class="s2">"main"</span> org.apache.flinkcompiler.CompilerException: + Available instances could not be determined from job manager: Connection timed out.</code></pre></div> + <p>You can check the number of TaskManagers in the JobManager web interface. The address of this interface is printed in the YARN session console.</p> <p>If the TaskManagers do not show up after a minute, you should investigate the issue using the log files.</p> -<h1 id="build-yarn-client-for-a-specific-hadoop-version">Build YARN client for a specific Hadoop version</h1> +<h2 id="debugging-a-failed-yarn-session">Debugging a failed YARN session</h2> + +<p>There are many reasons why a Flink YARN session deployment can fail. A misconfigured Hadoop setup (HDFS permissions, YARN configuration), version incompatibilities (running Flink with vanilla Hadoop dependencies on Cloudera Hadoop) or other errors.</p> + +<h3 id="log-files">Log Files</h3> + +<p>In cases where the Flink YARN session fails during the deployment itself, users have to rely on the logging capabilities of Hadoop YARN. The most useful feature for that is the <a href="http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/";>YARN log aggregation</a>. +To enable it, users have to set the <code>yarn.log-aggregation-enable</code> property to <code>true</code> in the <code>yarn-site.xml</code> file. +Once that is enabled, users can use the following command to retrieve all log files of a (failed) YARN session.</p> + +<div class="highlight"><pre><code>yarn logs -applicationId <application ID> +</code></pre></div> + +<p>Note that it takes a few seconds after the session has finished until the logs show up.</p> + +<h3 id="yarn-client-console--webinterfaces">YARN Client console & Webinterfaces</h3> + +<p>The Flink YARN client also prints error messages in the terminal if errors occur during runtime (for example if a TaskManager stops working after some time).</p> + +<p>In addition to that, there is the YARN Resource Manager webinterface (by default on port 8088). The port of the Resource Manager web interface is determined by the <code>yarn.resourcemanager.webapp.address</code> configuration value. </p> + +<p>It allows to access log files for running YARN applications and shows diagnostics for failed apps.</p> + +<h2 id="build-yarn-client-for-a-specific-hadoop-version">Build YARN client for a specific Hadoop version</h2> <p>Users using Hadoop distributions from companies like Hortonworks, Cloudera or MapR might have to build Flink against their specific versions of Hadoop (HDFS) and YARN. Please read the <a href="building.html">build instructions</a> for more details.</p> -<h1 id="background">Background</h1> +<h2 id="background">Background</h2> <p>This section briefly describes how Flink and YARN interact. </p> -<p><img src="img/FlinkOnYarn.svg" class="img-responsive"></p> +<p><img src="img/FlinkOnYarn.svg" class="img-responsive" /></p> <p>The YARN client needs to access the Hadoop configuration to connect to the YARN resource manager and to HDFS. It determines the Hadoop configuration using the following strategy:</p> <ul> -<li>Test if <code>YARN_CONF_DIR</code>, <code>HADOOP_CONF_DIR</code> or <code>HADOOP_CONF_PATH</code> are set (in that order). If one of these variables are set, they are used to read the configuration.</li> -<li>If the above strategy fails (this should not be the case in a correct YARN setup), the client is using the <code>HADOOP_HOME</code> environment variable. If it is set, the client tries to access <code>$HADOOP_HOME/etc/hadoop</code> (Hadoop 2) and <code>$HADOOP_HOME/conf</code> (Hadoop 1).</li> + <li>Test if <code>YARN_CONF_DIR</code>, <code>HADOOP_CONF_DIR</code> or <code>HADOOP_CONF_PATH</code> are set (in that order). If one of these variables are set, they are used to read the configuration.</li> + <li>If the above strategy fails (this should not be the case in a correct YARN setup), the client is using the <code>HADOOP_HOME</code> environment variable. If it is set, the client tries to access <code>$HADOOP_HOME/etc/hadoop</code> (Hadoop 2) and <code>$HADOOP_HOME/conf</code> (Hadoop 1).</li> </ul> <p>When starting a new Flink YARN session, the client first checks if the requested resources (containers and memory) are available. After that, it uploads a jar that contains Flink and the configuration to HDFS (step 1).</p> <p>The next step of the client is to request (step 2) a YARN container to start the <em>ApplicationMaster</em> (step 3). Since the client registered the configuration and jar-file as a resource for the container, the NodeManager of YARN running on that particular machine will take care of preparing the container (e.g. downloading the files). Once that has finished, the <em>ApplicationMaster</em> (AM) is started.</p> -<p>The <em>JobManager</em> and AM are running in the same container. Once they successfully started, the AM knows the address of the JobManager (its own host). It is generating a new Flink configuration file for the TaskManagers (so that they can connect to the JobManager). The file is also uploaded to HDFS. Additionally, the <em>AM</em> container is also serving Flink's web interface.</p> +<p>The <em>JobManager</em> and AM are running in the same container. Once they successfully started, the AM knows the address of the JobManager (its own host). It is generating a new Flink configuration file for the TaskManagers (so that they can connect to the JobManager). The file is also uploaded to HDFS. Additionally, the <em>AM</em> container is also serving Flinkâs web interface. The ports Flink is using for its services are the standard ports configured by the user + the application id as an offset. This allows users to execute multiple Flink YARN sessions in parallel.</p> -<p>After that, the AM starts allocating the containers for Flink's TaskManagers, which will download the jar file and the modified configuration from the HDFS. Once these steps are completed, Flink is set up and ready to accept Jobs.</p> +<p>After that, the AM starts allocating the containers for Flinkâs TaskManagers, which will download the jar file and the modified configuration from the HDFS. Once these steps are completed, Flink is set up and ready to accept Jobs.</p> + + <!-- Disqus Area --> <div style="padding-top:30px" id="disqus_thread"></div> <script type="text/javascript"> @@ -256,14 +344,12 @@ Action <span class="s2">"run"< })(); </script> <noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript> - <a href="http://disqus.com"; class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a> - </div> </div> <div class="footer"> - <p><hr class="divider"></p> + <hr class="divider" /> <p><small>Apache Flink is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is @@ -274,9 +360,10 @@ incubation status is not necessarily a r stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</small></p> -<p><a href="http://incubator.apache.org/";><img src="/img/apache-incubator-logo.png" alt="Incubator Logo"></a></p> +<p><a href="http://incubator.apache.org/";><img src="/img/apache-incubator-logo.png" alt="Incubator Logo" /></a></p> -<p class="text-center"><a href="/privacy-policy.html">Privacy Policy<a></p> +<p class="text-center"><a href="privacy-policy.html">Privacy Policy<a> +</a></a></p> </div> </div> Modified: flink/site/docs/0.8/building.html URL: http://svn.apache.org/viewvc/flink/site/docs/0.8/building.html?rev=1657551&r1=1657550&r2=1657551&view=diff ============================================================================== --- flink/site/docs/0.8/building.html (original) +++ flink/site/docs/0.8/building.html Thu Feb 5 12:21:38 2015 @@ -42,7 +42,7 @@ under the License. <div class="container"> <div class="row"> <div class="col-md-1 af-mobile-nav-bar"> - <a href="index.html" title="Home"> + <a href="http://flink.apache.org"; title="Home"> <img class="hidden-xs hidden-sm img-responsive" src="img/logo.png" alt="Apache Flink Logo"> </a> @@ -157,7 +157,7 @@ under the License. <p>To clone from git, enter:</p> -<div class="highlight"><pre><code class="language-bash">git clone https://github.com/apache/incubator-flink</code></pre></div> +<div class="highlight"><pre><code class="language-bash">git clone https://github.com/apache/flink</code></pre></div> <p>The simplest way of building Flink is by running:</p> Modified: flink/site/docs/0.8/cli.html URL: http://svn.apache.org/viewvc/flink/site/docs/0.8/cli.html?rev=1657551&r1=1657550&r2=1657551&view=diff ============================================================================== --- flink/site/docs/0.8/cli.html (original) +++ flink/site/docs/0.8/cli.html Thu Feb 5 12:21:38 2015 @@ -42,7 +42,7 @@ under the License. <div class="container"> <div class="row"> <div class="col-md-1 af-mobile-nav-bar"> - <a href="index.html" title="Home"> + <a href="http://flink.apache.org"; title="Home"> <img class="hidden-xs hidden-sm img-responsive" src="img/logo.png" alt="Apache Flink Logo"> </a> Modified: flink/site/docs/0.8/cluster_execution.html URL: http://svn.apache.org/viewvc/flink/site/docs/0.8/cluster_execution.html?rev=1657551&r1=1657550&r2=1657551&view=diff ============================================================================== --- flink/site/docs/0.8/cluster_execution.html (original) +++ flink/site/docs/0.8/cluster_execution.html Thu Feb 5 12:21:38 2015 @@ -42,7 +42,7 @@ under the License. <div class="container"> <div class="row"> <div class="col-md-1 af-mobile-nav-bar"> - <a href="index.html" title="Home"> + <a href="http://flink.apache.org"; title="Home"> <img class="hidden-xs hidden-sm img-responsive" src="img/logo.png" alt="Apache Flink Logo"> </a> Modified: flink/site/docs/0.8/cluster_setup.html URL: http://svn.apache.org/viewvc/flink/site/docs/0.8/cluster_setup.html?rev=1657551&r1=1657550&r2=1657551&view=diff ============================================================================== --- flink/site/docs/0.8/cluster_setup.html (original) +++ flink/site/docs/0.8/cluster_setup.html Thu Feb 5 12:21:38 2015 @@ -42,7 +42,7 @@ under the License. <div class="container"> <div class="row"> <div class="col-md-1 af-mobile-nav-bar"> - <a href="index.html" title="Home"> + <a href="http://flink.apache.org"; title="Home"> <img class="hidden-xs hidden-sm img-responsive" src="img/logo.png" alt="Apache Flink Logo"> </a> Modified: flink/site/docs/0.8/coding_guidelines.html URL: http://svn.apache.org/viewvc/flink/site/docs/0.8/coding_guidelines.html?rev=1657551&r1=1657550&r2=1657551&view=diff ============================================================================== --- flink/site/docs/0.8/coding_guidelines.html (original) +++ flink/site/docs/0.8/coding_guidelines.html Thu Feb 5 12:21:38 2015 @@ -42,7 +42,7 @@ under the License. <div class="container"> <div class="row"> <div class="col-md-1 af-mobile-nav-bar"> - <a href="index.html" title="Home"> + <a href="http://flink.apache.org"; title="Home"> <img class="hidden-xs hidden-sm img-responsive" src="img/logo.png" alt="Apache Flink Logo"> </a> Modified: flink/site/docs/0.8/config.html URL: http://svn.apache.org/viewvc/flink/site/docs/0.8/config.html?rev=1657551&r1=1657550&r2=1657551&view=diff ============================================================================== --- flink/site/docs/0.8/config.html (original) +++ flink/site/docs/0.8/config.html Thu Feb 5 12:21:38 2015 @@ -42,7 +42,7 @@ under the License. <div class="container"> <div class="row"> <div class="col-md-1 af-mobile-nav-bar"> - <a href="index.html" title="Home"> + <a href="http://flink.apache.org"; title="Home"> <img class="hidden-xs hidden-sm img-responsive" src="img/logo.png" alt="Apache Flink Logo"> </a> Modified: flink/site/docs/0.8/dataset_transformations.html URL: http://svn.apache.org/viewvc/flink/site/docs/0.8/dataset_transformations.html?rev=1657551&r1=1657550&r2=1657551&view=diff ============================================================================== --- flink/site/docs/0.8/dataset_transformations.html (original) +++ flink/site/docs/0.8/dataset_transformations.html Thu Feb 5 12:21:38 2015 @@ -42,7 +42,7 @@ under the License. <div class="container"> <div class="row"> <div class="col-md-1 af-mobile-nav-bar"> - <a href="index.html" title="Home"> + <a href="http://flink.apache.org"; title="Home"> <img class="hidden-xs hidden-sm img-responsive" src="img/logo.png" alt="Apache Flink Logo"> </a> @@ -572,7 +572,7 @@ implement (override) the <code>combine() <span class="kd">public</span> <span class="kt">void</span> <span class="nf">reduce</span><span class="o">(</span><span class="n">Iterable</span><span class="o"><</span><span class="n">Tuple3</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">,</span> <span class="n">Double</span><span class="o">>></span> <span class="n">in</span><span class="o">,</span> <span class="n">Collector</span><span class="o"><</span><span class="n">Tuple3</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">,</span> <span class="n">Double</span><span class="o">>></span> <span class="n">out</span><span class="o">)</span> <span class="o">{</span> - <span class="n">String</span> <span class="n">key</span> <span class="o">=</span> <span class="kc">null</span> + <span class="n">String</span> <span class="n">key</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span> <span class="kt">int</span> <span class="n">intSum</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="kt">double</span> <span class="n">doubleSum</span> <span class="o">=</span> <span class="mf">0.0</span><span class="o">;</span> @@ -587,7 +587,7 @@ implement (override) the <code>combine() <span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">combine</span><span class="o">(</span><span class="n">Iterable</span><span class="o"><</span><span class="n">Tuple3</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">,</span> <span class="n">Double</span><span class="o">>></span> <span class="n">in</span><span class="o">,</span> - <span class="n">Collector</span><span class="o"><</span><span class="n">Tuple3</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">,</span> <span class="n">Double</span><span class="o">>></span> <span class="n">out</span><span class="o">))</span> <span class="o">{</span> + <span class="n">Collector</span><span class="o"><</span><span class="n">Tuple3</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">,</span> <span class="n">Double</span><span class="o">>></span> <span class="n">out</span><span class="o">)</span> <span class="o">{</span> <span class="c1">// in some cases combine() calls can simply be forwarded to reduce().</span> <span class="k">this</span><span class="o">.</span><span class="na">reduce</span><span class="o">(</span><span class="n">in</span><span class="o">,</span> <span class="n">out</span><span class="o">);</span> <span class="o">}</span> Modified: flink/site/docs/0.8/example_connectors.html URL: http://svn.apache.org/viewvc/flink/site/docs/0.8/example_connectors.html?rev=1657551&r1=1657550&r2=1657551&view=diff ============================================================================== --- flink/site/docs/0.8/example_connectors.html (original) +++ flink/site/docs/0.8/example_connectors.html Thu Feb 5 12:21:38 2015 @@ -42,7 +42,7 @@ under the License. <div class="container"> <div class="row"> <div class="col-md-1 af-mobile-nav-bar"> - <a href="index.html" title="Home"> + <a href="http://flink.apache.org"; title="Home"> <img class="hidden-xs hidden-sm img-responsive" src="img/logo.png" alt="Apache Flink Logo"> </a> @@ -213,6 +213,8 @@ For example for tachyon support, there m <span class="nt"><value></span>tachyon.hadoop.TFS<span class="nt"></value></span> <span class="nt"></property></span></code></pre></div> +<p>Also, the required classes for using the file system need to be placed in the <code>lib/</code> folder of the Flink installation (on all machines running Flink). If putting the files into the directory is not possible, Flink is also respecting the <code>HADOOP_CLASSPATH</code> environment variable to add Hadoop jar files to the classpath.</p> + <h2 id="connecting-to-other-systems-using-input--output-format-wrappers-for-hadoop">Connecting to other systems using Input / Output Format wrappers for Hadoop</h2> <p>Apache Flink allows users to access many different systems as data sources or sinks. Modified: flink/site/docs/0.8/examples.html URL: http://svn.apache.org/viewvc/flink/site/docs/0.8/examples.html?rev=1657551&r1=1657550&r2=1657551&view=diff ============================================================================== --- flink/site/docs/0.8/examples.html (original) +++ flink/site/docs/0.8/examples.html Thu Feb 5 12:21:38 2015 @@ -42,7 +42,7 @@ under the License. <div class="container"> <div class="row"> <div class="col-md-1 af-mobile-nav-bar"> - <a href="index.html" title="Home"> + <a href="http://flink.apache.org"; title="Home"> <img class="hidden-xs hidden-sm img-responsive" src="img/logo.png" alt="Apache Flink Logo"> </a> @@ -203,7 +203,7 @@ or <strong>flink-scala-examples</strong> <span class="o">}</span> <span class="o">}</span></code></pre></div> - <p>The <a href="https://github.com/apache/incubator-flink/blob/master//flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/WordCount.java";>WordCount example</a> implements the above described algorithm with input parameters: <code><text input path>, <output path></code>. As test data, any text file will do.</p> + <p>The <a href="https://github.com/apache/flink/blob/master//flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/WordCount.java";>WordCount example</a> implements the above described algorithm with input parameters: <code><text input path>, <output path></code>. As test data, any text file will do.</p> </div> <div data-lang="scala"> @@ -220,7 +220,7 @@ or <strong>flink-scala-examples</strong> <span class="n">counts</span><span class="o">.</span><span class="n">writeAsCsv</span><span class="o">(</span><span class="n">outputPath</span><span class="o">,</span> <span class="s">"\n"</span><span class="o">,</span> <span class="s">" "</span><span class="o">)</span></code></pre></div> - <p>The <a href="https://github.com/apache/incubator-flink/blob/master//flink-examples/flink-scala-examples/src/main/scala/org/apache/flink/examples/scala/wordcount/WordCount.scala";>WordCount example</a> implements the above described algorithm with input parameters: <code><text input path>, <output path></code>. As test data, any text file will do.</p> + <p>The <a href="https://github.com/apache/flink/blob/master//flink-examples/flink-scala-examples/src/main/scala/org/apache/flink/examples/scala/wordcount/WordCount.scala";>WordCount example</a> implements the above described algorithm with input parameters: <code><text input path>, <output path></code>. As test data, any text file will do.</p> </div> </div> @@ -305,7 +305,7 @@ or <strong>flink-scala-examples</strong> <span class="o">}</span> <span class="o">}</span></code></pre></div> - <p>The <a href="https://github.com/apache/incubator-flink/blob/master//flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/graph/PageRankBasic.java";>PageRank program</a> implements the above example. + <p>The <a href="https://github.com/apache/flink/blob/master//flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/graph/PageRankBasic.java";>PageRank program</a> implements the above example. It requires the following parameters to run: <code><pages input path>, <links input path>, <output path>, <num pages>, <num iterations></code>.</p> </div> @@ -370,7 +370,7 @@ It requires the following parameters to <span class="c1">// emit result</span> <span class="n">result</span><span class="o">.</span><span class="n">writeAsCsv</span><span class="o">(</span><span class="n">outputPath</span><span class="o">,</span> <span class="s">"\n"</span><span class="o">,</span> <span class="s">" "</span><span class="o">)</span></code></pre></div> - <p>he <a href="https://github.com/apache/incubator-flink/blob/master//flink-examples/flink-scala-examples/src/main/scala/org/apache/flink/examples/scala/graph/PageRankBasic.scala";>PageRank program</a> implements the above example. + <p>he <a href="https://github.com/apache/flink/blob/master//flink-examples/flink-scala-examples/src/main/scala/org/apache/flink/examples/scala/graph/PageRankBasic.scala";>PageRank program</a> implements the above example. It requires the following parameters to run: <code><pages input path>, <links input path>, <output path>, <num pages>, <num iterations></code>.</p> </div> </div> @@ -464,7 +464,7 @@ It requires the following parameters to <span class="o">}</span> <span class="o">}</span></code></pre></div> - <p>The <a href="https://github.com/apache/incubator-flink/blob/master//flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/graph/ConnectedComponents.java";>ConnectedComponents program</a> implements the above example. It requires the following parameters to run: <code><vertex input path>, <edge input path>, <output path> <max num iterations></code>.</p> + <p>The <a href="https://github.com/apache/flink/blob/master//flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/graph/ConnectedComponents.java";>ConnectedComponents program</a> implements the above example. It requires the following parameters to run: <code><vertex input path>, <edge input path>, <output path> <max num iterations></code>.</p> </div> <div data-lang="scala"> @@ -504,7 +504,7 @@ It requires the following parameters to <span class="n">verticesWithComponents</span><span class="o">.</span><span class="n">writeAsCsv</span><span class="o">(</span><span class="n">outputPath</span><span class="o">,</span> <span class="s">"\n"</span><span class="o">,</span> <span class="s">" "</span><span class="o">)</span></code></pre></div> - <p>The <a href="https://github.com/apache/incubator-flink/blob/master//flink-examples/flink-scala-examples/src/main/scala/org/apache/flink/examples/scala/graph/ConnectedComponents.scala";>ConnectedComponents program</a> implements the above example. It requires the following parameters to run: <code><vertex input path>, <edge input path>, <output path> <max num iterations></code>.</p> + <p>The <a href="https://github.com/apache/flink/blob/master//flink-examples/flink-scala-examples/src/main/scala/org/apache/flink/examples/scala/graph/ConnectedComponents.scala";>ConnectedComponents program</a> implements the above example. It requires the following parameters to run: <code><vertex input path>, <edge input path>, <output path> <max num iterations></code>.</p> </div> </div> @@ -576,13 +576,13 @@ It requires the following parameters to <span class="c1">// emit result</span> <span class="n">priceSums</span><span class="o">.</span><span class="na">writeAsCsv</span><span class="o">(</span><span class="n">outputPath</span><span class="o">);</span></code></pre></div> - <p>The <a href="https://github.com/apache/incubator-flink/blob/master//flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/relational/RelationalQuery.java";>Relational Query program</a> implements the above query. It requires the following parameters to run: <code><orders input path>, <lineitem input path>, <output path></code>.</p> + <p>The <a href="https://github.com/apache/flink/blob/master//flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/relational/RelationalQuery.java";>Relational Query program</a> implements the above query. It requires the following parameters to run: <code><orders input path>, <lineitem input path>, <output path></code>.</p> </div> <div data-lang="scala"> <p>Coming soonâ¦</p> - <p>The <a href="https://github.com/apache/incubator-flink/blob/master//flink-examples/flink-scala-examples/src/main/scala/org/apache/flink/examples/scala/relational/RelationalQuery.scala";>Relational Query program</a> implements the above query. It requires the following parameters to run: <code><orders input path>, <lineitem input path>, <output path></code>.</p> + <p>The <a href="https://github.com/apache/flink/blob/master//flink-examples/flink-scala-examples/src/main/scala/org/apache/flink/examples/scala/relational/RelationalQuery.scala";>Relational Query program</a> implements the above query. It requires the following parameters to run: <code><orders input path>, <lineitem input path>, <output path></code>.</p> </div> </div>
