This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 7cbd122113c98ce7438388e9f7d2489b03a20e1e Author: Mergebot <[email protected]> AuthorDate: Mon Jul 23 07:58:35 2018 +0000 Prepare repository for deployment. --- content/contribute/portability/index.html | 32 ++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/content/contribute/portability/index.html b/content/contribute/portability/index.html index 30b3958..f0f247d 100644 --- a/content/contribute/portability/index.html +++ b/content/contribute/portability/index.html @@ -226,7 +226,10 @@ limitations under the License. <li><a href="#development" id="markdown-toc-development">Development</a> <ul> <li><a href="#roadmap" id="markdown-toc-roadmap">Roadmap</a></li> <li><a href="#issues" id="markdown-toc-issues">Issues</a></li> - <li><a href="#status" id="markdown-toc-status">Status</a></li> + <li><a href="#status" id="markdown-toc-status">Status</a> <ul> + <li><a href="#running-python-wordcount-on-flink" id="markdown-toc-running-python-wordcount-on-flink">Running Python wordcount on Flink</a></li> + </ul> + </li> </ul> </li> </ul> @@ -364,7 +367,7 @@ portably.</p> for remaining execution-side features, so that any pipeline from any SDK can run portably on any runner. These features include side inputs -[<a href="https://issues.apache.org/jira/browse/BEAM-2863";>BEAM-2863</a>], User +[<a href="https://issues.apache.org/jira/browse/BEAM-2863";>BEAM-2863</a>], User state [<a href="https://issues.apache.org/jira/browse/BEAM-2862";>BEAM-2862</a>], User timers [<a href="https://issues.apache.org/jira/browse/BEAM-2925";>BEAM-2925</a>], Splittable DoFn @@ -403,10 +406,33 @@ their respective components.</p> <h3 id="status">Status</h3> -<p>MVP in progress. See the +<p>MVP in progress (near completion for Flink runner). See the <a href="https://docs.google.com/spreadsheets/d/1KDa_FGn1ShjomGd-UUDOhuh2q73de2tPz6BqHpzqvNI/edit?usp=sharing";>Portability support table</a> for details.</p> +<h4 id="running-python-wordcount-on-flink">Running Python wordcount on Flink</h4> + +<p>The Flink runner is currently the only runner to support portable pipeline execution. +To run a basic Python wordcount (in batch mode) with embedded Flink:</p> + +<ol> + <li>Run once to build the SDK harness container: <code class="highlighter-rouge">./gradlew -p sdks/python/container +docker</code></li> + <li>Start the Flink portable JobService endpoint: <code class="highlighter-rouge">./gradlew :beam-runners-flink_2.11-job-server:runShadow</code></li> + <li>Submit the wordcount pipeline to above endpoint: <code class="highlighter-rouge">./gradlew :beam-sdks-python:portableWordCount</code></li> +</ol> + +<p>To run on a separate <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html";>Flink cluster</a>:</p> + +<ol> + <li>Start local Flink cluster</li> + <li>Create shaded JobService jar: <code class="highlighter-rouge">./gradlew :beam-runners-flink_2.11-job-server:installShadowDist</code></li> + <li>Start JobService with Flink web service endpoint: <code class="highlighter-rouge">java -jar ./runners/flink/job-server/build/install/beam-runners-flink_2.11-job-server-shadow/lib/beam-runners-flink_2.11-job-server-*.jar--job-host=localhost:8099" "--artifacts-dir=/tmp/flink-artifacts" "--flink-master-url=localhost:8081"</code></li> + <li>Submit the pipeline.</li> +</ol> + +<p>Note: A subset of the functionality is also supported in streaming mode; use <code class="highlighter-rouge">--streaming</code> in the command line to enable it.</p> + </div> </div> <!--
