This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit ba5a7574fb7e3ec750b2f2cd2609a5ef6d403509 Author: Thomas Weise <[email protected]> AuthorDate: Fri Jul 20 13:41:09 2018 -0700 Add Flink instructions to portability contrib page. --- src/contribute/portability.md | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/src/contribute/portability.md b/src/contribute/portability.md index cd17a92..cb16a66 100644 --- a/src/contribute/portability.md +++ b/src/contribute/portability.md @@ -87,7 +87,7 @@ design documents are the following: * [Runner API](https://s.apache.org/beam-runner-api). Pipeline representation and discussion on primitive/composite transforms and optimizations. - + * [Job API](https://s.apache.org/beam-job-api). Job submission and management protocol. @@ -134,12 +134,12 @@ portability effort. Work already done is not tracked here either. windowed wordcount [[BEAM-2941](https://issues.apache.org/jira/browse/BEAM-2941)] run portably. - + * **P2 [Feature complete]**: Design and implement portability support for remaining execution-side features, so that any pipeline from any SDK can run portably on any runner. These features include side inputs - [[BEAM-2863](https://issues.apache.org/jira/browse/BEAM-2863)], User + [[BEAM-2863](https://issues.apache.org/jira/browse/BEAM-2863)], User state [[BEAM-2862](https://issues.apache.org/jira/browse/BEAM-2862)], User timers [[BEAM-2925](https://issues.apache.org/jira/browse/BEAM-2925)], Splittable DoFn @@ -148,7 +148,7 @@ portability effort. Work already done is not tracked here either. least to the extent that the mobile gaming examples [[BEAM-2940](https://issues.apache.org/jira/browse/BEAM-2940)] run portably. - + * **P3 [Performance]**: Measure and tune performance of portable pipelines using benchmarks such as Nexmark. Features such as progress reporting @@ -174,6 +174,18 @@ their respective components. ### Status -MVP in progress. See the +MVP complete (for Flink runner). See the [Portability support table](https://docs.google.com/spreadsheets/d/1KDa_FGn1ShjomGd-UUDOhuh2q73de2tPz6BqHpzqvNI/edit?usp=sharing) for details. + +#### Running Python wordcount on Flink + +The Flink runner is currently the only runner to support portable pipeline execution. +To run a basic Python wordcount (in batch mode): + +1. Run once to build the SDK harness container: `./gradlew -p sdks/python/container +docker` +2. Start the Flink portable JobService endpoint: `./gradlew :beam-runners-flink_2.11-job-server:runShadow` +3. Submit the wordcount pipeline to above endpoint: `./gradlew :beam-sdks-python:portableWordCount` + +Note: A subset of the functionality is also supported in streaming mode; use `--streaming` in the command line to enable it.
