This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c68ab5b Publishing website 2019/11/09 22:39:14 at commit ede14d4
c68ab5b is described below
commit c68ab5bc1864d9902a24aef76e903cb7745bfe2a
Author: jenkins <[email protected]>
AuthorDate: Sat Nov 9 22:39:15 2019 +0000
Publishing website 2019/11/09 22:39:14 at commit ede14d4
---
.../roadmap/connectors-multi-sdk/index.html | 117 ++++++++++++++++++---
1 file changed, 104 insertions(+), 13 deletions(-)
diff --git a/website/generated-content/roadmap/connectors-multi-sdk/index.html
b/website/generated-content/roadmap/connectors-multi-sdk/index.html
index 70eac06..4e3df3f 100644
--- a/website/generated-content/roadmap/connectors-multi-sdk/index.html
+++ b/website/generated-content/roadmap/connectors-multi-sdk/index.html
@@ -226,6 +226,20 @@
+<ul class="nav">
+ <li><a href="#completed-and-ongoing-efforts">Completed and Ongoing
Efforts</a>
+ <ul>
+ <li><a
href="#cross-language-transforms-api-and-expansion-service">Cross-language
transforms API and expansion service</a></li>
+ <li><a href="#support-for-flink-runner">Support for Flink runner</a></li>
+ <li><a href="#support-for-dataflow-runner">Support for Dataflow
runner</a></li>
+ <li><a href="#support-for-direct-runner">Support for Direct
runner</a></li>
+ <li><a href="#connectortransform-support">Connector/transform
support</a></li>
+ <li><a href="#portable-beam-schema">Portable Beam schema</a></li>
+ <li><a href="#integrationperformance-testing">Integration/Performance
testing</a></li>
+ <li><a href="#documentation">Documentation</a></li>
+ </ul>
+ </li>
+</ul>
</nav>
@@ -255,23 +269,100 @@ efforts. See <a
href="https://beam.apache.org/roadmap/portability/";>Beam portabi
<h1 id="cross-language-transforms">Cross-language transforms</h1>
-<p>As an added benefit of Beam portability efforts, in the future, we’ll be
-able to utilize Beam transforms across languages. This has many benefits.
-For example.</p>
+<p><em>Last updated on November 2019.</em></p>
+
+<p>As an added benefit of Beam portability effort, we are able to utilize Beam
transforms across SDKs. This has many benefits.</p>
+
+<ul>
+ <li>Connector sharing across SDKs. For example,
+ <ul>
+ <li>Beam pipelines written using Python and Go SDKs will be able to
utilize the vast selection of connectors that are currently implemented for
Java SDK.</li>
+ <li>Java SDK will be able to utilize connectors for systems that only
offer a Python API.</li>
+ <li>Go SDK, will be able to utilize connectors currently available for
Java and Python SDKs.</li>
+ </ul>
+ </li>
+ <li>Ease of developing and maintaining Beam transforms - in general, with
cross-language transforms, Beam transform authors will be able to implement new
Beam transforms using a
+language of choice and utilize these transforms from other languages reducing
the maintenance and support overheads.</li>
+ <li><a href="https://beam.apache.org/documentation/dsls/sql/overview/";>Beam
SQL</a>, that is currently only available to Java SDK, will become available to
Python and Go SDKs.</li>
+ <li><a href="https://www.tensorflow.org/tfx/transform/get_started";>Beam TFX
transforms</a>, that are currently only available to Beam Python SDK pipelines
will become available to Java and Go SDKs.</li>
+</ul>
+
+<h2 id="completed-and-ongoing-efforts">Completed and Ongoing Efforts</h2>
+
+<p>Many efforts related to cross-language transforms are currently in flux.
Some of the completed and ongoing efforts are given below.</p>
+
+<h3 id="cross-language-transforms-api-and-expansion-service">Cross-language
transforms API and expansion service</h3>
+
+<p>Work related to developing/updating the cross-language transforms API for
Java/Python/Go SDKs and work related to cross-language transform expansion
services.</p>
+
+<ul>
+ <li>Basic API for Java SDK - completed</li>
+ <li>Basic API for Python SDK - completed</li>
+ <li>Basic API for Go SDK - Not started</li>
+ <li>Basic cross-language transform expansion service for Java and Python
SDKs - completed</li>
+ <li>Artifact staging - In progress - <a
href="https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E";>email
thread</a>, <a
href="https://docs.google.com/document/d/1XaiNekAY2sptuQRIXpjGAyaYdSc-wlJ-VKjl04c8N48/edit#heading=h.900gc947qrw8";>doc</a></li>
+</ul>
+
+<h3 id="support-for-flink-runner">Support for Flink runner</h3>
+
+<p>Work related to making cross-language transforms available for Flink
runner.</p>
+
+<ul>
+ <li>Basic support for executing cross-language transforms on portable Flink
runner - completed</li>
+</ul>
+
+<h3 id="support-for-dataflow-runner">Support for Dataflow runner</h3>
+
+<p>Work related to making cross-language transforms available for Dataflow
runner.</p>
+
+<ul>
+ <li>Basic support for executing cross-language transforms on Dataflow runner
+ <ul>
+ <li>This work requires updates to Dataflow service’s job submission and
job execution logic. This is currently being developed at Google.</li>
+ </ul>
+ </li>
+</ul>
+
+<h3 id="support-for-direct-runner">Support for Direct runner</h3>
+
+<p>Work related to making cross-language transforms available on Direct
runner</p>
+
+<ul>
+ <li>Basic support for executing cross-language transforms on portable Direct
runner - Not started</li>
+</ul>
+
+<h3 id="connectortransform-support">Connector/transform support</h3>
+
+<p>Ongoing and planned work related to making existing connectors/transforms
available to other SDKs through the cross-language transforms framework.</p>
+
+<ul>
+ <li>Java KafkIO - In progress - <a
href="https://issues.apache.org/jira/browse/BEAM-7029";>BEAM-7029</a></li>
+ <li>Java PubSubIO - In progress - <a
href="https://issues.apache.org/jira/browse/BEAM-7738";>BEAM-7738</a></li>
+</ul>
+
+<h3 id="portable-beam-schema">Portable Beam schema</h3>
+
+<p>Portable Beam schema support will provide a generalized mechanism for
serializing and transferring data across language boundaries which will be
extremely useful for pipelines that employ cross-language transforms.</p>
+
+<ul>
+ <li>Make row coder a standard coder and implement in python - In progress -
<a href="https://issues.apache.org/jira/browse/BEAM-7886";>BEAM-7886</a></li>
+</ul>
+
+<h3 id="integrationperformance-testing">Integration/Performance testing</h3>
+
+<ul>
+ <li>Add an integration test suite for cross-language transforms on Flink
runner - In progress - <a
href="https://issues.apache.org/jira/browse/BEAM-6683";>BEAM-6683</a></li>
+</ul>
+
+<h3 id="documentation">Documentation</h3>
+
+<p>Work related to adding documenting on cross-language transforms to Beam
Website.</p>
<ul>
- <li>Beam pipelines written using Python and Go SDKs will be able to utilize
-the vast selection of connectors that are currently available for Java
SDK.</li>
- <li>Java SDK will be able to utilize connectors for systems that only offer a
-Python API.</li>
- <li>Go SDK, will be able to utilize connectors currently available for Java
and
-Python SDKs.</li>
- <li>Connector authors will be able to implement new Beam connectors using a
-language of choice and utilize these connectors from other languages reducing
-the maintenance and support efforts.</li>
+ <li>Document cross-language transforms API for Java/Python - Not started</li>
+ <li>Document API for making existing transforms available as cross-language
transforms for Java/Python - Not started</li>
</ul>
-<p>See <a href="https://beam.apache.org/roadmap/portability/";>Beam portability
framework roadmap</a> for more details.</p>
</div>
</div>
