This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 904c86a34cb Publishing website 2023/08/10 22:38:36 at commit 3c5c728
904c86a34cb is described below
commit 904c86a34cb9cc53f4b495cb97b01ad207c6f79c
Author: jenkins <bui...@apache.org>
AuthorDate: Thu Aug 10 22:38:36 2023 +0000
Publishing website 2023/08/10 22:38:36 at commit 3c5c728
---
website/generated-content/documentation/index.xml | 91 ++++++++++++++++++++-
.../documentation/programming-guide/index.html | 8 +-
.../generated-content/images/transform_service.png | Bin 0 -> 41817 bytes
website/generated-content/sitemap.xml | 2 +-
4 files changed, 97 insertions(+), 4 deletions(-)
diff --git a/website/generated-content/documentation/index.xml
b/website/generated-content/documentation/index.xml
index 5abf4d553e7..95ee18835da 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -14981,7 +14981,96 @@ implementations. This information can be accessed by
using the conventional
<span class="k">def</span> <span
class="nf">infer_output_type</span><span class="p">(</span><span
class="bp">self</span><span class="p">,</span> <span
class="n">input_type</span><span class="p">):</span>
<span class="k">return</span> <span
class="n">input_type</span></code></pre></div>
</div>
-</div></description></item><item><title>Documentation: BigQuery ML
integration</title><link>/documentation/patterns/bqml/</link><pubDate>Mon, 01
Jan 0001 00:00:00
+0000</pubDate><guid>/documentation/patterns/bqml/</guid><description>
+</div>
+<h2 id="transform-service">15 Transform Service</h2>
+<p>Starting version 2.49.0, Beam introduced a <a
href="https://docs.docker.com/compose/";>Docker Compose</a> based service
named <em>Transform Service</em>. The Transform Service allows Beam
portable
+pipelines to perform expansion of supported transforms using Docker.</p>
+<p>The basic architecture of the Transform Service is given below.</p>
+<p><img src="/images/transform_service.png" alt="Diagram of the
transform service architecture"></p>
+<p>The Transform Service can be useful in many contexts. We have identified
two primary use-cases below. Note that to use the transform service, Docker
(and Docker Compose) needs to be available in the machine where the service
will be started at.</p>
+<ul>
+<li>Perform expansion of cross-language transforms without installing other
language runtimes.</li>
+</ul>
+<p>Transforms Service allows multi-language pipelines to use/expand
cross-language transforms implemented in other SDKs without installing runtimes
for implementation languages of such SDKs.
+For example, with the Transform Service, a Beam Python pipeline can use Java
GCP I/O transforms and Java Kafka I/O transforms without installing a Java
runtime locally.</p>
+<ul>
+<li>Upgrade transforms without upgrading the Beam version.</li>
+</ul>
+<p>The transform service can be used to upgrade individual transforms used
by Beam pipelines to a new Beam version without upgrading the Beam version used
by the pipeline.
+This feature is currently in development. Please see the <a
href="https://github.com/apache/beam/issues/27943";>tracking issue</a> for
more details.</p>
+<h3 id="transform-service-usage">15.1 Using the the Transform
Service</h3>
+<p>Beam SDKs may automatically startup a transform service to perform
expansion when using cross-language transforms. More specifically,</p>
+<ul>
+<li>
+<p>Java <a
href="https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/extensions/python/PythonExternalTransform.java";>PythonExternalTransform
API</a> will automatically
+startup the Transform Service for you if a Python runtime is not available
locally but Docker is available.</p>
+</li>
+<li>
+<p>Beam Python multi-language wrappers may automatically startup a
Transform Service for you when using Java transforms, if a Java language
runtime is not available locally but Docker is available.</p>
+</li>
+</ul>
+<p>Additionally, if needed, a Transform Service instance can be manually
started using utilities provided with Beam SDKs.</p>
+<div class='language-java snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code
class="language-java" data-lang="java"><span class="n">java</span>
<span class="o">-</span><span class="n">jar</span> <span
class="n">beam</span><span class="o">-</span><span
class="n">sdks</span><span class="o">-</span><span
class="n">java</span><span class="o">-</span><span
class="n">transform</span><span class="o">-</span><span
class="n">service</span><s [...]
+</div>
+</div>
+<div class='language-py snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-py"
data-lang="py"><span class="n">python</span> <span
class="o">-</span><span class="n">m</span> <span
class="n">apache_beam</span><span class="o">.</span><span
class="n">utils</span><span class="o">.</span><span
class="n">transform_service_launcher</span> <span
class="o">--</span><span class="n">port</span> <span
class="o">&lt;</span><span c [...]
+</div>
+</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">feature</span> <span class="nx">is</span> <span
class="nx">currently</span> <span class="nx">in</span> <span
class="nx">development</span><span
class="p">.</span></code></pre></div>
+</div>
+</div>
+<p>To stop the transform service use the following commands.</p>
+<div class='language-java snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code
class="language-java" data-lang="java"><span class="n">java</span>
<span class="o">-</span><span class="n">jar</span> <span
class="n">beam</span><span class="o">-</span><span
class="n">sdks</span><span class="o">-</span><span
class="n">java</span><span class="o">-</span><span
class="n">transform</span><span class="o">-</span><span
class="n">service</span><s [...]
+</div>
+</div>
+<div class='language-py snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-py"
data-lang="py"><span class="n">python</span> <span
class="o">-</span><span class="n">m</span> <span
class="n">apache_beam</span><span class="o">.</span><span
class="n">utils</span><span class="o">.</span><span
class="n">transform_service_launcher</span> <span
class="o">--</span><span class="n">port</span> <span
class="o">&lt;</span><span c [...]
+</div>
+</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">feature</span> <span class="nx">is</span> <span
class="nx">currently</span> <span class="nx">in</span> <span
class="nx">development</span><span
class="p">.</span></code></pre></div>
+</div>
+</div>
+<h3 id="transform-service-included-transforms">15.2 Portable Transforms
included in the Transform Service</h3>
+<p>Transforms service includes a number of portable transforms implemented
in Beam Java and Python SDKs.</p>
+<p>Some of the transforms currently included in the Trasnform Service are
given below.</p>
+<ul>
+<li>
+<p>Java transforms - GCP I/O connectors, Kafka I/O connector, JDBC I/O
connector.</p>
+</li>
+<li>
+<p>Python transforms - all portable transforms implemented within Beam
Python SDK, for example, RunInference and Dataframe transforms.</p>
+</li>
+</ul>
+<p>For a more detailed list of available transforms, please see <a
href="https://cwiki.apache.org/confluence/display/BEAM/Transform+Service";>here</a>.</p></description></item><item><title>Documentation:
BigQuery ML
integration</title><link>/documentation/patterns/bqml/</link><pubDate>Mon, 01
Jan 0001 00:00:00
+0000</pubDate><guid>/documentation/patterns/bqml/</guid><description>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
diff --git
a/website/generated-content/documentation/programming-guide/index.html
b/website/generated-content/documentation/programming-guide/index.html
index 73a88798ec7..0662fe78d72 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -22,7 +22,7 @@
function addPlaceholder(){$('input:text').attr('placeholder',"What are you
looking for?");}
function endSearch(){var
search=document.querySelector(".searchBar");search.classList.add("disappear");var
icons=document.querySelector("#iconsBar");icons.classList.remove("disappear");}
function blockScroll(){$("body").toggleClass("fixedPosition");}
-function openMenu(){addPlaceholder();blockScroll();}</script><div
class="clearfix container-main-content"><div class="section-nav closed"
data-offset-top=90 data-offset-bottom=500><span class="section-nav-back
glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list
data-section-nav><li><span
class=section-nav-list-main-title>Documentation</span></li><li><a
href=/documentation>Using the Documentation</a></li><li
class=section-nav-item--collapsible><span class=section-nav-lis [...]
+function openMenu(){addPlaceholder();blockScroll();}</script><div
class="clearfix container-main-content"><div class="section-nav closed"
data-offset-top=90 data-offset-bottom=500><span class="section-nav-back
glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list
data-section-nav><li><span
class=section-nav-list-main-title>Documentation</span></li><li><a
href=/documentation>Using the Documentation</a></li><li
class=section-nav-item--collapsible><span class=section-nav-lis [...]
Beam SDKs to create data processing pipelines. It provides guidance for using
the Beam SDK classes to build and test your pipeline. The programming guide is
not intended as an exhaustive reference, but as a language-agnostic, high-level
@@ -5646,7 +5646,11 @@ implementations. This information can be accessed by
using the conventional
<span class=o>...</span>
<span class=k>def</span> <span class=nf>infer_output_type</span><span
class=p>(</span><span class=bp>self</span><span class=p>,</span> <span
class=n>input_type</span><span class=p>):</span>
- <span class=k>return</span> <span
class=n>input_type</span></code></pre></div></div></div><div class=feedback><p
class=update>Last updated on 2023/08/10</p><h3>Have you found everything you
were looking for?</h3><p class=description>Was it all useful and clear? Is
there anything that you would like to change? Let us know!</p><button
class=load-button><a href="mailto:d...@beam.apache.org?subject=Beam Website
Feedback">SEND FEEDBACK</a></button></div></div></div><footer class=footer><di
[...]
+ <span class=k>return</span> <span
class=n>input_type</span></code></pre></div></div></div><h2
id=transform-service>15 Transform Service</h2><p>Starting version 2.49.0, Beam
introduced a <a href=https://docs.docker.com/compose/>Docker Compose</a> based
service named <em>Transform Service</em>. The Transform Service allows Beam
portable
+pipelines to perform expansion of supported transforms using Docker.</p><p>The
basic architecture of the Transform Service is given below.</p><p><img
src=/images/transform_service.png alt="Diagram of the transform service
architecture"></p><p>The Transform Service can be useful in many contexts. We
have identified two primary use-cases below. Note that to use the transform
service, Docker (and Docker Compose) needs to be available in the machine where
the service will be started at.</p>< [...]
+For example, with the Transform Service, a Beam Python pipeline can use Java
GCP I/O transforms and Java Kafka I/O transforms without installing a Java
runtime locally.</p><ul><li>Upgrade transforms without upgrading the Beam
version.</li></ul><p>The transform service can be used to upgrade individual
transforms used by Beam pipelines to a new Beam version without upgrading the
Beam version used by the pipeline.
+This feature is currently in development. Please see the <a
href=https://github.com/apache/beam/issues/27943>tracking issue</a> for more
details.</p><h3 id=transform-service-usage>15.1 Using the the Transform
Service</h3><p>Beam SDKs may automatically startup a transform service to
perform expansion when using cross-language transforms. More
specifically,</p><ul><li><p>Java <a
href=https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/ex
[...]
+startup the Transform Service for you if a Python runtime is not available
locally but Docker is available.</p></li><li><p>Beam Python multi-language
wrappers may automatically startup a Transform Service for you when using Java
transforms, if a Java language runtime is not available locally but Docker is
available.</p></li></ul><p>Additionally, if needed, a Transform Service
instance can be manually started using utilities provided with Beam
SDKs.</p><div class="language-java snippet">< [...]
<a href=https://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam
logo, and the Apache feather logo are either registered trademarks or
trademarks of The Apache Software Foundation. All other products or name brands
are trademarks of their respective holders, including The Apache Software
Foundation.</div></div><div class="footer__cols__col
footer__cols__col__logos"><div class=footer__cols__col--group><div
class=footer__cols__col__logo><a href=https://github.com/apache/beam><im [...]
\ No newline at end of file
diff --git a/website/generated-content/images/transform_service.png
b/website/generated-content/images/transform_service.png
new file mode 100644
index 00000000000..f3763bac64e
Binary files /dev/null and
b/website/generated-content/images/transform_service.png differ
diff --git a/website/generated-content/sitemap.xml
b/website/generated-content/sitemap.xml
index 19c51a904f3..4c798142171 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.49.0/</loc><lastmod>2023-08-10T18:03:34-04:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2023-08-10T18:03:34-04:00</lastmod></url><url><loc>/blog/</loc><lastmod>2023-08-10T18:03:34-04:00</lastmod></url><url><loc>/categories/</loc><lastmod>2023-08-10T18:03:34-04:00</lastmod></url><url><loc>/catego
[...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.49.0/</loc><lastmod>2023-08-10T15:24:25-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2023-08-10T15:24:25-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2023-08-10T15:24:25-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2023-08-10T15:24:25-07:00</lastmod></url><url><loc>/catego
[...]
\ No newline at end of file
