This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push: new 904c86a34cb Publishing website 2023/08/10 22:38:36 at commit 3c5c728 904c86a34cb is described below commit 904c86a34cb9cc53f4b495cb97b01ad207c6f79c Author: jenkins <bui...@apache.org> AuthorDate: Thu Aug 10 22:38:36 2023 +0000 Publishing website 2023/08/10 22:38:36 at commit 3c5c728 --- website/generated-content/documentation/index.xml | 91 ++++++++++++++++++++- .../documentation/programming-guide/index.html | 8 +- .../generated-content/images/transform_service.png | Bin 0 -> 41817 bytes website/generated-content/sitemap.xml | 2 +- 4 files changed, 97 insertions(+), 4 deletions(-) diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml index 5abf4d553e7..95ee18835da 100644 --- a/website/generated-content/documentation/index.xml +++ b/website/generated-content/documentation/index.xml @@ -14981,7 +14981,96 @@ implementations. This information can be accessed by using the conventional <span class="k">def</span> <span class="nf">infer_output_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">input_type</span><span class="p">):</span> <span class="k">return</span> <span class="n">input_type</span></code></pre></div> </div> -</div></description></item><item><title>Documentation: BigQuery ML integration</title><link>/documentation/patterns/bqml/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/patterns/bqml/</guid><description> +</div> +<h2 id="transform-service">15 Transform Service</h2> +<p>Starting version 2.49.0, Beam introduced a <a href="https://docs.docker.com/compose/">Docker Compose</a> based service named <em>Transform Service</em>. The Transform Service allows Beam portable +pipelines to perform expansion of supported transforms using Docker.</p> +<p>The basic architecture of the Transform Service is given below.</p> +<p><img src="/images/transform_service.png" alt="Diagram of the transform service architecture"></p> +<p>The Transform Service can be useful in many contexts. We have identified two primary use-cases below. Note that to use the transform service, Docker (and Docker Compose) needs to be available in the machine where the service will be started at.</p> +<ul> +<li>Perform expansion of cross-language transforms without installing other language runtimes.</li> +</ul> +<p>Transforms Service allows multi-language pipelines to use/expand cross-language transforms implemented in other SDKs without installing runtimes for implementation languages of such SDKs. +For example, with the Transform Service, a Beam Python pipeline can use Java GCP I/O transforms and Java Kafka I/O transforms without installing a Java runtime locally.</p> +<ul> +<li>Upgrade transforms without upgrading the Beam version.</li> +</ul> +<p>The transform service can be used to upgrade individual transforms used by Beam pipelines to a new Beam version without upgrading the Beam version used by the pipeline. +This feature is currently in development. Please see the <a href="https://github.com/apache/beam/issues/27943">tracking issue</a> for more details.</p> +<h3 id="transform-service-usage">15.1 Using the the Transform Service</h3> +<p>Beam SDKs may automatically startup a transform service to perform expansion when using cross-language transforms. More specifically,</p> +<ul> +<li> +<p>Java <a href="https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/extensions/python/PythonExternalTransform.java">PythonExternalTransform API</a> will automatically +startup the Transform Service for you if a Python runtime is not available locally but Docker is available.</p> +</li> +<li> +<p>Beam Python multi-language wrappers may automatically startup a Transform Service for you when using Java transforms, if a Java language runtime is not available locally but Docker is available.</p> +</li> +</ul> +<p>Additionally, if needed, a Transform Service instance can be manually started using utilities provided with Beam SDKs.</p> +<div class='language-java snippet'> +<div class="notebook-skip code-snippet"> +<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> +<img src="/images/copy-icon.svg"/> +</a> +<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="n">java</span> <span class="o">-</span><span class="n">jar</span> <span class="n">beam</span><span class="o">-</span><span class="n">sdks</span><span class="o">-</span><span class="n">java</span><span class="o">-</span><span class="n">transform</span><span class="o">-</span><span class="n">service</span><s [...] +</div> +</div> +<div class='language-py snippet'> +<div class="notebook-skip code-snippet"> +<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> +<img src="/images/copy-icon.svg"/> +</a> +<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">apache_beam</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">transform_service_launcher</span> <span class="o">--</span><span class="n">port</span> <span class="o">&lt;</span><span c [...] +</div> +</div> +<div class='language-go snippet'> +<div class="notebook-skip code-snippet"> +<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> +<img src="/images/copy-icon.svg"/> +</a> +<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">feature</span> <span class="nx">is</span> <span class="nx">currently</span> <span class="nx">in</span> <span class="nx">development</span><span class="p">.</span></code></pre></div> +</div> +</div> +<p>To stop the transform service use the following commands.</p> +<div class='language-java snippet'> +<div class="notebook-skip code-snippet"> +<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> +<img src="/images/copy-icon.svg"/> +</a> +<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="n">java</span> <span class="o">-</span><span class="n">jar</span> <span class="n">beam</span><span class="o">-</span><span class="n">sdks</span><span class="o">-</span><span class="n">java</span><span class="o">-</span><span class="n">transform</span><span class="o">-</span><span class="n">service</span><s [...] +</div> +</div> +<div class='language-py snippet'> +<div class="notebook-skip code-snippet"> +<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> +<img src="/images/copy-icon.svg"/> +</a> +<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">apache_beam</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">transform_service_launcher</span> <span class="o">--</span><span class="n">port</span> <span class="o">&lt;</span><span c [...] +</div> +</div> +<div class='language-go snippet'> +<div class="notebook-skip code-snippet"> +<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> +<img src="/images/copy-icon.svg"/> +</a> +<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">feature</span> <span class="nx">is</span> <span class="nx">currently</span> <span class="nx">in</span> <span class="nx">development</span><span class="p">.</span></code></pre></div> +</div> +</div> +<h3 id="transform-service-included-transforms">15.2 Portable Transforms included in the Transform Service</h3> +<p>Transforms service includes a number of portable transforms implemented in Beam Java and Python SDKs.</p> +<p>Some of the transforms currently included in the Trasnform Service are given below.</p> +<ul> +<li> +<p>Java transforms - GCP I/O connectors, Kafka I/O connector, JDBC I/O connector.</p> +</li> +<li> +<p>Python transforms - all portable transforms implemented within Beam Python SDK, for example, RunInference and Dataframe transforms.</p> +</li> +</ul> +<p>For a more detailed list of available transforms, please see <a href="https://cwiki.apache.org/confluence/display/BEAM/Transform+Service">here</a>.</p></description></item><item><title>Documentation: BigQuery ML integration</title><link>/documentation/patterns/bqml/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/patterns/bqml/</guid><description> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html index 73a88798ec7..0662fe78d72 100644 --- a/website/generated-content/documentation/programming-guide/index.html +++ b/website/generated-content/documentation/programming-guide/index.html @@ -22,7 +22,7 @@ function addPlaceholder(){$('input:text').attr('placeholder',"What are you looking for?");} function endSearch(){var search=document.querySelector(".searchBar");search.classList.add("disappear");var icons=document.querySelector("#iconsBar");icons.classList.remove("disappear");} function blockScroll(){$("body").toggleClass("fixedPosition");} -function openMenu(){addPlaceholder();blockScroll();}</script><div class="clearfix container-main-content"><div class="section-nav closed" data-offset-top=90 data-offset-bottom=500><span class="section-nav-back glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list data-section-nav><li><span class=section-nav-list-main-title>Documentation</span></li><li><a href=/documentation>Using the Documentation</a></li><li class=section-nav-item--collapsible><span class=section-nav-lis [...] +function openMenu(){addPlaceholder();blockScroll();}</script><div class="clearfix container-main-content"><div class="section-nav closed" data-offset-top=90 data-offset-bottom=500><span class="section-nav-back glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list data-section-nav><li><span class=section-nav-list-main-title>Documentation</span></li><li><a href=/documentation>Using the Documentation</a></li><li class=section-nav-item--collapsible><span class=section-nav-lis [...] Beam SDKs to create data processing pipelines. It provides guidance for using the Beam SDK classes to build and test your pipeline. The programming guide is not intended as an exhaustive reference, but as a language-agnostic, high-level @@ -5646,7 +5646,11 @@ implementations. This information can be accessed by using the conventional <span class=o>...</span> <span class=k>def</span> <span class=nf>infer_output_type</span><span class=p>(</span><span class=bp>self</span><span class=p>,</span> <span class=n>input_type</span><span class=p>):</span> - <span class=k>return</span> <span class=n>input_type</span></code></pre></div></div></div><div class=feedback><p class=update>Last updated on 2023/08/10</p><h3>Have you found everything you were looking for?</h3><p class=description>Was it all useful and clear? Is there anything that you would like to change? Let us know!</p><button class=load-button><a href="mailto:d...@beam.apache.org?subject=Beam Website Feedback">SEND FEEDBACK</a></button></div></div></div><footer class=footer><di [...] + <span class=k>return</span> <span class=n>input_type</span></code></pre></div></div></div><h2 id=transform-service>15 Transform Service</h2><p>Starting version 2.49.0, Beam introduced a <a href=https://docs.docker.com/compose/>Docker Compose</a> based service named <em>Transform Service</em>. The Transform Service allows Beam portable +pipelines to perform expansion of supported transforms using Docker.</p><p>The basic architecture of the Transform Service is given below.</p><p><img src=/images/transform_service.png alt="Diagram of the transform service architecture"></p><p>The Transform Service can be useful in many contexts. We have identified two primary use-cases below. Note that to use the transform service, Docker (and Docker Compose) needs to be available in the machine where the service will be started at.</p>< [...] +For example, with the Transform Service, a Beam Python pipeline can use Java GCP I/O transforms and Java Kafka I/O transforms without installing a Java runtime locally.</p><ul><li>Upgrade transforms without upgrading the Beam version.</li></ul><p>The transform service can be used to upgrade individual transforms used by Beam pipelines to a new Beam version without upgrading the Beam version used by the pipeline. +This feature is currently in development. Please see the <a href=https://github.com/apache/beam/issues/27943>tracking issue</a> for more details.</p><h3 id=transform-service-usage>15.1 Using the the Transform Service</h3><p>Beam SDKs may automatically startup a transform service to perform expansion when using cross-language transforms. More specifically,</p><ul><li><p>Java <a href=https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/ex [...] +startup the Transform Service for you if a Python runtime is not available locally but Docker is available.</p></li><li><p>Beam Python multi-language wrappers may automatically startup a Transform Service for you when using Java transforms, if a Java language runtime is not available locally but Docker is available.</p></li></ul><p>Additionally, if needed, a Transform Service instance can be manually started using utilities provided with Beam SDKs.</p><div class="language-java snippet">< [...] <a href=https://www.apache.org>The Apache Software Foundation</a> | <a href=/privacy_policy>Privacy Policy</a> | <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></div><div class="footer__cols__col footer__cols__col__logos"><div class=footer__cols__col--group><div class=footer__cols__col__logo><a href=https://github.com/apache/beam><im [...] \ No newline at end of file diff --git a/website/generated-content/images/transform_service.png b/website/generated-content/images/transform_service.png new file mode 100644 index 00000000000..f3763bac64e Binary files /dev/null and b/website/generated-content/images/transform_service.png differ diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 19c51a904f3..4c798142171 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.49.0/</loc><lastmod>2023-08-10T18:03:34-04:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2023-08-10T18:03:34-04:00</lastmod></url><url><loc>/blog/</loc><lastmod>2023-08-10T18:03:34-04:00</lastmod></url><url><loc>/categories/</loc><lastmod>2023-08-10T18:03:34-04:00</lastmod></url><url><loc>/catego [...] \ No newline at end of file +<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.49.0/</loc><lastmod>2023-08-10T15:24:25-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2023-08-10T15:24:25-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2023-08-10T15:24:25-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2023-08-10T15:24:25-07:00</lastmod></url><url><loc>/catego [...] \ No newline at end of file