This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 10ec0c0d89487106100d1273d49003c329d49bed Author: Mergebot <[email protected]> AuthorDate: Thu Feb 22 14:19:11 2018 -0800 Prepare repository for deployment. --- content/documentation/runners/dataflow/index.html | 50 +++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/content/documentation/runners/dataflow/index.html b/content/documentation/runners/dataflow/index.html index 527b9b1..1ad92b1 100644 --- a/content/documentation/runners/dataflow/index.html +++ b/content/documentation/runners/dataflow/index.html @@ -105,6 +105,7 @@ <li><a href="#cloud-dataflow-runner-prerequisites-and-setup">Cloud Dataflow Runner prerequisites and setup</a> <ul> <li><a href="#specify-your-dependency">Specify your dependency</a></li> + <li><a href="#self-executing-jar">Self executing JAR</a></li> <li><a href="#authentication">Authentication</a></li> </ul> </li> @@ -196,6 +197,55 @@ Pub/Sub, or Cloud Datastore) if you use them in your pipeline code.</p> <p><span class="language-py">This section is not applicable to the Beam SDK for Python.</span></p> +<h3 id="self-executing-jar">Self executing JAR</h3> + +<p class="language-py">This section is not applicable to the Beam SDK for Python.</p> + +<p class="language-java">In some cases, such as starting a pipeline using a scheduler such as <a href="https://airflow.apache.org";>Apache AirFlow</a>, you must have a self-contained application. You can pack a self-executing JAR by explicitly adding the following dependency on the Project section of your pom.xml, in addition to the adding existing dependency shown in the previous section.</p> + +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="o"><</span><span class="n">dependency</span><span class="o">></span> + <span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span> + <span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">beam</span><span class="o">-</span><span class="n">runners</span><span class="o">-</span><span class="n">google</span><span class="o">-</span><span class="n">cloud</span><span class="o">-</span><span class="n">dataflow</span><span class="o">-</span><span class="n">java</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span> + <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="err">$</span><span class="o">{</span><span class="n">beam</span><span class="o">.</span><span class="na">version</span><span class="o">}</</span><span class="n">version</span><span class="o">></span> + <span class="o"><</span><span class="n">scope</span><span class="o">></span><span class="n">runtime</span><span class="o"></</span><span class="n">scope</span><span class="o">></span> +<span class="o"></</span><span class="n">dependency</span><span class="o">></span> +</code></pre> +</div> + +<p class="language-java">Then, add the mainClass name in the Maven JAR plugin.</p> + +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="o"><</span><span class="n">plugin</span><span class="o">></span> + <span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">maven</span><span class="o">.</span><span class="na">plugins</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span> + <span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">maven</span><span class="o">-</span><span class="n">jar</span><span class="o">-</span><span class="n">plugin</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span> + <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="err">$</span><span class="o">{</span><span class="n">maven</span><span class="o">-</span><span class="n">jar</span><span class="o">-</span><span class="n">plugin</span><span class="o">.</span><span class="na">version</span><span class="o">}</</span><span class="n">version</span><span class="o">></span> + <span class="o"><</span><span class="n">configuration</span><span class="o">></span> + <span class="o"><</span><span class="n">archive</span><span class="o">></span> + <span class="o"><</span><span class="n">manifest</span><span class="o">></span> + <span class="o"><</span><span class="n">addClasspath</span><span class="o">></span><span class="kc">true</span><span class="o"></</span><span class="n">addClasspath</span><span class="o">></span> + <span class="o"><</span><span class="n">classpathPrefix</span><span class="o">></span><span class="n">lib</span><span class="o">/</</span><span class="n">classpathPrefix</span><span class="o">></span> + <span class="o"><</span><span class="n">mainClass</span><span class="o">></span><span class="n">YOUR_MAIN_CLASS_NAME</span><span class="o"></</span><span class="n">mainClass</span><span class="o">></span> + <span class="o"></</span><span class="n">manifest</span><span class="o">></span> + <span class="o"></</span><span class="n">archive</span><span class="o">></span> + <span class="o"></</span><span class="n">configuration</span><span class="o">></span> +<span class="o"></</span><span class="n">plugin</span><span class="o">></span> +</code></pre> +</div> + +<p class="language-java">After running <code>mvn package</code>, run <code>ls target</code> and you should see (assuming your artifactId is <code class="highlighter-rouge">beam-examples</code> and the version is 1.0.0) the following output.</p> + +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">beam</span><span class="o">-</span><span class="n">examples</span><span class="o">-</span><span class="n">bundled</span><span class="o">-</span><span class="mf">1.0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="na">jar</span> +</code></pre> +</div> + +<p class="language-java">To run the self-executing JAR on Cloud Dataflow, use the following command.</p> + +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">java</span> <span class="o">-</span><span class="n">jar</span> <span class="n">target</span><span class="o">/</span><span class="n">beam</span><span class="o">-</span><span class="n">examples</span><span class="o">-</span><span class="n">bundled</span><span class="o">-</span><span class="mf">1.0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="na">jar</span [...] + <span class="o">--</span><span class="n">runner</span><span class="o">=</span><span class="n">DataflowRunner</span> <span class="err">\</span> + <span class="o">--</span><span class="n">project</span><span class="o">=<</span><span class="n">YOUR_GCP_PROJECT_ID</span><span class="o">></span> <span class="err">\</span> + <span class="o">--</span><span class="n">tempLocation</span><span class="o">=</span><span class="nl">gs:</span><span class="c1">//<YOUR_GCS_BUCKET>/temp/</span> +</code></pre> +</div> + <h3 id="authentication">Authentication</h3> <p>Before running your pipeline, you must authenticate with the Google Cloud Platform. Run the following command to get <a href="https://developers.google.com/identity/protocols/application-default-credentials";>Application Default Credentials</a>.</p> -- To stop receiving notification emails like this one, please contact [email protected].
