Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/reprocessing.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/reprocessing.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/jobs/reprocessing.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/jobs/reprocessing.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/jobs/reprocessing.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -106,23 +114,6 @@ <h2>Reprocessing previously processed data</h2> -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> - <p>From time to time you may want to deploy a new version of your Samza job that computes results differently. Perhaps you fixed a bug or introduced a new feature. For example, say you have a Samza job that classifies messages as spam or not-spam, using a machine learning model that you train offline. Periodically you want to deploy an updated version of your Samza job which includes the latest classification model.</p> <p>When you start up a new version of your job, a question arises: what do you want to do with messages that were previously processed with the old version of your job? The answer depends on the behavior you want:</p>
Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/yarn-jobs.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/yarn-jobs.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/jobs/yarn-jobs.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/jobs/yarn-jobs.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/jobs/yarn-jobs.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -123,12 +131,11 @@ limitations under the License. --> -<p>When you define <code>job.factory.class=org.apache.samza.job.yarn.YarnJobFactory</code> in your job’s configuration, Samza will use YARN to execute your job. The YarnJobFactory will use the YARN_HOME environment variable on the machine that run-job.sh is executed on to get the appropriate YARN configuration, which will define where the YARN resource manager is. The YarnJob will work with the resource manager to get your job started on the YARN cluster.</p> +<p>When you define job.factory.class=org.apache.samza.job.yarn.YarnJobFactory in your job’s configuration, Samza will use YARN to execute your job. The YarnJobFactory will use the YARN_HOME environment variable on the machine that run-job.sh is executed on to get the appropriate YARN configuration, which will define where the YARN resource manager is. The YarnJob will work with the resource manager to get your job started on the YARN cluster.</p> <p>If you want to use YARN to run your Samza job, you’ll also need to define the location of your Samza job’s package. For example, you might say:</p> - -<div class="highlight"><pre><code class="jproperties"><span class="na">yarn.package.path</span><span class="o">=</span><span class="s">http://my.http.server/jobs/ingraphs-package-0.0.55.tgz</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">yarn.package.path=http://my.http.server/jobs/ingraphs-package-0.0.55.tgz +</code></pre></div> <p>This .tgz file follows the conventions outlined on the <a href="packaging.html">Packaging</a> page (it has bin/run-am.sh and bin/run-container.sh). YARN NodeManagers will take responsibility for downloading this .tgz file on the appropriate machines, and untar'ing them. From there, YARN will execute run-am.sh or run-container.sh for the Samza Application Master, and SamzaContainer, respectively.</p> <!-- TODO document yarn.container.count and other key configs --> Modified: incubator/samza/site/learn/documentation/0.7.0/operations/kafka.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/operations/kafka.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/operations/kafka.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/operations/kafka.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/operations/kafka.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -132,9 +140,8 @@ <h3 id="auto-create-topics">Auto-Create Topics</h3> <p>Kafka brokers should be configured to automatically create topics. Without this, it’s going to be very cumbersome to run Samze jobs, since jobs will write to arbitrary (and sometimes new) topics.</p> - -<div class="highlight"><pre><code class="jproperties"><span class="na">auto.create.topics.enable</span><span class="o">=</span><span class="s">true</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">auto.create.topics.enable=true +</code></pre></div> </div> </div> Modified: incubator/samza/site/learn/documentation/0.7.0/operations/security.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/operations/security.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/operations/security.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/operations/security.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/operations/security.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> Modified: incubator/samza/site/learn/documentation/0.7.0/yarn/application-master.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/yarn/application-master.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/yarn/application-master.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/yarn/application-master.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/yarn/application-master.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> Modified: incubator/samza/site/learn/documentation/0.7.0/yarn/isolation.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/yarn/isolation.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/yarn/isolation.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/yarn/isolation.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/yarn/isolation.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> Modified: incubator/samza/site/learn/tutorials/0.7.0/deploy-samza-job-from-hdfs.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/tutorials/0.7.0/deploy-samza-job-from-hdfs.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/tutorials/0.7.0/deploy-samza-job-from-hdfs.html (original) +++ incubator/samza/site/learn/tutorials/0.7.0/deploy-samza-job-from-hdfs.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/tutorials/latest/deploy-samza-job-from-hdfs.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -125,20 +133,45 @@ <p>This tutorial uses <a href="../../../startup/hello-samza/0.7.0/">hello-samza</a> to illustrate how to run a Samza job if you want to publish the Samza job’s .tar.gz package to HDFS.</p> -<h3 id="upload-the-package">Upload the package</h3> +<h3 id="build-a-new-samza-job-package">Build a new Samza job package</h3> -<div class="highlight"><pre><code class="bash">hadoop fs -put ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz /path/for/tgz</code></pre></div> +<p>Build a new Samza job package to include the hadoop-hdfs-version.jar.</p> +<ul> +<li>Add dependency statement in pom.xml of samza-job-package</li> +</ul> +<div class="highlight"><pre><code class="language-text" data-lang="text"><dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-hdfs</artifactId> + <version>2.2.0</version> +</dependency> +</code></pre></div> +<ul> +<li>Add the following code to src/main/assembly/src.xml in samza-job-package.</li> +</ul> +<div class="highlight"><pre><code class="language-text" data-lang="text"><include>org.apache.hadoop:hadoop-hdfs</include> +</code></pre></div> +<ul> +<li>Create .tar.gz package</li> +</ul> +<div class="highlight"><pre><code class="language-text" data-lang="text">mvn clean pacakge +</code></pre></div> +<ul> +<li>Make sure hadoop-common-version.jar has the same version as your hadoop-hdfs-version.jar. Otherwise, you may still have errors.</li> +</ul> + +<h3 id="upload-the-package">Upload the package</h3> +<div class="highlight"><pre><code class="language-text" data-lang="text">hadoop fs -put ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz /path/for/tgz +</code></pre></div> <h3 id="add-hdfs-configuration">Add HDFS configuration</h3> -<p>Put the hdfs-site.xml file of your cluster into ~/.samza/conf directory (The same place as the yarn-site.xml). If you set HADOOP_CONF_DIR, put the hdfs-site.xml in your configuration directory if the hdfs-site.xml is not there.</p> +<p>Put the hdfs-site.xml file of your cluster into ~/.samza/conf directory. (The same place as the yarn-site.xml)</p> <h3 id="change-properties-file">Change properties file</h3> <p>Change the yarn.package.path in the properties file to your HDFS location.</p> - -<div class="highlight"><pre><code class="jproperties"><span class="na">yarn.package.path</span><span class="o">=</span><span class="s">hdfs://<hdfs name node ip>:<hdfs name node port>/path/to/tgz</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">yarn.package.path=hdfs://<hdfs name node ip>:<hdfs name node port>/path/to/tgz +</code></pre></div> <p>Then you should be able to run the Samza job as described in <a href="../../../startup/hello-samza/0.7.0/">hello-samza</a>.</p> Modified: incubator/samza/site/learn/tutorials/0.7.0/index.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/tutorials/0.7.0/index.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/tutorials/0.7.0/index.html (original) +++ incubator/samza/site/learn/tutorials/0.7.0/index.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/tutorials/latest/index.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> Modified: incubator/samza/site/learn/tutorials/0.7.0/remote-debugging-samza.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/tutorials/0.7.0/remote-debugging-samza.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/tutorials/0.7.0/remote-debugging-samza.html (original) +++ incubator/samza/site/learn/tutorials/0.7.0/remote-debugging-samza.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/tutorials/latest/remote-debugging-samza.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -128,32 +136,27 @@ <h3 id="get-the-code">Get the Code</h3> <p>Start by checking out Samza, so we have access to the source.</p> - -<div class="highlight"><pre><code class="bash">git clone http://git-wip-us.apache.org/repos/asf/incubator-samza.git</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">git clone http://git-wip-us.apache.org/repos/asf/incubator-samza.git +</code></pre></div> <p>Next, grab hello-samza.</p> - -<div class="highlight"><pre><code class="bash">git clone git://git.apache.org/incubator-samza-hello-samza.git</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">git clone git://git.apache.org/incubator-samza-hello-samza.git +</code></pre></div> <h3 id="setup-the-environment">Setup the Environment</h3> <p>Now, let’s setup the Eclipse project files.</p> - -<div class="highlight"><pre><code class="bash"><span class="nb">cd </span>incubator-samza -./gradlew eclipse</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">cd incubator-samza +./gradlew eclipse +</code></pre></div> <p>Let’s also release Samza to Maven’s local repository, so hello-samza has access to the JARs that it needs.</p> - -<div class="highlight"><pre><code class="bash">./gradlew -PscalaVersion<span class="o">=</span>2.9.2 clean publishToMavenLocal</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">./gradlew -PscalaVersion=2.9.2 clean publishToMavenLocal +</code></pre></div> <p>Next, open Eclipse, and import the Samza source code into your workspace: “File” > “Import” > “Existing Projects into Workspace” > “Browse”. Select ‘incubator-samza’ folder, and hit ‘finish’.</p> <h3 id="enable-remote-debugging">Enable Remote Debugging</h3> <p>Now, go back to the hello-samza project, and edit ./samza-job-package/src/main/config/wikipedia-feed.properties to add the following line:</p> - -<div class="highlight"><pre><code class="jproperties"><span class="na">task.opts</span><span class="o">=</span><span class="s">-agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">task.opts=-agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y +</code></pre></div> <p>The <a href="../../documentation/0.7.0/jobs/configuration-table.html">task.opts</a> configuration parameter is a way to override Java parameters at runtime for your Samza containers. In this example, we’re setting the agentlib parameter to enable remote debugging on localhost, port 9009. In a more realistic environment, you might also set Java heap settings (-Xmx, -Xms, etc), as well as garbage collection and logging settings.</p> <p><em>NOTE: If you’re running multiple Samza containers on the same machine, there is a potential for port collisions. You must configure your task.opts to assign different ports for different Samza jobs. If a Samza job has more than one container (e.g. if you’re using YARN with yarn.container.count=2), those containers must be run on different machines.</em></p> @@ -161,20 +164,17 @@ <h3 id="start-the-grid">Start the Grid</h3> <p>Now that the Samza job has been setup to enable remote debugging when a Samza container starts, let’s start the ZooKeeper, Kafka, and YARN.</p> - -<div class="highlight"><pre><code class="bash">bin/grid</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">bin/grid +</code></pre></div> <p>If you get a complaint that JAVA_HOME is not set, then you’ll need to set it. This can be done on OSX by running:</p> - -<div class="highlight"><pre><code class="bash"><span class="nb">export </span><span class="nv">JAVA_HOME</span><span class="o">=</span><span class="k">$(</span>/usr/libexec/java_home<span class="k">)</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">export JAVA_HOME=$(/usr/libexec/java_home) +</code></pre></div> <p>Once the grid starts, you can start the wikipedia-feed Samza job.</p> - -<div class="highlight"><pre><code class="bash">mvn clean package +<div class="highlight"><pre><code class="language-text" data-lang="text">mvn clean package mkdir -p deploy/samza tar -xvf ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz -C deploy/samza -deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></div> - +deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties +</code></pre></div> <p>When the wikipedia-feed job starts up, a single Samza container will be created to process all incoming messages. This is the container that we’ll want to connect to from the remote debugger.</p> <h3 id="connect-the-remote-debugger">Connect the Remote Debugger</h3> Modified: incubator/samza/site/learn/tutorials/0.7.0/run-hello-samza-without-internet.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/tutorials/0.7.0/run-hello-samza-without-internet.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/tutorials/0.7.0/run-hello-samza-without-internet.html (original) +++ incubator/samza/site/learn/tutorials/0.7.0/run-hello-samza-without-internet.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/tutorials/latest/run-hello-samza-without-internet.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -128,9 +136,8 @@ <h3 id="test-your-connection">Test Your Connection</h3> <p>Ping irc.wikimedia.org. Sometimes the firewall in your company blocks this service.</p> - -<div class="highlight"><pre><code class="bash">telnet irc.wikimedia.org 6667</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">telnet irc.wikimedia.org 6667 +</code></pre></div> <p>You should see something like this:</p> <div class="highlight"><pre><code class="language-text" data-lang="text">Trying 208.80.152.178... Connected to ekrem.wikimedia.org. @@ -145,25 +152,23 @@ NOTICE AUTH :*** Found your hostname <h3 id="use-local-data-to-run-hello-samza">Use Local Data to Run Hello Samza</h3> <p>We provide an alternative to get wikipedia feed data. Instead of running</p> - -<div class="highlight"><pre><code class="bash">deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></div> - -<p>You will run</p> - -<div class="highlight"><pre><code class="bash">bin/produce-wikipedia-raw-data.sh</code></pre></div> +<div class="highlight"><pre><code class="language-text" data-lang="text">deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties +</code></pre></div> +<p>You will run +<code> +bin/produce-wikipedia-raw-data.sh +</code> </p> <p>This script will read wikipedia feed data from local file and produce them to the Kafka broker. By default, it produces to localhost:9092 as the Kafka broker and uses localhost:2181 as zookeeper. You can overwrite them:</p> - -<div class="highlight"><pre><code class="bash">bin/produce-wikipedia-raw-data.sh -b yourKafkaBrokerAddress -z yourZookeeperAddress</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">bin/produce-wikipedia-raw-data.sh -b yourKafkaBrokerAddress -z yourZookeeperAddress +</code></pre></div> <p>Now you can go back to Generate Wikipedia Statistics section in <a href="../../../startup/hello-samza/0.7.0/">Hello Samza</a> and follow the remaining steps.</p> <h3 id="a-little-explanation">A Little Explanation</h3> -<p>The goal of</p> - -<div class="highlight"><pre><code class="bash">deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></div> - +<p>The goal of </p> +<div class="highlight"><pre><code class="language-text" data-lang="text">deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties +</code></pre></div> <p>is to deploy a Samza job which listens to wikipedia API, receives the feed in realtime and produces the feed to the Kafka topic wikipedia-raw. The alternative in this tutorial is reading local wikipedia feed in an infinite loop and producing the data to Kafka wikipedia-raw. The follow-up job, wikipedia-parser is getting data from Kafka topic wikipedia-raw, so as long as we have correct data in Kafka topic wikipedia-raw, we are fine. All Samza jobs are connected by the Kafka and do not depend on each other.</p> Modified: incubator/samza/site/learn/tutorials/0.7.0/run-in-multi-node-yarn.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/tutorials/0.7.0/run-in-multi-node-yarn.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/tutorials/0.7.0/run-in-multi-node-yarn.html (original) +++ incubator/samza/site/learn/tutorials/0.7.0/run-in-multi-node-yarn.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/tutorials/latest/run-in-multi-node-yarn.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -132,30 +140,26 @@ <h3 id="basic-yarn-setting">Basic YARN Setting</h3> <p>1. Dowload <a href="http://mirror.symnds.com/software/Apache/hadoop/common/hadoop-2.3.0/hadoop-2.3.0.tar.gz";>YARN 2.3</a> to /tmp and untar it.</p> - -<div class="highlight"><pre><code class="bash"><span class="nb">cd</span> /tmp +<div class="highlight"><pre><code class="language-text" data-lang="text">cd /tmp tar -xvf hadoop-2.3.0.tar.gz -<span class="nb">cd </span>hadoop-2.3.0</code></pre></div> - +cd hadoop-2.3.0 +</code></pre></div> <p>2. Set up environment variables.</p> - -<div class="highlight"><pre><code class="bash"><span class="nb">export </span><span class="nv">HADOOP_YARN_HOME</span><span class="o">=</span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span> +<div class="highlight"><pre><code class="language-text" data-lang="text">export HADOOP_YARN_HOME=$(pwd) mkdir conf -<span class="nb">export </span><span class="nv">HADOOP_CONF_DIR</span><span class="o">=</span><span class="nv">$HADOOP_YARN_HOME</span>/conf</code></pre></div> - +export HADOOP_CONF_DIR=$HADOOP_YARN_HOME/conf +</code></pre></div> <p>3. Configure YARN setting file.</p> - -<div class="highlight"><pre><code class="bash">cp ./etc/hadoop/yarn-site.xml conf -vi conf/yarn-site.xml</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">cp ./etc/hadoop/yarn-site.xml conf +vi conf/yarn-site.xml +</code></pre></div> <p>Add the following property to yarn-site.xml:</p> - -<div class="highlight"><pre><code class="xml"><span class="nt"><property></span> - <span class="nt"><name></span>yarn.resourcemanager.hostname<span class="nt"></name></span> - <span class="c"><!-- hostname that is accessible from all NMs --></span> - <span class="nt"><value></span>yourHostname<span class="nt"></value></span> -<span class="nt"></property></span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text"><property> + <name>yarn.resourcemanager.hostname</name> + <!-- hostname that is accessible from all NMs --> + <value>yourHostname</value> +</property> +</code></pre></div> <p>Download and add capacity-schedule.xml.</p> <div class="highlight"><pre><code class="language-text" data-lang="text">curl http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml?view=co > conf/capacity-scheduler.xml </code></pre></div> @@ -164,39 +168,34 @@ vi conf/yarn-site.xml</code></pre></div> <p>The goal of these steps is to configure YARN to read http filesystem because we will use Http server to deploy Samza job package. If you want to use HDFS to deploy Samza job package, you can skip step 4~6 and follow <a href="deploy-samza-job-from-hdfs.html">Deploying a Samza Job from HDFS</a></p> <p>4. Download Scala package and untar it.</p> - -<div class="highlight"><pre><code class="bash"><span class="nb">cd</span> /tmp +<div class="highlight"><pre><code class="language-text" data-lang="text">cd /tmp curl http://www.scala-lang.org/files/archive/scala-2.10.3.tgz > scala-2.10.3.tgz -tar -xvf scala-2.10.3.tgz</code></pre></div> - +tar -xvf scala-2.10.3.tgz +</code></pre></div> <p>5. Add Scala and its log jars.</p> - -<div class="highlight"><pre><code class="bash">cp /tmp/scala-2.10.3/lib/scala-compiler.jar <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib -cp /tmp/scala-2.10.3/lib/scala-library.jar <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib -curl http://search.maven.org/remotecontent?filepath<span class="o">=</span>org/clapper/grizzled-slf4j_2.10/1.0.1/grizzled-slf4j_2.10-1.0.1.jar > <span class="nv">$HADOOP_YARN_HOME</span>/share/hadoop/hdfs/lib/grizzled-slf4j_2.10-1.0.1.jar</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">cp /tmp/scala-2.10.3/lib/scala-compiler.jar $HADOOP_YARN_HOME/share/hadoop/hdfs/lib +cp /tmp/scala-2.10.3/lib/scala-library.jar $HADOOP_YARN_HOME/share/hadoop/hdfs/lib +curl http://search.maven.org/remotecontent?filepath=org/clapper/grizzled-slf4j_2.10/1.0.1/grizzled-slf4j_2.10-1.0.1.jar > $HADOOP_YARN_HOME/share/hadoop/hdfs/lib/grizzled-slf4j_2.10-1.0.1.jar +</code></pre></div> <p>6. Add http configuration in core-site.xml (create the core-site.xml file and add content).</p> - -<div class="highlight"><pre><code class="xml">vi $HADOOP_YARN_HOME/conf/core-site.xml</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">vi $HADOOP_YARN_HOME/conf/core-site.xml +</code></pre></div> <p>Add the following code:</p> - -<div class="highlight"><pre><code class="xml"><span class="cp"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></span> -<span class="nt"><configuration></span> - <span class="nt"><property></span> - <span class="nt"><name></span>fs.http.impl<span class="nt"></name></span> - <span class="nt"><value></span>org.apache.samza.util.hadoop.HttpFileSystem<span class="nt"></value></span> - <span class="nt"></property></span> -<span class="nt"></configuration></span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?> +<configuration> + <property> + <name>fs.http.impl</name> + <value>org.apache.samza.util.hadoop.HttpFileSystem</value> + </property> +</configuration> +</code></pre></div> <h3 id="distribute-hadoop-file-to-slaves">Distribute Hadoop File to Slaves</h3> <p>7. Basically, you copy the hadoop file in your host machine to slave machines. (172.21.100.35, in my case):</p> - -<div class="highlight"><pre><code class="bash">scp -r . 172.21.100.35:/tmp/hadoop-2.3.0 -<span class="nb">echo </span>172.21.100.35 > conf/slaves -sbin/start-yarn.sh</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">scp -r . 172.21.100.35:/tmp/hadoop-2.3.0 +echo 172.21.100.35 > conf/slaves +sbin/start-yarn.sh +</code></pre></div> <ul> <li>If you get “172.21.100.35: Error: JAVA<em>HOME is not set and could not be found.”, you’ll need to add a conf/hadoop-env.sh file to the machine with the failure (172.21.100.35, in this case), which has “export JAVA</em>HOME=/export/apps/jdk/JDK-1<em>6</em>0<em>27” (or wherever your JAVA</em>HOME actually is).</li> </ul> @@ -208,39 +207,33 @@ sbin/start-yarn.sh</code></pre></div> <p>Some of the following steps are exactlly identical to what you have seen in <a href="../../../startup/hello-samza/0.7.0/">hello-samza</a>. You may skip them if you have already done so.</p> <p>1. Download Samza and publish it to Maven local repository.</p> - -<div class="highlight"><pre><code class="bash"><span class="nb">cd</span> /tmp +<div class="highlight"><pre><code class="language-text" data-lang="text">cd /tmp git clone http://git-wip-us.apache.org/repos/asf/incubator-samza.git -<span class="nb">cd </span>incubator-samza +cd incubator-samza ./gradlew clean publishToMavenLocal -<span class="nb">cd</span> ..</code></pre></div> - +cd .. +</code></pre></div> <p>2. Download hello-samza project and change the job properties file.</p> - -<div class="highlight"><pre><code class="bash">git clone git://github.com/linkedin/hello-samza.git -<span class="nb">cd </span>hello-samza -vi samza-job-package/src/main/config/wikipedia-feed.properties</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">git clone git://github.com/linkedin/hello-samza.git +cd hello-samza +vi samza-job-package/src/main/config/wikipedia-feed.properties +</code></pre></div> <p>Change the yarn.package.path property to be:</p> - -<div class="highlight"><pre><code class="jproperties"><span class="na">yarn.package.path</span><span class="o">=</span><span class="s">http://yourHostname:8000/samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">yarn.package.path=http://yourHostname:8000/samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz +</code></pre></div> <p>3. Complie hello-samza.</p> - -<div class="highlight"><pre><code class="bash">mvn clean package +<div class="highlight"><pre><code class="language-text" data-lang="text">mvn clean package mkdir -p deploy/samza -tar -xvf ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz -C deploy/samza</code></pre></div> - +tar -xvf ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz -C deploy/samza +</code></pre></div> <p>4. Deploy Samza job package to Http server..</p> <p>Open a new terminal, and run:</p> - -<div class="highlight"><pre><code class="bash"><span class="nb">cd</span> /tmp/hello-samza <span class="o">&&</span> python -m SimpleHTTPServer</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">cd /tmp/hello-samza && python -m SimpleHTTPServer +</code></pre></div> <p>Go back to the original terminal (not the one running the HTTP server):</p> - -<div class="highlight"><pre><code class="bash">deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties +</code></pre></div> <p>Go to http://yourHostname:8088 and find the wikipedia-feed job. Click on the ApplicationMaster link to see that it’s running.</p> <p>Congratulations! You now run the Samza job in a “real” YARN grid!</p> Modified: incubator/samza/site/less/main.less URL: http://svn.apache.org/viewvc/incubator/samza/site/less/main.less?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/less/main.less (original) +++ incubator/samza/site/less/main.less Fri Aug 15 05:36:35 2014 @@ -1,21 +1,19 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. @import "variables.less"; @import "non-responsive.less"; @@ -92,8 +90,6 @@ pre { border: 0px !important; border-radius: 0px !important; overflow-x: auto; - background-color: #f7f7f7; - font-size: 12px; code { overflow-wrap: normal; white-space: pre; @@ -166,10 +162,6 @@ td.key { font-size: 16px; } -img.diagram-large { - width: 100%; -} - ul.documentation-list { list-style: none; padding-left: 20px; Modified: incubator/samza/site/sitemap.xml URL: http://svn.apache.org/viewvc/incubator/samza/site/sitemap.xml?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/sitemap.xml (original) +++ incubator/samza/site/sitemap.xml Fri Aug 15 05:36:35 2014 @@ -29,28 +29,28 @@ <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/yarn/application-master.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/yarn/application-master.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/introduction/architecture.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/introduction/architecture.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/introduction/background.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/introduction/background.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/container/checkpointing.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/checkpointing.html</loc> <lastmod>2014-08-14</lastmod> @@ -78,21 +78,21 @@ </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/introduction/concepts.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/introduction/concepts.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/jobs/configuration.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/configuration.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/tutorials/versioned/deploy-samza-job-from-hdfs.html</loc> + <loc>http://samza.incubator.apache.org/learn/tutorials/0.7.0/deploy-samza-job-from-hdfs.html</loc> <lastmod>2014-08-14</lastmod> @@ -106,21 +106,21 @@ </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/container/event-loop.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/event-loop.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/index.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/index.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/archive/index.html</loc> + <loc>http://samza.incubator.apache.org/learn/tutorials/0.7.0/index.html</loc> <lastmod>2014-08-14</lastmod> @@ -134,13 +134,6 @@ </url> <url> - <loc>http://samza.incubator.apache.org/learn/tutorials/versioned/index.html</loc> - <lastmod>2014-08-14</lastmod> - - - </url> - - <url> <loc>http://samza.incubator.apache.org/startup/download/index.html</loc> <lastmod>2014-08-14</lastmod> @@ -148,14 +141,14 @@ </url> <url> - <loc>http://samza.incubator.apache.org/startup/hello-samza/versioned/index.html</loc> + <loc>http://samza.incubator.apache.org/startup/hello-samza/0.7.0/index.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/comparisons/introduction.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/comparisons/introduction.html</loc> <lastmod>2014-08-14</lastmod> @@ -169,35 +162,35 @@ </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/yarn/isolation.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/yarn/isolation.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/container/jmx.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/jmx.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/jobs/job-runner.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/job-runner.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/operations/kafka.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/operations/kafka.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/jobs/logging.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/logging.html</loc> <lastmod>2014-08-14</lastmod> @@ -211,28 +204,28 @@ </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/container/metrics.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/metrics.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/comparisons/mupd8.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/comparisons/mupd8.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/api/overview.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/api/overview.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/jobs/packaging.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/packaging.html</loc> <lastmod>2014-08-14</lastmod> @@ -246,14 +239,14 @@ </url> <url> - <loc>http://samza.incubator.apache.org/learn/tutorials/versioned/remote-debugging-samza.html</loc> + <loc>http://samza.incubator.apache.org/learn/tutorials/0.7.0/remote-debugging-samza.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/jobs/reprocessing.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/reprocessing.html</loc> <lastmod>2014-08-14</lastmod> @@ -267,28 +260,28 @@ </url> <url> - <loc>http://samza.incubator.apache.org/learn/tutorials/versioned/run-hello-samza-without-internet.html</loc> + <loc>http://samza.incubator.apache.org/learn/tutorials/0.7.0/run-hello-samza-without-internet.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/tutorials/versioned/run-in-multi-node-yarn.html</loc> + <loc>http://samza.incubator.apache.org/learn/tutorials/0.7.0/run-in-multi-node-yarn.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/container/samza-container.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/samza-container.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/operations/security.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/operations/security.html</loc> <lastmod>2014-08-14</lastmod> @@ -302,49 +295,42 @@ </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/container/serialization.html</loc> - <lastmod>2014-08-14</lastmod> - - - </url> - - <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/comparisons/spark-streaming.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/serialization.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/container/state-management.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/state-management.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/comparisons/storm.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/comparisons/storm.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/container/streams.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/streams.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/container/windowing.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/windowing.html</loc> <lastmod>2014-08-14</lastmod> </url> <url> - <loc>http://samza.incubator.apache.org/learn/documentation/versioned/jobs/yarn-jobs.html</loc> + <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/yarn-jobs.html</loc> <lastmod>2014-08-14</lastmod> Modified: incubator/samza/site/startup/download/index.html URL: http://svn.apache.org/viewvc/incubator/samza/site/startup/download/index.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/startup/download/index.html (original) +++ incubator/samza/site/startup/download/index.html Fri Aug 15 05:36:35 2014 @@ -129,93 +129,74 @@ limitations under the License. --> -<p>Samza is released as a source artifact, and also through Maven.</p> +<!-- TODO update maven dependency versions appropriately --> -<p>If you just want to play around with Samza for the first time, go to <a href="/startup/hello-samza/latest">Hello Samza</a>.</p> +<p>If you want to play around with Samza for the first time, go to <a href="/startup/hello-samza/0.7.0">Hello Samza</a>.</p> -<h3 id="source-releases">Source Releases</h3> +<!-- +### Maven -<ul> -<li><a href="http://www.apache.org/dyn/closer.cgi/incubator/samza/0.7.0-incubating";>samza-sources-0.7.0-incubating.tgz</a></li> -</ul> - -<h3 id="maven">Maven</h3> - -<p>All Samza JARs are published through <a href="https://repository.apache.org/content/groups/public/org/apache/samza/";>Apache’s Maven repository</a>.</p> - -<h4 id="artifacts">Artifacts</h4> - -<p>A Maven-based Samza project can pull in all required dependencies Samza dependencies this XML block:</p> - -<div class="highlight"><pre><code class="xml"><span class="nt"><dependency></span> - <span class="nt"><groupId></span>org.apache.samza<span class="nt"></groupId></span> - <span class="nt"><artifactId></span>samza-api<span class="nt"></artifactId></span> - <span class="nt"><version></span>0.7.0<span class="nt"></version></span> -<span class="nt"></dependency></span> -<span class="nt"><dependency></span> - <span class="nt"><groupId></span>org.apache.samza<span class="nt"></groupId></span> - <span class="nt"><artifactId></span>samza-core_2.10<span class="nt"></artifactId></span> - <span class="nt"><version></span>0.7.0<span class="nt"></version></span> - <span class="nt"><scope></span>runtime<span class="nt"></scope></span> -<span class="nt"></dependency></span> -<span class="nt"><dependency></span> - <span class="nt"><groupId></span>org.apache.samza<span class="nt"></groupId></span> - <span class="nt"><artifactId></span>samza-serializers_2.10<span class="nt"></artifactId></span> - <span class="nt"><version></span>0.7.0<span class="nt"></version></span> -<span class="nt"></dependency></span> -<span class="nt"><dependency></span> - <span class="nt"><groupId></span>org.apache.samza<span class="nt"></groupId></span> - <span class="nt"><artifactId></span>samza-shell<span class="nt"></artifactId></span> - <span class="nt"><classifier></span>dist<span class="nt"></classifier></span> - <span class="nt"><type></span>tgz<span class="nt"></type></span> - <span class="nt"><version></span>0.7.0<span class="nt"></version></span> - <span class="nt"><scope></span>runtime<span class="nt"></scope></span> -<span class="nt"></dependency></span> -<span class="nt"><dependency></span> - <span class="nt"><groupId></span>org.apache.samza<span class="nt"></groupId></span> - <span class="nt"><artifactId></span>samza-yarn_2.10<span class="nt"></artifactId></span> - <span class="nt"><version></span>0.7.0<span class="nt"></version></span> - <span class="nt"><scope></span>runtime<span class="nt"></scope></span> -<span class="nt"></dependency></span> -<span class="nt"><dependency></span> - <span class="nt"><groupId></span>org.apache.samza<span class="nt"></groupId></span> - <span class="nt"><artifactId></span>samza-kv_2.10<span class="nt"></artifactId></span> - <span class="nt"><version></span>0.7.0<span class="nt"></version></span> - <span class="nt"><scope></span>runtime<span class="nt"></scope></span> -<span class="nt"></dependency></span> -<span class="nt"><dependency></span> - <span class="nt"><groupId></span>org.apache.samza<span class="nt"></groupId></span> - <span class="nt"><artifactId></span>samza-kafka_2.10<span class="nt"></artifactId></span> - <span class="nt"><version></span>0.7.0<span class="nt"></version></span> - <span class="nt"><scope></span>runtime<span class="nt"></scope></span> -<span class="nt"></dependency></span></code></pre></div> - -<p><a href="/startup/hello-samza/latest">Hello Samza</a> is a working Maven project that illustrates how to build projects that have Samza jobs in them.</p> - -<h4 id="repositories">Repositories</h4> - -<p>Samza is available in the Apache Maven repository.</p> - -<div class="highlight"><pre><code class="xml"><span class="nt"><repository></span> - <span class="nt"><id></span>apache-releases<span class="nt"></id></span> - <span class="nt"><url></span>https://repository.apache.org/content/groups/public<span class="nt"></url></span> -<span class="nt"></repository></span></code></pre></div> - -<p>Snapshot builds are available in the Apache Maven snapshot repository.</p> - -<div class="highlight"><pre><code class="xml"><span class="nt"><repository></span> - <span class="nt"><id></span>apache-snapshots<span class="nt"></id></span> - <span class="nt"><url></span>https://repository.apache.org/content/groups/snapshots<span class="nt"></url></span> -<span class="nt"></repository></span></code></pre></div> +All Samza JARs are published through Maven. + +#### Artifacts + +A Samza project that runs with Kafka and YARN should depend on the following artifacts. + + <dependency> + <groupId>samza</groupId> + <artifactId>samza-api</artifactId> + <version>0.7.0</version> + </dependency> + <dependency> + <groupId>samza</groupId> + <artifactId>samza-core_2.9.2</artifactId> + <version>0.7.0</version> + <scope>runtime</scope> + </dependency> + <dependency> + <groupId>samza</groupId> + <artifactId>samza-serializers_2.9.2</artifactId> + <version>0.7.0</version> + <scope>runtime</scope> + </dependency> + <dependency> + <groupId>samza</groupId> + <artifactId>samza-yarn_2.9.2</artifactId> + <version>0.7.0</version> + <classifier>yarn-2.0.5-alpha</classifier> + <scope>runtime</scope> + </dependency> + <dependency> + <groupId>samza</groupId> + <artifactId>samza-kafka_2.9.2</artifactId> + <version>0.7.0</version> + <scope>runtime</scope> + </dependency> + +#### Repositories + +Samza is available in the Apache Maven repository. + + <repository> + <id>apache-releases</id> + <url>https://repository.apache.org/content/groups/public</url> + </repository> + +Snapshot builds are available in the Apache Maven snapshot repository. + + <repository> + <id>apache-snapshots</id> + <url>https://repository.apache.org/content/groups/snapshots</url> + </repository> +–> <h3 id="checking-out-and-building">Checking out and Building</h3> <p>If you’re interested in working on Samza, or building the JARs from scratch, then you’ll need to checkout and build the code. Samza does not have a binary release at this time. To check out and build Samza, run these commands.</p> - -<div class="highlight"><pre><code class="bash">git clone http://git-wip-us.apache.org/repos/asf/incubator-samza.git -<span class="nb">cd </span>incubator-samza -./gradlew clean build</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">git clone http://git-wip-us.apache.org/repos/asf/incubator-samza.git +cd incubator-samza +./gradlew clean build +</code></pre></div> <p>See the README.md file for details on building.</p> Modified: incubator/samza/site/startup/hello-samza/0.7.0/index.html URL: http://svn.apache.org/viewvc/incubator/samza/site/startup/hello-samza/0.7.0/index.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/startup/hello-samza/0.7.0/index.html (original) +++ incubator/samza/site/startup/hello-samza/0.7.0/index.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/startup/hello-samza/latest/index.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -128,18 +136,16 @@ <h3 id="get-the-code">Get the Code</h3> <p>Check out the hello-samza project:</p> - -<div class="highlight"><pre><code class="bash">git clone git://git.apache.org/incubator-samza-hello-samza.git hello-samza -<span class="nb">cd </span>hello-samza</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">git clone git://git.apache.org/incubator-samza-hello-samza.git hello-samza +cd hello-samza +</code></pre></div> <p>This project contains everything you’ll need to run your first Samza jobs.</p> <h3 id="start-a-grid">Start a Grid</h3> <p>A Samza grid usually comprises three different systems: <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html";>YARN</a>, <a href="http://kafka.apache.org/";>Kafka</a>, and <a href="http://zookeeper.apache.org/";>ZooKeeper</a>. The hello-samza project comes with a script called “grid” to help you setup these systems. Start by running:</p> - -<div class="highlight"><pre><code class="bash">bin/grid bootstrap</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">bin/grid bootstrap +</code></pre></div> <p>This command will download, install, and start ZooKeeper, Kafka, and YARN. It will also check out the latest version of Samza and build it. All package files will be put in a sub-directory called “deploy” inside hello-samza’s root folder.</p> <p>If you get a complaint that JAVA_HOME is not set, then you’ll need to set it to the path where Java is installed on your system.</p> @@ -149,21 +155,18 @@ <h3 id="build-a-samza-job-package">Build a Samza Job Package</h3> <p>Before you can run a Samza job, you need to build a package for it. This package is what YARN uses to deploy your jobs on the grid.</p> - -<div class="highlight"><pre><code class="bash">mvn clean package +<div class="highlight"><pre><code class="language-text" data-lang="text">mvn clean package mkdir -p deploy/samza -tar -xvf ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz -C deploy/samza</code></pre></div> - +tar -xvf ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz -C deploy/samza +</code></pre></div> <h3 id="run-a-samza-job">Run a Samza Job</h3> <p>After you’ve built your Samza package, you can start a job on the grid using the run-job.sh script.</p> - -<div class="highlight"><pre><code class="bash">deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties +</code></pre></div> <p>The job will consume a feed of real-time edits from Wikipedia, and produce them to a Kafka topic called “wikipedia-raw”. Give the job a minute to startup, and then tail the Kafka topic:</p> - -<div class="highlight"><pre><code class="bash">deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-raw</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-raw +</code></pre></div> <p>Pretty neat, right? Now, check out the YARN UI again (<a href="http://localhost:8088";>http://localhost:8088</a>). This time around, you’ll see your Samza job is running!</p> <p>If you can not see any output from Kafka consumer, you may have connection problem. Check <a href="../../../learn/tutorials/0.7.0/run-hello-samza-without-internet.html">here</a>.</p> @@ -171,33 +174,28 @@ tar -xvf ./samza-job-package/target/samz <h3 id="generate-wikipedia-statistics">Generate Wikipedia Statistics</h3> <p>Let’s calculate some statistics based on the messages in the wikipedia-raw topic. Start two more jobs:</p> - -<div class="highlight"><pre><code class="bash">deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-parser.properties -deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-stats.properties</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-parser.properties +deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-stats.properties +</code></pre></div> <p>The first job (wikipedia-parser) parses the messages in wikipedia-raw, and extracts information about the size of the edit, who made the change, etc. You can take a look at its output with:</p> - -<div class="highlight"><pre><code class="bash">deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-edits</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-edits +</code></pre></div> <p>The last job (wikipedia-stats) reads messages from the wikipedia-edits topic, and calculates counts, every ten seconds, for all edits that were made during that window. It outputs these counts to the wikipedia-stats topic.</p> - -<div class="highlight"><pre><code class="bash">deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats +</code></pre></div> <p>The messages in the stats topic look like this:</p> - -<div class="highlight"><pre><code class="json"><span class="p">{</span><span class="nt">"is-talk"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">5276</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">13</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">13</span><span class="p">}</span> -<span class="p">{</span><span class="nt">"is-bot-edit"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-talk"</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">4211</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">7</span><span class="p">}</span> -<span class="p">{</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">3180</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span> -<span class="p">{</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">2218</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">{"is-talk":2,"bytes-added":5276,"edits":13,"unique-titles":13} +{"is-bot-edit":1,"is-talk":3,"bytes-added":4211,"edits":30,"unique-titles":30,"is-unpatrolled":1,"is-new":2,"is-minor":7} +{"bytes-added":3180,"edits":19,"unique-titles":19,"is-unpatrolled":1,"is-new":1,"is-minor":3} +{"bytes-added":2218,"edits":18,"unique-titles":18,"is-unpatrolled":2,"is-new":2,"is-minor":3} +</code></pre></div> <p>If you check the YARN UI, again, you’ll see that all three jobs are now listed.</p> <h3 id="shutdown">Shutdown</h3> <p>After you’re done, you can clean everything up using the same grid script.</p> - -<div class="highlight"><pre><code class="bash">bin/grid stop all</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">bin/grid stop all +</code></pre></div> <p>Congratulations! You’ve now setup a local grid that includes YARN, Kafka, and ZooKeeper, and run a Samza job on it. Next up, check out the <a href="/learn/documentation/0.7.0/introduction/background.html">Background</a> and <a href="/learn/documentation/0.7.0/api/overview.html">API Overview</a> pages.</p>
