Modified: incubator/samza/site/learn/documentation/0.7.0/container/samza-container.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/samza-container.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/container/samza-container.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/container/samza-container.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/container/samza-container.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -143,17 +151,16 @@ <h3 id="tasks-and-partitions">Tasks and Partitions</h3> <p>When the container starts, it creates instances of the <a href="../api/overview.html">task class</a> that you’ve written. If the task class implements the <a href="../api/javadocs/org/apache/samza/task/InitableTask.html">InitableTask</a> interface, the SamzaContainer will also call the init() method.</p> - -<div class="highlight"><pre><code class="java"><span class="cm">/** Implement this if you want a callback when your task starts up. */</span> -<span class="kd">public</span> <span class="kd">interface</span> <span class="nc">InitableTask</span> <span class="o">{</span> - <span class="kt">void</span> <span class="nf">init</span><span class="o">(</span><span class="n">Config</span> <span class="n">config</span><span class="o">,</span> <span class="n">TaskContext</span> <span class="n">context</span><span class="o">);</span> -<span class="o">}</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">/** Implement this if you want a callback when your task starts up. */ +public interface InitableTask { + void init(Config config, TaskContext context); +} +</code></pre></div> <p>How many instances of your task class are created depends on the number of partitions in the job’s input streams. If your Samza job has ten partitions, there will be ten instantiations of your task class: one for each partition. The first task instance will receive all messages for partition one, the second instance will receive all messages for partition two, and so on.</p> <p><img src="/img/0.7.0/learn/documentation/container/tasks-and-partitions.svg" alt="Illustration of tasks consuming partitions" class="diagram-large"></p> -<p>The number of partitions in the input streams is determined by the systems from which you are consuming. For example, if your input system is Kafka, you can specify the number of partitions when you create a topic from the command line or using the num.partitions in Kafka’s server properties file.</p> +<p>The number of partitions in the input streams is determined by the systems from which you are consuming. For example, if your input system is Kafka, you can specify the number of partitions when you create a topic.</p> <p>If a Samza job has more than one input stream, the number of task instances for the Samza job is the maximum number of partitions across all input streams. For example, if a Samza job is reading from PageViewEvent (12 partitions), and ServiceMetricEvent (14 partitions), then the Samza job would have 14 task instances (numbered 0 through 13). Task instances 12 and 13 only receive events from ServiceMetricEvent, because there is no corresponding PageViewEvent partition.</p> @@ -173,27 +180,12 @@ <p>If your job has multiple input streams, Samza provides a simple but powerful mechanism for joining data from different streams: each task instance receives messages from one partition of <em>each</em> of the input streams. For example, say you have two input streams, A and B, each with four partitions. Samza creates four task instances to process them, and assigns the partitions as follows:</p> -<table class="table table-condensed table-bordered table-striped"> - <thead> - <tr> - <th>Task instance</th> - <th>Consumes stream partitions</th> - </tr> - </thead> - <tbody> - <tr> - <td>0</td><td>stream A partition 0, stream B partition 0</td> - </tr> - <tr> - <td>1</td><td>stream A partition 1, stream B partition 1</td> - </tr> - <tr> - <td>2</td><td>stream A partition 2, stream B partition 2</td> - </tr> - <tr> - <td>3</td><td>stream A partition 3, stream B partition 3</td> - </tr> - </tbody> +<table class="documentation"> +<tr><th>Task instance</th><th>Consumes stream partitions</th></tr> +<tr><td>0</td><td>stream A partition 0, stream B partition 0</td></tr> +<tr><td>1</td><td>stream A partition 1, stream B partition 1</td></tr> +<tr><td>2</td><td>stream A partition 2, stream B partition 2</td></tr> +<tr><td>3</td><td>stream A partition 3, stream B partition 3</td></tr> </table> <p>Thus, if you want two events in different streams to be processed by the same task instance, you need to ensure they are sent to the same partition number. You can achieve this by using the same partitioning key when <a href="../api/overview.html">sending the messages</a>. Joining streams is discussed in detail in the <a href="state-management.html">state management</a> section.</p>
Modified: incubator/samza/site/learn/documentation/0.7.0/container/serialization.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/serialization.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/container/serialization.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/container/serialization.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/container/serialization.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -132,31 +140,30 @@ </ol> <p>You can use whatever makes sense for your job; Samza doesn’t impose any particular data model or serialization scheme on you. However, the cleanest solution is usually to use Samza’s serde layer. The following configuration example shows how to use it.</p> +<div class="highlight"><pre><code class="language-text" data-lang="text"># Define a system called "kafka" +systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory -<div class="highlight"><pre><code class="jproperties"><span class="c"># Define a system called "kafka"</span> -<span class="na">systems.kafka.samza.factory</span><span class="o">=</span><span class="s">org.apache.samza.system.kafka.KafkaSystemFactory</span> - -<span class="c"># The job is going to consume a topic called "PageViewEvent" from the "kafka" system</span> -<span class="na">task.inputs</span><span class="o">=</span><span class="s">kafka.PageViewEvent</span> - -<span class="c"># Define a serde called "json" which parses/serializes JSON objects</span> -<span class="na">serializers.registry.json.class</span><span class="o">=</span><span class="s">org.apache.samza.serializers.JsonSerdeFactory</span> +# The job is going to consume a topic called "PageViewEvent" from the "kafka" system +task.inputs=kafka.PageViewEvent -<span class="c"># Define a serde called "integer" which encodes an integer as 4 binary bytes (big-endian)</span> -<span class="na">serializers.registry.integer.class</span><span class="o">=</span><span class="s">org.apache.samza.serializers.IntegerSerdeFactory</span> - -<span class="c"># For messages in the "PageViewEvent" topic, the key (the ID of the user viewing the page)</span> -<span class="c"># is encoded as a binary integer, and the message is encoded as JSON.</span> -<span class="na">systems.kafka.streams.PageViewEvent.samza.key.serde</span><span class="o">=</span><span class="s">integer</span> -<span class="na">systems.kafka.streams.PageViewEvent.samza.msg.serde</span><span class="o">=</span><span class="s">json</span> - -<span class="c"># Define a key-value store which stores the most recent page view for each user ID.</span> -<span class="c"># Again, the key is an integer user ID, and the value is JSON.</span> -<span class="na">stores.LastPageViewPerUser.factory</span><span class="o">=</span><span class="s">org.apache.samza.storage.kv.KeyValueStorageEngineFactory</span> -<span class="na">stores.LastPageViewPerUser.changelog</span><span class="o">=</span><span class="s">kafka.last-page-view-per-user</span> -<span class="na">stores.LastPageViewPerUser.key.serde</span><span class="o">=</span><span class="s">integer</span> -<span class="na">stores.LastPageViewPerUser.msg.serde</span><span class="o">=</span><span class="s">json</span></code></pre></div> +# Define a serde called "json" which parses/serializes JSON objects +serializers.registry.json.class=org.apache.samza.serializers.JsonSerdeFactory +# Define a serde called "integer" which encodes an integer as 4 binary bytes (big-endian) +serializers.registry.integer.class=org.apache.samza.serializers.IntegerSerdeFactory + +# For messages in the "PageViewEvent" topic, the key (the ID of the user viewing the page) +# is encoded as a binary integer, and the message is encoded as JSON. +systems.kafka.streams.PageViewEvent.samza.key.serde=integer +systems.kafka.streams.PageViewEvent.samza.msg.serde=json + +# Define a key-value store which stores the most recent page view for each user ID. +# Again, the key is an integer user ID, and the value is JSON. +stores.LastPageViewPerUser.factory=org.apache.samza.storage.kv.KeyValueStorageEngineFactory +stores.LastPageViewPerUser.changelog=kafka.last-page-view-per-user +stores.LastPageViewPerUser.key.serde=integer +stores.LastPageViewPerUser.msg.serde=json +</code></pre></div> <p>Each serde is defined with a factory class. Samza comes with several builtin serdes for UTF-8 strings, binary-encoded integers, JSON (requires the samza-serializers dependency) and more. You can also create your own serializer by implementing the <a href="../api/javadocs/org/apache/samza/serializers/SerdeFactory.html">SerdeFactory</a> interface.</p> <p>The name you give to a serde (such as “json” and “integer” in the example above) is only for convenience in your job configuration; you can choose whatever name you like. For each stream and each state store, you can use the serde name to declare how messages should be serialized and deserialized.</p> Modified: incubator/samza/site/learn/documentation/0.7.0/container/state-management.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/state-management.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/container/state-management.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/container/state-management.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/container/state-management.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -245,49 +253,46 @@ <p>Samza includes an additional in-memory caching layer in front of LevelDB, which avoids the cost of deserialization for frequently-accessed objects and batches writes. If the same key is updated multiple times in quick succession, the batching coalesces those updates into a single write. The writes are flushed to the changelog when a task <a href="checkpointing.html">commits</a>.</p> <p>To use a key-value store in your job, add the following to your job config:</p> +<div class="highlight"><pre><code class="language-text" data-lang="text"># Use the key-value store implementation for a store called "my-store" +stores.my-store.factory=org.apache.samza.storage.kv.KeyValueStorageEngineFactory -<div class="highlight"><pre><code class="jproperties"><span class="c"># Use the key-value store implementation for a store called "my-store"</span> -<span class="na">stores.my-store.factory</span><span class="o">=</span><span class="s">org.apache.samza.storage.kv.KeyValueStorageEngineFactory</span> - -<span class="c"># Use the Kafka topic "my-store-changelog" as the changelog stream for this store.</span> -<span class="c"># This enables automatic recovery of the store after a failure. If you don't</span> -<span class="c"># configure this, no changelog stream will be generated.</span> -<span class="na">stores.my-store.changelog</span><span class="o">=</span><span class="s">kafka.my-store-changelog</span> - -<span class="c"># Encode keys and values in the store as UTF-8 strings.</span> -<span class="na">serializers.registry.string.class</span><span class="o">=</span><span class="s">org.apache.samza.serializers.StringSerdeFactory</span> -<span class="na">stores.my-store.key.serde</span><span class="o">=</span><span class="s">string</span> -<span class="na">stores.my-store.msg.serde</span><span class="o">=</span><span class="s">string</span></code></pre></div> - +# Use the Kafka topic "my-store-changelog" as the changelog stream for this store. +# This enables automatic recovery of the store after a failure. If you don't +# configure this, no changelog stream will be generated. +stores.my-store.changelog=kafka.my-store-changelog + +# Encode keys and values in the store as UTF-8 strings. +serializers.registry.string.class=org.apache.samza.serializers.StringSerdeFactory +stores.my-store.key.serde=string +stores.my-store.msg.serde=string +</code></pre></div> <p>See the <a href="serialization.html">serialization section</a> for more information on the <em>serde</em> options.</p> <p>Here is a simple example that writes every incoming message to the store:</p> +<div class="highlight"><pre><code class="language-text" data-lang="text">public class MyStatefulTask implements StreamTask, InitableTask { + private KeyValueStore<String, String> store; -<div class="highlight"><pre><code class="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyStatefulTask</span> <span class="kd">implements</span> <span class="n">StreamTask</span><span class="o">,</span> <span class="n">InitableTask</span> <span class="o">{</span> - <span class="kd">private</span> <span class="n">KeyValueStore</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">></span> <span class="n">store</span><span class="o">;</span> - - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">init</span><span class="o">(</span><span class="n">Config</span> <span class="n">config</span><span class="o">,</span> <span class="n">TaskContext</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span> - <span class="k">this</span><span class="o">.</span><span class="na">store</span> <span class="o">=</span> <span class="o">(</span><span class="n">KeyValueStore</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">>)</span> <span class="n">context</span><span class="o">.</span><span class="na">getStore</span><span class="o">(</span><span class="s">"my-store"</span><span class="o">);</span> - <span class="o">}</span> - - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="n">IncomingMessageEnvelope</span> <span class="n">envelope</span><span class="o">,</span> - <span class="n">MessageCollector</span> <span class="n">collector</span><span class="o">,</span> - <span class="n">TaskCoordinator</span> <span class="n">coordinator</span><span class="o">)</span> <span class="o">{</span> - <span class="n">store</span><span class="o">.</span><span class="na">put</span><span class="o">((</span><span class="n">String</span><span class="o">)</span> <span class="n">envelope</span><span class="o">.</span><span class="na">getKey</span><span class="o">(),</span> <span class="o">(</span><span class="n">String</span><span class="o">)</span> <span class="n">envelope</span><span class="o">.</span><span class="na">getMessage</span><span class="o">());</span> - <span class="o">}</span> -<span class="o">}</span></code></pre></div> - + public void init(Config config, TaskContext context) { + this.store = (KeyValueStore<String, String>) context.getStore("my-store"); + } + + public void process(IncomingMessageEnvelope envelope, + MessageCollector collector, + TaskCoordinator coordinator) { + store.put((String) envelope.getKey(), (String) envelope.getMessage()); + } +} +</code></pre></div> <p>Here is the complete key-value store API:</p> - -<div class="highlight"><pre><code class="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">KeyValueStore</span><span class="o"><</span><span class="n">K</span><span class="o">,</span> <span class="n">V</span><span class="o">></span> <span class="o">{</span> - <span class="n">V</span> <span class="nf">get</span><span class="o">(</span><span class="n">K</span> <span class="n">key</span><span class="o">);</span> - <span class="kt">void</span> <span class="nf">put</span><span class="o">(</span><span class="n">K</span> <span class="n">key</span><span class="o">,</span> <span class="n">V</span> <span class="n">value</span><span class="o">);</span> - <span class="kt">void</span> <span class="nf">putAll</span><span class="o">(</span><span class="n">List</span><span class="o"><</span><span class="n">Entry</span><span class="o"><</span><span class="n">K</span><span class="o">,</span><span class="n">V</span><span class="o">>></span> <span class="n">entries</span><span class="o">);</span> - <span class="kt">void</span> <span class="nf">delete</span><span class="o">(</span><span class="n">K</span> <span class="n">key</span><span class="o">);</span> - <span class="n">KeyValueIterator</span><span class="o"><</span><span class="n">K</span><span class="o">,</span><span class="n">V</span><span class="o">></span> <span class="n">range</span><span class="o">(</span><span class="n">K</span> <span class="n">from</span><span class="o">,</span> <span class="n">K</span> <span class="n">to</span><span class="o">);</span> - <span class="n">KeyValueIterator</span><span class="o"><</span><span class="n">K</span><span class="o">,</span><span class="n">V</span><span class="o">></span> <span class="n">all</span><span class="o">();</span> -<span class="o">}</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">public interface KeyValueStore<K, V> { + V get(K key); + void put(K key, V value); + void putAll(List<Entry<K,V>> entries); + void delete(K key); + KeyValueIterator<K,V> range(K from, K to); + KeyValueIterator<K,V> all(); +} +</code></pre></div> <p>Additional configuration properties for the key-value store are documented in the <a href="../jobs/configuration-table.html#keyvalue">configuration reference</a>.</p> <h3 id="implementing-common-use-cases-with-the-key-value-store">Implementing common use cases with the key-value store</h3> Modified: incubator/samza/site/learn/documentation/0.7.0/container/streams.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/streams.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/container/streams.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/container/streams.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/container/streams.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -124,49 +132,48 @@ --> <p>The <a href="samza-container.html">samza container</a> reads and writes messages using the <a href="../api/javadocs/org/apache/samza/system/SystemConsumer.html">SystemConsumer</a> and <a href="../api/javadocs/org/apache/samza/system/SystemProducer.html">SystemProducer</a> interfaces. You can integrate any message broker with Samza by implementing these two interfaces.</p> +<div class="highlight"><pre><code class="language-text" data-lang="text">public interface SystemConsumer { + void start(); -<div class="highlight"><pre><code class="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">SystemConsumer</span> <span class="o">{</span> - <span class="kt">void</span> <span class="nf">start</span><span class="o">();</span> - - <span class="kt">void</span> <span class="nf">stop</span><span class="o">();</span> + void stop(); - <span class="kt">void</span> <span class="nf">register</span><span class="o">(</span> - <span class="n">SystemStreamPartition</span> <span class="n">systemStreamPartition</span><span class="o">,</span> - <span class="n">String</span> <span class="n">lastReadOffset</span><span class="o">);</span> + void register( + SystemStreamPartition systemStreamPartition, + String lastReadOffset); - <span class="n">List</span><span class="o"><</span><span class="n">IncomingMessageEnvelope</span><span class="o">></span> <span class="nf">poll</span><span class="o">(</span> - <span class="n">Map</span><span class="o"><</span><span class="n">SystemStreamPartition</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">systemStreamPartitions</span><span class="o">,</span> - <span class="kt">long</span> <span class="n">timeout</span><span class="o">)</span> - <span class="kd">throws</span> <span class="n">InterruptedException</span><span class="o">;</span> -<span class="o">}</span> + List<IncomingMessageEnvelope> poll( + Map<SystemStreamPartition, Integer> systemStreamPartitions, + long timeout) + throws InterruptedException; +} -<span class="kd">public</span> <span class="kd">class</span> <span class="nc">IncomingMessageEnvelope</span> <span class="o">{</span> - <span class="kd">public</span> <span class="n">Object</span> <span class="nf">getMessage</span><span class="o">()</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span> +public class IncomingMessageEnvelope { + public Object getMessage() { ... } - <span class="kd">public</span> <span class="n">Object</span> <span class="nf">getKey</span><span class="o">()</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span> + public Object getKey() { ... } - <span class="kd">public</span> <span class="n">SystemStreamPartition</span> <span class="nf">getSystemStreamPartition</span><span class="o">()</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span> -<span class="o">}</span> + public SystemStreamPartition getSystemStreamPartition() { ... } +} -<span class="kd">public</span> <span class="kd">interface</span> <span class="nc">SystemProducer</span> <span class="o">{</span> - <span class="kt">void</span> <span class="nf">start</span><span class="o">();</span> +public interface SystemProducer { + void start(); - <span class="kt">void</span> <span class="nf">stop</span><span class="o">();</span> + void stop(); - <span class="kt">void</span> <span class="nf">register</span><span class="o">(</span><span class="n">String</span> <span class="n">source</span><span class="o">);</span> + void register(String source); - <span class="kt">void</span> <span class="nf">send</span><span class="o">(</span><span class="n">String</span> <span class="n">source</span><span class="o">,</span> <span class="n">OutgoingMessageEnvelope</span> <span class="n">envelope</span><span class="o">);</span> + void send(String source, OutgoingMessageEnvelope envelope); - <span class="kt">void</span> <span class="nf">flush</span><span class="o">(</span><span class="n">String</span> <span class="n">source</span><span class="o">);</span> -<span class="o">}</span> + void flush(String source); +} -<span class="kd">public</span> <span class="kd">class</span> <span class="nc">OutgoingMessageEnvelope</span> <span class="o">{</span> - <span class="o">...</span> - <span class="kd">public</span> <span class="n">Object</span> <span class="nf">getKey</span><span class="o">()</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span> - - <span class="kd">public</span> <span class="n">Object</span> <span class="nf">getMessage</span><span class="o">()</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span> -<span class="o">}</span></code></pre></div> +public class OutgoingMessageEnvelope { + ... + public Object getKey() { ... } + public Object getMessage() { ... } +} +</code></pre></div> <p>Out of the box, Samza supports Kafka (KafkaSystemConsumer and KafkaSystemProducer). However, any message bus system can be plugged in, as long as it can provide the semantics required by Samza, as described in the <a href="../api/javadocs/org/apache/samza/system/SystemConsumer.html">javadoc</a>.</p> <p>SystemConsumers and SystemProducers may read and write messages of any data type. It’s ok if they only support byte arrays — Samza has a separate <a href="serialization.html">serialization layer</a> which converts to and from objects that application code can use. Samza does not prescribe any particular data model or serialization format.</p> @@ -184,18 +191,16 @@ <p>When a Samza container has several incoming messages on different stream partitions, how does it decide which to process first? The behavior is determined by a <a href="../api/javadocs/org/apache/samza/system/chooser/MessageChooser.html">MessageChooser</a>. The default chooser is RoundRobinChooser, but you can override it by implementing a custom chooser.</p> <p>To plug in your own message chooser, you need to implement the <a href="../api/javadocs/org/apache/samza/system/chooser/MessageChooserFactory.html">MessageChooserFactory</a> interface, and set the “task.chooser.class” configuration to the fully-qualified class name of your implementation:</p> - -<div class="highlight"><pre><code class="jproperties"><span class="na">task.chooser.class</span><span class="o">=</span><span class="s">com.example.samza.YourMessageChooserFactory</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">task.chooser.class=com.example.samza.YourMessageChooserFactory +</code></pre></div> <h4 id="prioritizing-input-streams">Prioritizing input streams</h4> <p>There are certain times when messages from one stream should be processed with higher priority than messages from another stream. For example, some Samza jobs consume two streams: one stream is fed by a real-time system and the other stream is fed by a batch system. In this case, it’s useful to prioritize the real-time stream over the batch stream, so that the real-time processing doesn’t slow down if there is a sudden burst of data on the batch stream.</p> <p>Samza provides a mechanism to prioritize one stream over another by setting this configuration parameter: systems.<system>.streams.<stream>.samza.priority=<number>. For example:</p> - -<div class="highlight"><pre><code class="jproperties"><span class="na">systems.kafka.streams.my-real-time-stream.samza.priority</span><span class="o">=</span><span class="s">2</span> -<span class="na">systems.kafka.streams.my-batch-stream.samza.priority</span><span class="o">=</span><span class="s">1</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">systems.kafka.streams.my-real-time-stream.samza.priority=2 +systems.kafka.streams.my-batch-stream.samza.priority=1 +</code></pre></div> <p>This declares that my-real-time-stream’s messages should be processed with higher priority than my-batch-stream’s messages. If my-real-time-stream has any messages available, they are processed first. Only if there are no messages currently waiting on my-real-time-stream, the Samza job continues processing my-batch-stream.</p> <p>Each priority level gets its own MessageChooser. It is valid to define two streams with the same priority. If messages are available from two streams at the same priority level, it’s up to the MessageChooser for that priority level to decide which message should be processed first.</p> @@ -211,11 +216,10 @@ <p>Another difference between a bootstrap stream and a high-priority stream is that the bootstrap stream’s special treatment is temporary: when it has been fully consumed (we say it has “caught up”), its priority drops to be the same as all the other input streams.</p> <p>To configure a stream called “my-bootstrap-stream” to be a fully-consumed bootstrap stream, use the following settings:</p> - -<div class="highlight"><pre><code class="jproperties"><span class="na">systems.kafka.streams.my-bootstrap-stream.samza.bootstrap</span><span class="o">=</span><span class="s">true</span> -<span class="na">systems.kafka.streams.my-bootstrap-stream.samza.reset.offset</span><span class="o">=</span><span class="s">true</span> -<span class="na">systems.kafka.streams.my-bootstrap-stream.samza.offset.default</span><span class="o">=</span><span class="s">oldest</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">systems.kafka.streams.my-bootstrap-stream.samza.bootstrap=true +systems.kafka.streams.my-bootstrap-stream.samza.reset.offset=true +systems.kafka.streams.my-bootstrap-stream.samza.offset.default=oldest +</code></pre></div> <p>The bootstrap=true parameter enables the bootstrap behavior (prioritization over other streams). The combination of reset.offset=true and offset.default=oldest tells Samza to always start reading the stream from the oldest offset, every time a container starts up (rather than starting to read from the most recent checkpoint).</p> <p>It is valid to define multiple bootstrap streams. In this case, the order in which they are bootstrapped is determined by the priority.</p> @@ -225,9 +229,8 @@ <p>In some cases, you can improve performance by consuming several messages from the same stream partition in sequence. Samza supports this mode of operation, called <em>batching</em>.</p> <p>For example, if you want to read 100 messages in a row from each stream partition (regardless of the MessageChooser), you can use this configuration parameter:</p> - -<div class="highlight"><pre><code class="jproperties"><span class="na">task.consumer.batch.size</span><span class="o">=</span><span class="s">100</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">task.consumer.batch.size=100 +</code></pre></div> <p>With this setting, Samza tries to read a message from the most recently used <a href="../api/javadocs/org/apache/samza/system/SystemStreamPartition.html">SystemStreamPartition</a>. This behavior continues either until no more messages are available for that SystemStreamPartition, or until the batch size has been reached. When that happens, Samza defers to the MessageChooser to determine the next message to process. It then again tries to continue consume from the chosen message’s SystemStreamPartition until the batch size is reached.</p> <h2 id="serialization-»"><a href="serialization.html">Serialization »</a></h2> Modified: incubator/samza/site/learn/documentation/0.7.0/container/windowing.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/windowing.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/container/windowing.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/container/windowing.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/container/windowing.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -126,34 +134,32 @@ <p>Sometimes a stream processing job needs to do something in regular time intervals, regardless of how many incoming messages the job is processing. For example, say you want to report the number of page views per minute. To do this, you increment a counter every time you see a page view event. Once per minute, you send the current counter value to an output stream and reset the counter to zero.</p> <p>Samza’s <em>windowing</em> feature provides a way for tasks to do something in regular time intervals, for example once per minute. To enable windowing, you just need to set one property in your job configuration:</p> - -<div class="highlight"><pre><code class="jproperties"><span class="c"># Call the window() method every 60 seconds</span> -<span class="na">task.window.ms</span><span class="o">=</span><span class="s">60000</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text"># Call the window() method every 60 seconds +task.window.ms=60000 +</code></pre></div> <p>Next, your stream task needs to implement the <a href="../api/javadocs/org/apache/samza/task/WindowableTask.html">WindowableTask</a> interface. This interface defines a window() method which is called by Samza in the regular interval that you configured.</p> <p>For example, this is how you would implement a basic per-minute event counter:</p> +<div class="highlight"><pre><code class="language-text" data-lang="text">public class EventCounterTask implements StreamTask, WindowableTask { -<div class="highlight"><pre><code class="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">EventCounterTask</span> <span class="kd">implements</span> <span class="n">StreamTask</span><span class="o">,</span> <span class="n">WindowableTask</span> <span class="o">{</span> - - <span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">SystemStream</span> <span class="n">OUTPUT_STREAM</span> <span class="o">=</span> - <span class="k">new</span> <span class="nf">SystemStream</span><span class="o">(</span><span class="s">"kafka"</span><span class="o">,</span> <span class="s">"events-per-minute"</span><span class="o">);</span> - - <span class="kd">private</span> <span class="kt">int</span> <span class="n">eventsSeen</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> + public static final SystemStream OUTPUT_STREAM = + new SystemStream("kafka", "events-per-minute"); - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="n">IncomingMessageEnvelope</span> <span class="n">envelope</span><span class="o">,</span> - <span class="n">MessageCollector</span> <span class="n">collector</span><span class="o">,</span> - <span class="n">TaskCoordinator</span> <span class="n">coordinator</span><span class="o">)</span> <span class="o">{</span> - <span class="n">eventsSeen</span><span class="o">++;</span> - <span class="o">}</span> - - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">window</span><span class="o">(</span><span class="n">MessageCollector</span> <span class="n">collector</span><span class="o">,</span> - <span class="n">TaskCoordinator</span> <span class="n">coordinator</span><span class="o">)</span> <span class="o">{</span> - <span class="n">collector</span><span class="o">.</span><span class="na">send</span><span class="o">(</span><span class="k">new</span> <span class="n">OutgoingMessageEnvelope</span><span class="o">(</span><span class="n">OUTPUT_STREAM</span><span class="o">,</span> <span class="n">eventsSeen</span><span class="o">));</span> - <span class="n">eventsSeen</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> - <span class="o">}</span> -<span class="o">}</span></code></pre></div> + private int eventsSeen = 0; + public void process(IncomingMessageEnvelope envelope, + MessageCollector collector, + TaskCoordinator coordinator) { + eventsSeen++; + } + + public void window(MessageCollector collector, + TaskCoordinator coordinator) { + collector.send(new OutgoingMessageEnvelope(OUTPUT_STREAM, eventsSeen)); + eventsSeen = 0; + } +} +</code></pre></div> <p>If you need to send messages to output streams, you can use the <a href="../api/javadocs/org/apache/samza/task/MessageCollector.html">MessageCollector</a> object passed to the window() method. Please only use that MessageCollector object for sending messages, and don’t use it outside of the call to window().</p> <p>Note that Samza uses <a href="event-loop.html">single-threaded execution</a>, so the window() call can never happen concurrently with a process() call. This has the advantage that you don’t need to worry about thread safety in your code (no need to synchronize anything), but the downside that the window() call may be delayed if your process() method takes a long time to return.</p> Modified: incubator/samza/site/learn/documentation/0.7.0/index.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/index.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/index.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/index.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/index.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -137,7 +145,6 @@ <li><a href="comparisons/introduction.html">Introduction</a></li> <li><a href="comparisons/mupd8.html">MUPD8</a></li> <li><a href="comparisons/storm.html">Storm</a></li> - <li><a href="comparisons/spark-streaming.html">Spark Streaming</a></li> <!-- TODO comparisons pages <li><a href="comparisons/aurora.html">Aurora</a></li> <li><a href="comparisons/jms.html">JMS</a></li> Modified: incubator/samza/site/learn/documentation/0.7.0/introduction/architecture.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/introduction/architecture.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/introduction/architecture.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/introduction/architecture.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/introduction/architecture.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -202,9 +210,8 @@ <h3 id="example">Example</h3> <p>Let’s take a look at a real example: suppose we want to count the number of page views. In SQL, you would write something like:</p> - -<div class="highlight"><pre><code class="sql"><span class="k">SELECT</span> <span class="n">user_id</span><span class="p">,</span> <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">PageViewEvent</span> <span class="k">GROUP</span> <span class="k">BY</span> <span class="n">user_id</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">SELECT user_id, COUNT(*) FROM PageViewEvent GROUP BY user_id +</code></pre></div> <p>Although Samza doesn’t support SQL right now, the idea is the same. Two jobs are required to calculate this query: one to group messages by user ID, and the other to do the counting.</p> <p>In the first job, the grouping is done by sending all messages with the same user ID to the same partition of an intermediate topic. You can do this by using the user ID as key of the messages that are emitted by the first job, and this key is mapped to one of the intermediate topic’s partitions (usually by taking a hash of the key mod the number of partitions). The second job consumes the intermediate topic. Each task in the second job consumes one partition of the intermediate topic, i.e. all the messages for a subset of user IDs. The task has a counter for each user ID in its partition, and the appropriate counter is incremented every time the task receives a message with a particular user ID.</p> Modified: incubator/samza/site/learn/documentation/0.7.0/introduction/background.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/introduction/background.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/introduction/background.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/introduction/background.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/introduction/background.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> Modified: incubator/samza/site/learn/documentation/0.7.0/introduction/concepts.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/introduction/concepts.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/introduction/concepts.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/introduction/concepts.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/introduction/concepts.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/configuration-table.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/configuration-table.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/jobs/configuration-table.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/jobs/configuration-table.html Fri Aug 15 05:36:35 2014 @@ -357,41 +357,6 @@ </tr> <tr> - <td class="property" id="task-drop-deserialization-errors">task.drop.deserialization.errors</td> - <td class="default"></td> - <td class="description"> - This property is to define how the system deals with deserialization failure situation. If set to true, the system will - skip the error messages and keep running. If set to false, the system with throw exceptions and fail the container. Default - is false. - </td> - </tr> - - <tr> - <td class="property" id="task-drop-serialization-errors">task.drop.serialization.errors</td> - <td class="default"></td> - <td class="description"> - This property is to define how the system deals with serialization failure situation. If set to true, the system will - drop the error messages and keep running. If set to false, the system with throw exceptions and fail the container. Default - is false. - </td> - </tr> - - <tr> - <td class="property" id="task-poll-interval-ms">task.poll.interval.ms</td> - <td class="default"></td> - <td class="description"> - Samza's container polls for more messages under two conditions. The first condition arises when there are simply no remaining - buffered messages to process for any input SystemStreamPartition. The second condition arises when some input - SystemStreamPartitions have empty buffers, but some do not. In the latter case, a polling interval is defined to determine how - often to refresh the empty SystemStreamPartition buffers. By default, this interval is 50ms, which means that any empty - SystemStreamPartition buffer will be refreshed at least every 50ms. A higher value here means that empty SystemStreamPartitions - will be refreshed less often, which means more latency is introduced, but less CPU and network will be used. Decreasing this - value means that empty SystemStreamPartitions are refreshed more frequently, thereby introducing less latency, but increasing - CPU and network utilization. - </td> - </tr> - - <tr> <th colspan="3" class="section" id="streams"><a href="../container/streams.html">Systems (input and output streams)</a></th> </tr> Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/configuration.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/configuration.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/jobs/configuration.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/jobs/configuration.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/jobs/configuration.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -124,24 +132,23 @@ --> <p>All Samza jobs have a configuration file that defines the job. A very basic configuration file looks like this:</p> - -<div class="highlight"><pre><code class="jproperties"><span class="c"># Job</span> -<span class="na">job.factory.class</span><span class="o">=</span><span class="s">samza.job.local.LocalJobFactory</span> -<span class="na">job.name</span><span class="o">=</span><span class="s">hello-world</span> - -<span class="c"># Task</span> -<span class="na">task.class</span><span class="o">=</span><span class="s">samza.task.example.MyJavaStreamerTask</span> -<span class="na">task.inputs</span><span class="o">=</span><span class="s">example-system.example-stream</span> - -<span class="c"># Serializers</span> -<span class="na">serializers.registry.json.class</span><span class="o">=</span><span class="s">org.apache.samza.serializers.JsonSerdeFactory</span> -<span class="na">serializers.registry.string.class</span><span class="o">=</span><span class="s">org.apache.samza.serializers.StringSerdeFactory</span> - -<span class="c"># Systems</span> -<span class="na">systems.example-system.samza.factory</span><span class="o">=</span><span class="s">samza.stream.example.ExampleConsumerFactory</span> -<span class="na">systems.example-system.samza.key.serde</span><span class="o">=</span><span class="s">string</span> -<span class="na">systems.example-system.samza.msg.serde</span><span class="o">=</span><span class="s">json</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text"># Job +job.factory.class=samza.job.local.LocalJobFactory +job.name=hello-world + +# Task +task.class=samza.task.example.MyJavaStreamerTask +task.inputs=example-system.example-stream + +# Serializers +serializers.registry.json.class=org.apache.samza.serializers.JsonSerdeFactory +serializers.registry.string.class=org.apache.samza.serializers.StringSerdeFactory + +# Systems +systems.example-system.samza.factory=samza.stream.example.ExampleConsumerFactory +systems.example-system.samza.key.serde=string +systems.example-system.samza.msg.serde=json +</code></pre></div> <p>There are four major sections to a configuration file:</p> <ol> @@ -156,10 +163,10 @@ <p>Configuration keys that absolutely must be defined for a Samza job are:</p> <ul> -<li><code>job.factory.class</code></li> -<li><code>job.name</code></li> -<li><code>task.class</code></li> -<li><code>task.inputs</code></li> +<li>job.factory.class</li> +<li>job.name</li> +<li>task.class</li> +<li>task.inputs</li> </ul> <h3 id="configuration-keys">Configuration Keys</h3> Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/job-runner.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/job-runner.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/jobs/job-runner.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/jobs/job-runner.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/jobs/job-runner.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -124,33 +132,30 @@ --> <p>Samza jobs are started using a script called run-job.sh.</p> - -<div class="highlight"><pre><code class="bash">samza-example/target/bin/run-job.sh <span class="se">\</span> - --config-factory<span class="o">=</span>samza.config.factories.PropertiesConfigFactory <span class="se">\</span> - --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/config/hello-world.properties</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">samza-example/target/bin/run-job.sh \ + --config-factory=samza.config.factories.PropertiesConfigFactory \ + --config-path=file://$PWD/config/hello-world.properties +</code></pre></div> <p>You provide two parameters to the run-job.sh script. One is the config location, and the other is a factory class that is used to read your configuration file. The run-job.sh script is actually executing a Samza class called JobRunner. The JobRunner uses your ConfigFactory to get a Config object from the config path.</p> - -<div class="highlight"><pre><code class="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">ConfigFactory</span> <span class="o">{</span> - <span class="n">Config</span> <span class="nf">getConfig</span><span class="o">(</span><span class="n">URI</span> <span class="n">configUri</span><span class="o">);</span> -<span class="o">}</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">public interface ConfigFactory { + Config getConfig(URI configUri); +} +</code></pre></div> <p>The Config object is just a wrapper around Map<String, String>, with some nice helper methods. Out of the box, Samza ships with the PropertiesConfigFactory, but developers can implement any kind of ConfigFactory they wish.</p> <p>Once the JobRunner gets your configuration, it gives your configuration to the StreamJobFactory class defined by the “job.factory” property. Samza ships with two job factory implementations: LocalJobFactory and YarnJobFactory. The StreamJobFactory’s responsibility is to give the JobRunner a job that it can run.</p> +<div class="highlight"><pre><code class="language-text" data-lang="text">public interface StreamJob { + StreamJob submit(); -<div class="highlight"><pre><code class="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">StreamJob</span> <span class="o">{</span> - <span class="n">StreamJob</span> <span class="nf">submit</span><span class="o">();</span> - - <span class="n">StreamJob</span> <span class="nf">kill</span><span class="o">();</span> - - <span class="n">ApplicationStatus</span> <span class="nf">waitForFinish</span><span class="o">(</span><span class="kt">long</span> <span class="n">timeoutMs</span><span class="o">);</span> + StreamJob kill(); - <span class="n">ApplicationStatus</span> <span class="nf">waitForStatus</span><span class="o">(</span><span class="n">ApplicationStatus</span> <span class="n">status</span><span class="o">,</span> <span class="kt">long</span> <span class="n">timeoutMs</span><span class="o">);</span> + ApplicationStatus waitForFinish(long timeoutMs); - <span class="n">ApplicationStatus</span> <span class="nf">getStatus</span><span class="o">();</span> -<span class="o">}</span></code></pre></div> + ApplicationStatus waitForStatus(ApplicationStatus status, long timeoutMs); + ApplicationStatus getStatus(); +} +</code></pre></div> <p>Once the JobRunner gets a job, it calls submit() on the job. This method is what tells the StreamJob implementation to start the SamzaContainer. In the case of LocalJobRunner, it uses a run-container.sh script to execute the SamzaContainer in a separate process, which will start one SamzaContainer locally on the machine that you ran run-job.sh on.</p> <p>This flow differs slightly when you use YARN, but we’ll get to that later.</p> Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/logging.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/logging.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/jobs/logging.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/jobs/logging.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/jobs/logging.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content"> @@ -128,57 +136,52 @@ <h3 id="log4j">Log4j</h3> <p>The <a href="/startup/hello-samza/0.7.0">hello-samza</a> project shows how to use <a href="http://logging.apache.org/log4j/1.2/";>log4j</a> with Samza. To turn on log4j logging, you just need to make sure slf4j-log4j12 is in your SamzaContainer’s classpath. In Maven, this can be done by adding the following dependency to your Samza package project.</p> - -<div class="highlight"><pre><code class="xml"><span class="nt"><dependency></span> - <span class="nt"><groupId></span>org.slf4j<span class="nt"></groupId></span> - <span class="nt"><artifactId></span>slf4j-log4j12<span class="nt"></artifactId></span> - <span class="nt"><scope></span>runtime<span class="nt"></scope></span> - <span class="nt"><version></span>1.6.2<span class="nt"></version></span> -<span class="nt"></dependency></span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text"><dependency> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + <scope>runtime</scope> + <version>1.6.2</version> +</dependency> +</code></pre></div> <p>If you’re not using Maven, just make sure that slf4j-log4j12 ends up in your Samza package’s lib directory.</p> <h4 id="log4j-configuration">Log4j configuration</h4> <p>Samza’s <a href="packaging.html">run-class.sh</a> script will automatically set the following setting if log4j.xml exists in your <a href="packaging.html">Samza package’s</a> lib directory.</p> - -<div class="highlight"><pre><code class="bash">-Dlog4j.configuration<span class="o">=</span>file:<span class="nv">$base_dir</span>/lib/log4j.xml</code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">-Dlog4j.configuration=file:$base_dir/lib/log4j.xml +</code></pre></div> <p>The <a href="packaging.html">run-class.sh</a> script will also set the following Java system properties:</p> - -<div class="highlight"><pre><code class="bash">-Dsamza.log.dir<span class="o">=</span><span class="nv">$SAMZA_LOG_DIR</span> -Dsamza.container.name<span class="o">=</span><span class="nv">$SAMZA_CONTAINER_NAME</span><span class="o">=</span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text">-Dsamza.log.dir=$SAMZA_LOG_DIR -Dsamza.container.name=$SAMZA_CONTAINER_NAME +</code></pre></div> <p>These settings are very useful if you’re using a file-based appender. For example, you can use a daily rolling appender by configuring log4j.xml like this:</p> - -<div class="highlight"><pre><code class="xml"><span class="nt"><appender</span> <span class="na">name=</span><span class="s">"RollingAppender"</span> <span class="na">class=</span><span class="s">"org.apache.log4j.DailyRollingFileAppender"</span><span class="nt">></span> - <span class="nt"><param</span> <span class="na">name=</span><span class="s">"File"</span> <span class="na">value=</span><span class="s">"${samza.log.dir}/${samza.container.name}.log"</span> <span class="nt">/></span> - <span class="nt"><param</span> <span class="na">name=</span><span class="s">"DatePattern"</span> <span class="na">value=</span><span class="s">"'.'yyyy-MM-dd"</span> <span class="nt">/></span> - <span class="nt"><layout</span> <span class="na">class=</span><span class="s">"org.apache.log4j.PatternLayout"</span><span class="nt">></span> - <span class="nt"><param</span> <span class="na">name=</span><span class="s">"ConversionPattern"</span> <span class="na">value=</span><span class="s">"%d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n"</span> <span class="nt">/></span> - <span class="nt"></layout></span> -<span class="nt"></appender></span></code></pre></div> - +<div class="highlight"><pre><code class="language-text" data-lang="text"><appender name="RollingAppender" class="org.apache.log4j.DailyRollingFileAppender"> + <param name="File" value="${samza.log.dir}/${samza.container.name}.log" /> + <param name="DatePattern" value="'.'yyyy-MM-dd" /> + <layout class="org.apache.log4j.PatternLayout"> + <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n" /> + </layout> +</appender> +</code></pre></div> <p>Setting up a file-based appender is recommended as a better alternative to using standard out. Standard out log files (see below) don’t roll, and can get quite large if used for logging.</p> -<p><strong>NOTE:</strong> If you use the <code>task.opts</code> configuration property, the log configuration is disrupted. This is a known bug; please see <a href="https://issues.apache.org/jira/browse/SAMZA-109";>SAMZA-109</a> for a workaround.</p> +<p><strong>NOTE:</strong> If you use the task.opts configuration property, the log configuration is disrupted. This is a known bug; please see <a href="https://issues.apache.org/jira/browse/SAMZA-109";>SAMZA-109</a> for a workaround.</p> <h3 id="log-directory">Log Directory</h3> -<p>Samza will look for the <code>SAMZA_LOG_DIR</code> environment variable when it executes. If this variable is defined, all logs will be written to this directory. If the environment variable is empty, or not defined, then Samza will use /tmp. This environment variable can also be referenced inside log4j.xml files (see above).</p> +<p>Samza will look for the <em>SAMZA</em>_<em>LOG</em>_<em>DIR</em> environment variable when it executes. If this variable is defined, all logs will be written to this directory. If the environment variable is empty, or not defined, then Samza will use /tmp. This environment variable can also be referenced inside log4j.xml files (see above).</p> <h3 id="garbage-collection-logging">Garbage Collection Logging</h3> -<p>Samza’s will automatically set the following garbage collection logging setting, and will output it to <code>$SAMZA_LOG_DIR/gc.log</code>.</p> - -<div class="highlight"><pre><code class="bash">-XX:+PrintGCDateStamps -Xloggc:<span class="nv">$SAMZA_LOG_DIR</span>/gc.log</code></pre></div> - +<p>Samza’s will automatically set the following garbage collection logging setting, and will output it to <em>$SAMZA</em>_<em>LOG</em>_<em>DIR</em>/gc.log.</p> +<div class="highlight"><pre><code class="language-text" data-lang="text">-XX:+PrintGCDateStamps -Xloggc:$SAMZA_LOG_DIR/gc.log +</code></pre></div> <h4 id="rotation">Rotation</h4> <p>In older versions of Java, it is impossible to have GC logs roll over based on time or size without the use of a secondary tool. This means that your GC logs will never be deleted until a Samza job ceases to run. As of <a href="http://www.oracle.com/technetwork/java/javase/2col/6u34-bugfixes-1733379.html";>Java 6 Update 34</a>, and <a href="http://www.oracle.com/technetwork/java/javase/7u2-relnotes-1394228.html";>Java 7 Update 2</a>, <a href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6941923";>new GC command line switches</a> have been added to support this functionality. If you are using a version of Java that supports GC log rotation, it’s highly recommended that you turn it on.</p> <h3 id="yarn">YARN</h3> -<p>When a Samza job executes on a YARN grid, the <code>$SAMZA_LOG_DIR</code> environment variable will point to a directory that is secured such that only the user executing the Samza job can read and write to it, if YARN is <a href="http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html";>securely configured</a>.</p> +<p>When a Samza job executes on a YARN grid, the <em>$SAMZA</em>_<em>LOG</em>_<em>DIR</em> environment variable will point to a directory that is secured such that only the user executing the Samza job can read and write to it, if YARN is <a href="http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html";>securely configured</a>.</p> <h4 id="stdout">STDOUT</h4> Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/packaging.html URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/packaging.html?rev=1618099&r1=1618098&r2=1618099&view=diff ============================================================================== --- incubator/samza/site/learn/documentation/0.7.0/jobs/packaging.html (original) +++ incubator/samza/site/learn/documentation/0.7.0/jobs/packaging.html Fri Aug 15 05:36:35 2014 @@ -40,6 +40,9 @@ <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> <a href="https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + + <a href="http://samza.incubator.apache.org/learn/documentation/latest/jobs/packaging.html";><i class="fa fa-history masthead-icon"></i></a> + </div> </div> </div><!-- /.container --> @@ -49,14 +52,14 @@ <div class="menu"> <h1><i class="fa fa-rocket"></i> Getting Started</h1> <ul> - <li><a href="/startup/hello-samza/0.7.0">Hello Samza</a></li> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> <li><a href="/startup/download">Download</a></li> </ul> <h1><i class="fa fa-book"></i> Learn</h1> <ul> - <li><a href="/learn/documentation/0.7.0">Documentation</a></li> - <li><a href="/learn/tutorials/0.7.0">Tutorials</a></li> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> <li><a href="http://wiki.apache.org/samza/FAQ";>FAQ</a></li> <li><a href="http://wiki.apache.org/samza";>Wiki</a></li> <li><a href="http://wiki.apache.org/samza/PapersAndTalks";>Papers & Talks</a></li> @@ -84,6 +87,11 @@ <li><a href="https://builds.apache.org/";>Unit Tests</a></li> <li><a href="/contribute/disclaimer.html">Disclaimer</a></li> </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html">0.7.0</a></li> + </ul> </div> <div class="content">
