Modified: samza/site/learn/documentation/latest/container/jmx.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/jmx.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/jmx.html (original) +++ samza/site/learn/documentation/latest/container/jmx.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/container/jmx.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/container/jmx.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -175,7 +176,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/container/jmx.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/container/jmx.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } });
Modified: samza/site/learn/documentation/latest/container/metrics.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/metrics.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/metrics.html (original) +++ samza/site/learn/documentation/latest/container/metrics.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/container/metrics.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/container/metrics.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -234,7 +235,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/container/metrics.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/container/metrics.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/samza-container.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/samza-container.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/samza-container.html (original) +++ samza/site/learn/documentation/latest/container/samza-container.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/container/samza-container.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/container/samza-container.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -176,9 +177,9 @@ <p>If a Samza job has more than one input stream, the number of task instances for the Samza job is the maximum number of partitions across all input streams. For example, if a Samza job is reading from PageViewEvent (12 partitions), and ServiceMetricEvent (14 partitions), then the Samza job would have 14 task instances (numbered 0 through 13). Task instances 12 and 13 only receive events from ServiceMetricEvent, because there is no corresponding PageViewEvent partition.</p> -<p>With this default approach to assigning input streams to task instances, Samza is effectively performing a group-by operation on the input streams with their partitions as the key. Other strategies for grouping input stream partitions are possible by implementing a new <a href="../api/javadocs/org/apache/samza/container/SystemStreamPartitionGrouper.html">SystemStreamPartitionGrouper</a> and factory, and configuring the job to use it via the job.systemstreampartition.grouper.factory configuration value.</p> +<p>With this default approach to assigning input streams to task instances, Samza is effectively performing a group-by operation on the input streams with their partitions as the key. Other strategies for grouping input stream partitions are possible by implementing a new <a href="../api/javadocs/org/apache/samza/container/grouper/stream/SystemStreamPartitionGrouper.html">SystemStreamPartitionGrouper</a> and factory, and configuring the job to use it via the job.systemstreampartition.grouper.factory configuration value.</p> -<p>Samza provides the above-discussed per-partition grouper as well as the <a href="../api/javadocs/org/apache/samza/container/systemstreampartition/groupers/GroupBySystemStreamPartition">GroupBySystemStreamPartitionGrouper</a>, which provides a separate task class instance for every input stream partition, effectively grouping by the input stream itself. This provides maximum scalability in terms of how many containers can be used to process those input streams and is appropriate for very high volume jobs that need no grouping of the input streams.</p> +<p>Samza provides the above-discussed per-partition grouper as well as the GroupBySystemStreamPartitionGrouper, which provides a separate task class instance for every input stream partition, effectively grouping by the input stream itself. This provides maximum scalability in terms of how many containers can be used to process those input streams and is appropriate for very high volume jobs that need no grouping of the input streams.</p> <p>Considering the above example of a PageViewEvent partitioned 12 ways and a ServiceMetricEvent partitioned 14 ways, the GroupBySystemStreamPartitionGrouper would create 12 + 14 = 26 task instances, which would then be distributed across the number of containers configured, as discussed below.</p> @@ -225,6 +226,14 @@ <p>There is one caveat in all of this: Samza currently assumes that a stream’s partition count will never change. Partition splitting or repartitioning is not supported. If an input stream has N partitions, it is expected that it has always had, and will always have N partitions. If you want to re-partition a stream, you can write a job that reads messages from the stream, and writes them out to a new stream with the required number of partitions. For example, you could read messages from PageViewEvent, and write them to PageViewEventRepartition.</p> +<h3 id="broadcast-streams">Broadcast Streams</h3> + +<p>After 0.10.0, Samza supports broadcast streams. You can assign partitions from some streams to all the tasks. For example, you want all the tasks can consume partition 0 and 1 from a stream called global-stream-1, and partition 2 from a stream called global-stream-2. You now can configure:</p> + +<div class="highlight"><pre><code class="jproperties"><span class="na">task.broadcast.inputs</span><span class="o">=</span><span class="s">yourSystem.broadcast-stream-1#[0-1], yourSystem.broadcast-stream-2#2</span></code></pre></div> + +<p>If you use “[]”, you are specifying a range.</p> + <h2 id="streams-»"><a href="streams.html">Streams »</a></h2> @@ -243,7 +252,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/container/samza-container.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/container/samza-container.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/serialization.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/serialization.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/serialization.html (original) +++ samza/site/learn/documentation/latest/container/serialization.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/container/serialization.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/container/serialization.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -176,7 +177,62 @@ <span class="na">stores.LastPageViewPerUser.key.serde</span><span class="o">=</span><span class="s">integer</span> <span class="na">stores.LastPageViewPerUser.msg.serde</span><span class="o">=</span><span class="s">json</span></code></pre></div> -<p>Each serde is defined with a factory class. Samza comes with several builtin serdes for UTF-8 strings, binary-encoded integers, JSON and more. You can also create your own serializer by implementing the <a href="../api/javadocs/org/apache/samza/serializers/SerdeFactory.html">SerdeFactory</a> interface.</p> +<p>Each serde is defined with a factory class. Samza comes with several builtin serdes for UTF-8 strings, binary-encoded integers, JSON and more. The following is a comprehensive list of supported serdes in Samza. +<style> + table th, table td { + text-align: left; + vertical-align: top; + padding: 12px; + border-bottom: 1px solid #ccc; + border-top: 1px solid #ccc; + border-left: 0; + border-right: 0; + }</p> +<div class="highlight"><pre><code class="language-text" data-lang="text"> table td.property, table td.default { + white-space: nowrap; + } + + table th { + background-color: #eee; + } +</code></pre></div> +<p></style> +<table> + <tr> + <th> Serde Name</th> + <th> Data Handled </th> + </tr> + <tr> + <td> string </td> + <td> UTF-8 strings </td> + </tr> + <tr> + <td> integer </td> + <td> binary-encoded integers </td> + </tr> + <tr> + <td> serializable </td> + <td> Serializable Object Type </td> + </tr> + <tr> + <td> long </td> + <td> long data type </td> + </tr> + <tr> + <td> json </td> + <td> JSON formatted data </td> + </tr> + <tr> + <td> byte </td> + <td> Plain Bytes (effectively no-op) - Useful for Binary Messages </td> + </tr> + <tr> + <td> bytebuffer </td> + <td> Byte Buffer </td> + </tr> +</table></p> + +<p>You can also create your own serializer by implementing the <a href="../api/javadocs/org/apache/samza/serializers/SerdeFactory.html">SerdeFactory</a> interface.</p> <p>The name you give to a serde (such as “json” and “integer” in the example above) is only for convenience in your job configuration; you can choose whatever name you like. For each stream and each state store, you can use the serde name to declare how messages should be serialized and deserialized.</p> @@ -202,7 +258,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/container/serialization.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/container/serialization.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/state-management.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/state-management.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/state-management.html (original) +++ samza/site/learn/documentation/latest/container/state-management.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/container/state-management.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/container/state-management.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -309,6 +310,48 @@ <p>Additional configuration properties for the key-value store are documented in the <a href="../jobs/configuration-table.html#keyvalue-rocksdb">configuration reference</a>.</p> +<h3 id="debug-key-value-storage">Debug Key-value storage</h3> + +<h4 id="materialize-a-state-store-from-the-changelog">Materialize a state store from the changelog</h4> + +<p>Currently Samza provides a state storage tool which can recover the state store from the changelog stream to user-specified directory for reusing and debugging.</p> + +<div class="highlight"><pre><code class="bash">samza-example/target/bin/state-storage-tool.sh <span class="se">\</span> + --config-path<span class="o">=</span>file:///path/to/job/config.properties <span class="se">\</span> + --path<span class="o">=</span>directory/to/put/state/stores</code></pre></div> + +<h4 id="read-the-value-from-a-running-rocksdb">Read the value from a running RocksDB</h4> + +<p>Samza also provides a tool to read the value from a running job’s RocksDB.</p> + +<div class="highlight"><pre><code class="bash">samza-example/target/bin/read-rocksdb-tool.sh <span class="se">\</span> + --config-path<span class="o">=</span>file:///path/to/job/config.properties <span class="se">\</span> + --db-path<span class="o">=</span>/tmp/nm-local-dir/state/test-state/Partition_0 <span class="se">\</span> + --db-name<span class="o">=</span><span class="nb">test</span>-state <span class="se">\</span> + --string-key<span class="o">=</span>a,b,c</code></pre></div> + +<ul> +<li><code>--config-path</code>(required): your job’s configuration file</li> +<li><code>--db-path</code>(required): the location of your RocksDB. This is convenient if the RocksDB is in the same machine as the tool. E.g. if you are running hello-samza in your local machine, the location maybe in +<em>/tmp/hadoop/nm-local-dir/usercache/username/appcache/applicationId/containerId/state/storeName/PartitionNumber</em></li> +<li><code>--db-name</code>(required): if you only have one state store specified in the config file, you can ignore this one. Otherwise, you need to provide the state store name here.</li> +<li><code>--string-key</code>: the key list. This one only works if your keys are string. There are also another two options: <code>--integer-key</code>, <code>--long-key</code>. They work for integer keys and long keys respectively.</li> +</ul> + +<p><strong>Limitations</strong>:</p> + +<ul> +<li>This only works with three kinds of keys: string, integer and long. This is because we can only accept those kinds of keys from the command line (it is really tricky to accept bytes, avro, json, etc from the command line). But it is also easy to use this tool programmatically (The key and value both are deserialized.)</li> +</ul> + +<div class="highlight"><pre><code class="bash">RocksDbKeyValueReader <span class="nv">kvReader</span> <span class="o">=</span> new RocksDbKeyValueReader<span class="o">(</span>dbName, pathOfdb, config<span class="o">)</span> +Object <span class="nv">value</span> <span class="o">=</span> kvReader.get<span class="o">(</span>key<span class="o">)</span></code></pre></div> + +<ul> +<li>Because Samza job has some caches and buffers, you may not be able to see expected values (or even not be able to see any value, if all the data is buffered). Some of the related configuration are <code>stores.store-name.container.write.buffer.size.bytes</code>, <code>stores.store-name.write.batch.size</code>, <code>stores.store-name.object.cache.size</code>. You may want to set them to very small for testing.</li> +<li>Since RocksDB memtable is not flushed to disk immediately on every write, you may not be able to see the expected values until it is written to the SST file on disk. For more details on RocksDb, you can refer the docs <a href="https://github.com/facebook/rocksdb/wiki/RocksDB-Basics";>here</a>.</li> +</ul> + <h4 id="known-issues">Known Issues</h4> <p>RocksDB has several rough edges. It’s recommended that you read the RocksDB <a href="https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide";>tuning guide</a>. Some other notes to be aware of are:</p> @@ -390,7 +433,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/container/state-management.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/container/state-management.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/streams.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/streams.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/streams.html (original) +++ samza/site/learn/documentation/latest/container/streams.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/container/streams.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/container/streams.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -267,7 +268,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/container/streams.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/container/streams.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/container/windowing.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/container/windowing.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/container/windowing.html (original) +++ samza/site/learn/documentation/latest/container/windowing.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/container/windowing.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/container/windowing.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -195,7 +196,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/container/windowing.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/container/windowing.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Added: samza/site/learn/documentation/latest/hdfs/producer.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/hdfs/producer.html?rev=1721446&view=auto ============================================================================== --- samza/site/learn/documentation/latest/hdfs/producer.html (added) +++ samza/site/learn/documentation/latest/hdfs/producer.html Tue Dec 22 19:03:17 2015 @@ -0,0 +1,232 @@ +<!DOCTYPE html> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<html lang="en"> + <head> + <meta charset="utf-8"> + <title>Samza - Isolation</title> + <link href='/css/ropa-sans.css' rel='stylesheet' type='text/css'/> + <link href="/css/bootstrap.min.css" rel="stylesheet"/> + <link href="/css/font-awesome.min.css" rel="stylesheet"/> + <link href="/css/main.css" rel="stylesheet"/> + <link href="/css/syntax.css" rel="stylesheet"/> + <link rel="icon" type="image/png" href="/img/samza-icon.png"> + <script src="/js/jquery-1.11.1.min.js"></script> + </head> + <body> + <div class="wrapper"> + <div class="wrapper-content"> + + <div class="masthead"> + <div class="container"> + <div class="masthead-logo"> + <a href="/" class="logo">samza</a> + </div> + <div class="masthead-icons"> + <div class="pull-right"> + <a href="/startup/download"><i class="fa fa-arrow-circle-o-down masthead-icon"></i></a> + <a href="https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tree"; target="_blank"><i class="fa fa-code masthead-icon" style="font-weight: bold;"></i></a> + <a href="https://twitter.com/samzastream"; target="_blank"><i class="fa fa-twitter masthead-icon"></i></a> + <!-- this icon only shows in versioned pages --> + + + + + <a href="http://samza.apache.org/learn/documentation/0.10/hdfs/producer.html";><i id="switch-version-button"></i></a> + <!-- links for the navigation bar --> + + + </div> + </div> + </div><!-- /.container --> + </div> + + <div class="container"> + <div class="menu"> + <h1><i class="fa fa-rocket"></i> Getting Started</h1> + <ul> + <li><a href="/startup/hello-samza/latest">Hello Samza</a></li> + <li><a href="/startup/download">Download</a></li> + </ul> + + <h1><i class="fa fa-book"></i> Learn</h1> + <ul> + <li><a href="/learn/documentation/latest">Documentation</a></li> + <li><a href="/learn/documentation/latest/jobs/configuration-table.html">Configuration</a></li> + <li><a href="/learn/documentation/latest/api/javadocs/">Javadocs</a></li> + <li><a href="/learn/tutorials/latest">Tutorials</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/SAMZA/FAQ";>FAQ</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/SAMZA/Apache+Samza";>Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=51812876";>Papers & Talks</a></li> + <li><a href="http://blogs.apache.org/samza";>Blog</a></li> + </ul> + + <h1><i class="fa fa-comments"></i> Community</h1> + <ul> + <li><a href="/community/mailing-lists.html">Mailing Lists</a></li> + <li><a href="/community/irc.html">IRC</a></li> + <li><a href="https://issues.apache.org/jira/browse/SAMZA";>Bugs</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/SAMZA/Powered+By";>Powered by</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/SAMZA/Ecosystem";>Ecosystem</a></li> + <li><a href="/community/committers.html">Committers</a></li> + </ul> + + <h1><i class="fa fa-code"></i> Contribute</h1> + <ul> + <li><a href="/contribute/rules.html">Rules</a></li> + <li><a href="/contribute/coding-guide.html">Coding Guide</a></li> + <li><a href="/contribute/projects.html">Projects</a></li> + <li><a href="/contribute/design-documents.html">Design Documents</a></li> + <li><a href="/contribute/code.html">Code</a></li> + <li><a href="https://reviews.apache.org/groups/samza";>Review Board</a></li> + <li><a href="/contribute/tests.html">Tests</a></li> + </ul> + + <h1><i class="fa fa-history"></i> Archive</h1> + <ul> + <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> + <li><a href="/archive/index.html#09">0.9</a></li> + <li><a href="/archive/index.html#08">0.8</a></li> + <li><a href="/archive/index.html#07">0.7</a></li> + </ul> + </div> + + <div class="content"> + <!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<h2>Isolation</h2> + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<h3 id="writing-to-hdfs-from-samza">Writing to HDFS from Samza</h3> + +<p>The <code>samza-hdfs</code> module implements a Samza Producer to write to HDFS. The current implementation includes a ready-to-use <code>HdfsSystemProducer</code>, and two <code>HdfsWriter</code>s: One that writes messages of raw bytes to a <code>SequenceFile</code> of <code>BytesWritable</code> keys and values. The other writes UTF-8 <code>String</code>s to a <code>SequenceFile</code> with <code>LongWritable</code> keys and <code>Text</code> values.</p> + +<h3 id="configuring-an-hdfssystemproducer">Configuring an HdfsSystemProducer</h3> + +<p>You can configure an HdfsSystemProducer like any other Samza system: using configuration keys and values set in a <code>job.properties</code> file. +You might configure the system producer for use by your <code>StreamTasks</code> like this:</p> +<div class="highlight"><pre><code class="language-text" data-lang="text"># set the SystemFactory implementation to instantiate HdfsSystemProducer aliased to 'hdfs-clickstream' +systems.hdfs-clickstream.samza.factory=org.apache.samza.system.hdfs.HdfsSystemFactory + +# define a serializer/deserializer for the hdfs-clickstream system +systems.hdfs-clickstream.samza.msg.serde=some-serde-impl + +# consumer configs not needed for HDFS system, reader is not implemented yet + +# Assign a Metrics implementation via a label we defined earlier in the props file +systems.hdfs-clickstream.streams.metrics.samza.msg.serde=some-metrics-impl + +# Assign the implementation class for this system's HdfsWriter +systems.hdfs-clickstream.producer.hdfs.writer.class=org.apache.samza.system.hdfs.writer.TextSequenceFileHdfsWriter + +# Set HDFS SequenceFile compression type. Only BLOCK compression is supported currently +systems.hdfs-clickstream.producer.hdfs.compression.type=snappy + +# The base dir for HDFS output. The default Bucketer for SequenceFile HdfsWriters +# is currently /BASE/JOB_NAME/DATE_PATH/FILES, where BASE is set below +systems.hdfs-clickstream.producer.hdfs.base.output.dir=/user/me/analytics/clickstream_data + +# Assign the implementation class for the HdfsWriter's Bucketer +systems.hdfs-clickstream.producer.hdfs.bucketer.class=org.apache.samza.system.hdfs.writer.JobNameDateTimeBucketer + +# Configure the DATE_PATH the Bucketer will set to bucket output files by day for this job run. +systems.hdfs-clickstream.producer.hdfs.bucketer.date.path.format=yyyy_MM_dd + +# Optionally set the max output bytes per file. A new file will be cut and output +# continued on the next write call each time this many bytes are written. +systems.hdfs-clickstream.producer.hdfs.write.batch.size.bytes=134217728 +</code></pre></div> +<p>The above configuration assumes a Metrics and Serde implemnetation has been properly configured against the <code>some-serde-impl</code> and <code>some-metrics-impl</code> labels somewhere else in the same <code>job.properties</code> file. Each of these properties has a reasonable default, so you can leave out the ones you don’t need to customize for your job run.</p> + + + </div> + </div> + + </div><!-- /.wrapper-content --> + </div><!-- /.wrapper --> + + <div class="footer"> + <div class="container"> + <!-- nothing for now. --> + </div> + </div> + + + <script> + $( document ).ready(function() { + if ( $.fn.urlExists( "/learn/documentation/0.10/hdfs/producer.html" ) ) { + $("#switch-version-button").addClass("fa fa-history masthead-icon"); + } + }); + + /* a function to test whether the url exists or not */ + (function( $ ) { + $.fn.urlExists = function(url) { + var http = new XMLHttpRequest(); + http.open('HEAD', url, false); + http.send(); + return http.status != 404; + }; + }( jQuery )); + </script> + + + <!-- Google Analytics --> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-43122768-1', 'apache.org'); + ga('send', 'pageview'); + + </script> + </body> +</html> Modified: samza/site/learn/documentation/latest/index.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/index.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/index.html (original) +++ samza/site/learn/documentation/latest/index.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/index.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/index.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -172,7 +173,7 @@ <li><a href="api/javadocs">Javadocs</a></li> </ul> -<h4>Container</h4> +<h4>Core</h4> <ul class="documentation-list"> <li><a href="container/samza-container.html">SamzaContainer</a></li> @@ -180,13 +181,14 @@ <li><a href="container/serialization.html">Serialization</a></li> <li><a href="container/checkpointing.html">Checkpointing</a></li> <li><a href="container/state-management.html">State Management</a></li> - <li><a href="container/metrics.html">Metrics</a></li> <li><a href="container/windowing.html">Windowing</a></li> + <li><a href="container/coordinator-stream.html">Coordinator Stream</a></li> <li><a href="container/event-loop.html">Event Loop</a></li> + <li><a href="container/metrics.html">Metrics</a></li> <li><a href="container/jmx.html">JMX</a></li> </ul> -<h4>Jobs</h4> +<h4>Job Deployment</h4> <ul class="documentation-list"> <li><a href="jobs/job-runner.html">JobRunner</a></li> @@ -203,6 +205,8 @@ <ul class="documentation-list"> <li><a href="yarn/application-master.html">Application Master</a></li> <li><a href="yarn/isolation.html">Isolation</a></li> + <li><a href="yarn/yarn-host-affinity.html">Host Affinity & Yarn</a></li> + <li><a href="hdfs/producer.html">Writing to HDFS</a></li> <!-- TODO write yarn pages <li><a href="">Fault Tolerance</a></li> <li><a href="">Security</a></li> @@ -214,6 +218,7 @@ <ul class="documentation-list"> <li><a href="operations/security.html">Security</a></li> <li><a href="operations/kafka.html">Kafka</a></li> +</ul> </div> @@ -232,7 +237,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/index.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/index.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/introduction/architecture.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/introduction/architecture.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/introduction/architecture.html (original) +++ samza/site/learn/documentation/latest/introduction/architecture.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/introduction/architecture.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/introduction/architecture.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -254,7 +255,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/introduction/architecture.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/introduction/architecture.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/introduction/background.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/introduction/background.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/introduction/background.html (original) +++ samza/site/learn/documentation/latest/introduction/background.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/introduction/background.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/introduction/background.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -215,7 +216,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/introduction/background.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/introduction/background.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/introduction/concepts.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/introduction/concepts.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/introduction/concepts.html (original) +++ samza/site/learn/documentation/latest/introduction/concepts.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/introduction/concepts.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/introduction/concepts.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -210,7 +211,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/introduction/concepts.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/introduction/concepts.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/jobs/configuration-table.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/jobs/configuration-table.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/jobs/configuration-table.html (original) +++ samza/site/learn/documentation/latest/jobs/configuration-table.html Tue Dec 22 19:03:17 2015 @@ -122,7 +122,7 @@ <strong>Required:</strong> The <a href="job-runner.html">job factory</a> to use for running this job. The value is a fully-qualified Java classname, which must implement <a href="../api/javadocs/org/apache/samza/job/StreamJobFactory.html">StreamJobFactory</a>. - Samza ships with two implementations: + Samza ships with three implementations: <dl> <dt><code>org.apache.samza.job.local.ThreadJobFactory</code></dt> <dd>Runs your job on your local machine using threads. This is intended only for @@ -156,6 +156,32 @@ others' checkpoints, and perhaps interfere with each other in other ways. </td> </tr> + <tr> + <td class="property" id="job-coordinator-system">job.coordinator.system</td> + <td class="default"></td> + <td class="description"> + <strong>Required:</strong> The <span class="system">system-name</span> to use for creating and maintaining the <a href="../container/coordinator-stream.html">Coordinator Stream</a>. + </td> + </tr> + + <tr> + <td class="property" id="job-coordinator-replication-factor">job.coordinator.<br />replication.factor</td> + <td class="default">3</td> + <td class="description"> + If you are using Kafka for coordinator stream, this is the number of Kafka nodes to which you want the + coordinator topic replicated for durability. + </td> + </tr> + + <tr> + <td class="property" id="job-coordinator-segment-bytes">job.coordinator.<br />segment.bytes</td> + <td class="default">26214400</td> + <td class="description"> + If you are using a Kafka system for coordinator stream, this is the segment size to be used for the coordinator + topic's log segments. Keeping this number small is useful because it increases the frequency + that Kafka will garbage collect old messages. + </td> + </tr> <tr> <td class="property" id="job-config-rewriter-class">job.config.rewriter.<br><span class="rewriter">rewriter-name</span>.class</td> @@ -381,9 +407,8 @@ <td class="property" id="task-log4j-system">task.log4j.system</td> <td class="default"></td> <td class="description"> - Specify the system name for the StreamAppender. If Samza can not find this property and there is - only one system in the config, it will guess to use that system for the log4j appender. If this property - is not specified and there are more than one system in the config, Samza throws exception. (See + Specify the system name for the StreamAppender. If this property is not specified in the config, + Samza throws exception. (See <a href="logging.html#stream-log4j-appender">Stream Log4j Appender</a>) <dl> <dt>Example: <code>task.log4j.system=kafka</code></dt> @@ -436,6 +461,31 @@ This property controls how long the Samza container will wait for an orderly shutdown of task instances. </td> </tr> + <tr> + <td class="property" id="task-name-grouper-factory">task.name.grouper.factory</td> + <td class="default">org.apache.samza.<br>container.grouper.task.<br>GroupByContainerCountFactory</td> + <td class="description"> + The fully-qualified name of the Java class which determines the factory class which will build the TaskNameGrouper. + The default configuration value if the property is not present is <code>task.name.grouper.factory=org.apache.samza.container.grouper.task.GroupByContainerCountFactory</code>.<br> + The user can specify a custom implementation of the TaskNameGrouperFactory where a custom logic is implemented for grouping the tasks. + </td> + </tr> + + <tr> + <td class="property" id="task-broadcast-inputs">task.broadcast.inputs</td> + <td class="default"></td> + <td class="description"> + This property specifies the partitions that all tasks should consume. The systemStreamPartitions you put + here will be sent to all the tasks. + <dl> + <dt>Format: <span class="system">system-name</span>.<span class="stream">stream-name</span>#<i>partitionId</i> + or <span class="system">system-name</span>.<span class="stream">stream-name</span>#[<i>startingPartitionId</i>-<i>endingPartitionId</i>]</dt> + </dl> + <dl> + <dt>Example: <code>task.broadcast.inputs=mySystem.broadcastStream#[0-2], mySystem.broadcastStream#0</code></dt> + </dl> + </td> + </tr> <tr> <th colspan="3" class="section" id="streams"><a href="../container/streams.html">Systems (input and output streams)</a></th> @@ -852,6 +902,31 @@ </tr> <tr> + <td class="property" id="systems-samza-fetch-threshold-bytes">systems.<span class="system">system-name</span>.<br>samza.fetch.threshold.bytes</td> + <td class="default">-1</td> + <td class="description"> + When consuming streams from Kafka, a Samza container maintains an in-memory buffer + for incoming messages in order to increase throughput (the stream task can continue + processing buffered messages while new messages are fetched from Kafka). This + parameter determines the total size of messages we aim to buffer across all stream + partitions consumed by a container based on bytes. Defines how many bytes to use for the buffered + prefetch messages for job as a whole. The bytes for a single system/stream/partition are computed based on this. + This fetches the entire messages, hence this bytes limit is a soft one, and the actual usage can be the bytes + limit + size of max message in the partition for a given stream. If the value of this property is > 0 + then this takes precedence over systems.<span class="system">system-name</span>.samza.fetch.threshold.<br> + For example, if fetchThresholdBytes is set to 100000 bytes, and there are 50 SystemStreamPartitions registered, + then the per-partition threshold is (100000 / 2) / 50 = 1000 bytes. As this is a soft limit, the actual usage + can be 1000 bytes + size of max message. As soon as a SystemStreamPartition's buffered messages bytes drops + below 1000, a fetch request will be executed to get more data for it. + + Increasing this parameter will decrease the latency between when a queue is drained of messages and when new + messages are enqueued, but also leads to an increase in memory usage since more messages will be held in memory. + + The default value is -1, which means this is not used. + </td> + </tr> + + <tr> <td class="property" id="task-checkpoint-system">task.checkpoint.system</td> <td class="default"></td> <td class="description"> @@ -893,7 +968,7 @@ </tr> <tr> - <td class="property" id="store-changelog-partitions">stores.<span class="store">store-name</span>.changelog.<br>topic-level-property</td> + <td class="property" id="store-changelog-partitions">stores.<span class="store">store-name</span>.changelog.<br>kafka.topic-level-property</td> <td class="default"></td> <td class="description"> The property allows you to specify topic level settings for the changelog topic to be created. @@ -1292,6 +1367,36 @@ </tr> <tr> + <td class="property" id="yarn-allocator-sleep-ms">yarn.allocator.sleep.ms</td> + <td class="default">3600</td> + <td class="description"> + The container allocator thread is responsible for matching requests to allocated containers. + The sleep interval for this thread is configured using this property. + </td> + </tr> + + <tr> + <td class="property" id="yarn-samza-host_affinity-enabled">yarn.samza.host-affinity.enabled</td> + <td class="default">false</td> + <td class="description"> + This property indicates whether host-affinity is enabled or not. Host-affinity refers to the ability of the Application Master to request and allocate a container on the same host every time the job is deployed. + When host-affinity is enabled, the AM makes a "best-effort" to honor the host-affinity constraint. + The property <a href="#yarn-container-request-timeout-ms" class="property">yarn.container.request.timeout.ms</a> determines how long to wait before de-prioritizing the host-affinity constraint and assigning the container to any available resource. + <b>Please Note</b>: This feature is tested to work with the FairScheduler in Yarn when continuous-scheduling is enabled. + </td> + </tr> + + <tr> + <td class="property" id="yarn-container-request-timeout-ms">yarn.container.request.timeout.ms</td> + <td class="default">5000</td> + <td class="description"> + The allocator thread periodically checks the state of the container requests and allocated containers to determine the assignment of a container to an allocated resource. + This property determines the number of milliseconds before a container request is considered to have expired / timed-out. + When a request expires, it gets allocated to any available container that was returned by the RM. + </td> + </tr> + + <tr> <td class="property" id="yarn-queue">yarn.queue</td> <td class="default"></td> <td class="description"> @@ -1368,6 +1473,78 @@ </ul> </td> </tr> + <tr> + <td class="property" id="metrics-reporter-polling-interval">metrics.reporter.<br><span class="reporter">reporter-name</span>.interval</td> + <td class="default"></td> + <td class="description"> + If you have registered the metrics reporter + <a href="#metrics-reporter-class" class="property">metrics.reporter.*.class</a> + <code>= org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory</code>, + you can use this property to configure how frequently the reporter will report the metrics + registered with it. The value for this property should be length of the interval between + consecutive metric reporting. This value is in seconds, and should be a positive integer value. + This property is optional and set to 60 by default, which means metrics will be reported every + 60 seconds. + </td> + </tr> + + <tr> + <th colspan="3" class="section" id="hdfs-system-producer"><a href="../hdfs/producer.html">Writing to HDFS</a></th> + </tr> + + <tr> + <td class="property" id="hdfs-writer-class">systems.*.producer.hdfs.writer.class</td> + <td class="default">org.apache.samza.system.hdfs.writer.BinarySequenceFileHdfsWriter</td> + <td class="description">Fully-qualified class name of the HdfsWriter implementation this HDFS Producer system should use</td> + </tr> + <tr> + <td class="property" id="hdfs-compression-type">systems.*.producer.hdfs.compression.type</td> + <td class="default">none</td> + <td class="description">A human-readable label for the compression type to use, such as "gzip" "snappy" etc. This label will be interpreted differently (or ignored) depending on the nature of the HdfsWriter implementation.</td> + </tr> + <tr> + <td class="property" id="hdfs-base-output-dir">systems.*.producer.hdfs.base.output.dir</td> + <td class="default">/user/USERNAME/SYSTEMNAME</td> + <td class="description">The base output directory for HDFS writes. Defaults to the home directory of the user who ran the job, followed by the systemName for this HdfsSystemProducer as defined in the job.properties file.</td> + </tr> + <tr> + <td class="property" id="hdfs-bucketer-class">systems.*.producer.hdfs.bucketer.class</td> + <td class="default">org.apache.samza.system.hdfs.writer.JobNameDateTimeBucketer</td> + <td class="description">Fully-qualified class name of the Bucketer implementation that will manage HDFS paths and file names. Used to batch writes by time, or other similar partitioning methods.</td> + </tr> + <tr> + <td class="property" id="hdfs-bucketer-date-path-format">systems.*.producer.hdfs.bucketer.date.path.format</td> + <td class="default"yyyy_MM_dd></td> + <td class="description">A date format (using Java's SimpleDataFormat syntax) appropriate for use in an HDFS Path, which can configure time-based bucketing of output files.</td> + </tr> + <tr> + <td class="property" id="hdfs-write-batch-size-bytes">systems.*.producer.write.batch.size.bytes</td> + <td class="default">268435456</td> + <td class="description">The number of bytes of outgoing messages to write to each HDFS output file before cutting a new file. Defaults to 256MB if not set.</td> + </tr> + + <tr> + <th colspan="3" class="section" id="task-migration"> + Migrating from Samza 0.9.1 to 0.10.0<br> + <span class="subtitle"> + (This section applies if you are upgrading from Samza 0.9.1 to 0.10.0 and have set + <a href="#task-checkpoint-factory" class="property">task.checkpoint.factory</a> to anything <b> other than </b> + <code>org.apache.samza.checkpoint.file.FileSystemCheckpointManagerFactory</code>) + </span> + </th> + </tr> + + <tr> + <td class="property" id="task-checkpoint-skip-migration">task.checkpoint.skip-migration</td> + <td class="default">false</td> + <td class="description"> + When migrating from 0.9.1 to 0.10.0, the taskName-to-changelog partition mapping was moved from the checkpoint stream to the coordinator stream. <br /> + If you are using a checkpoint manager other than kafka + (<code>org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory</code>), you have to + manually migrate taskName-to-changelog partition mapping to the coordinator stream. <br /> + This can be achieved with the assistance of the <code>checkpoint-tool.sh</code>. + </td> + </tr> </tbody> </table> </body> Modified: samza/site/learn/documentation/latest/jobs/configuration.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/jobs/configuration.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/jobs/configuration.html (original) +++ samza/site/learn/documentation/latest/jobs/configuration.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/jobs/configuration.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/jobs/configuration.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -206,7 +207,7 @@ encryption is done.</p> <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/jobs/configuration.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/jobs/configuration.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/jobs/job-runner.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/jobs/job-runner.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/jobs/job-runner.html (original) +++ samza/site/learn/documentation/latest/jobs/job-runner.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/jobs/job-runner.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/jobs/job-runner.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -192,7 +193,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/jobs/job-runner.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/jobs/job-runner.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/jobs/logging.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/jobs/logging.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/jobs/logging.html (original) +++ samza/site/learn/documentation/latest/jobs/logging.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/jobs/logging.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/jobs/logging.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -204,18 +205,24 @@ <h4 id="stream-log4j-appender">Stream Log4j Appender</h4> -<p>Samza provides a StreamAppender to publish the logs into a specific system. You can specify the system name using “task.log4j.system”. If there is only one system in the config, Samza will use that system for the log publishing. Also we have the <a href="http://logback.qos.ch/manual/mdc.html";>MDC</a> keys “containerName”, “jobName” and “jobId”, which help identify the source of the log. In order to use this appender, simply add:</p> +<p>Samza provides a StreamAppender to publish the logs into a specific system. You can specify the system name using “task.log4j.system” and change name of log stream with param ‘StreamName’. Also, we have the <a href="http://logback.qos.ch/manual/mdc.html";>MDC</a> keys “containerName”, “jobName” and “jobId”, which help identify the source of the log. In order to use this appender, simply add:</p> <div class="highlight"><pre><code class="xml"><span class="nt"><appender</span> <span class="na">name=</span><span class="s">"StreamAppender"</span> <span class="na">class=</span><span class="s">"org.apache.samza.logging.log4j.StreamAppender"</span><span class="nt">></span> + <span class="c"><!-- optional --></span> + <span class="nt"><param</span> <span class="na">name=</span><span class="s">"StreamName"</span> <span class="na">value=</span><span class="s">"EpicStreamName"</span><span class="nt">/></span> <span class="nt"><layout</span> <span class="na">class=</span><span class="s">"org.apache.log4j.PatternLayout"</span><span class="nt">></span> <span class="nt"><param</span> <span class="na">name=</span><span class="s">"ConversionPattern"</span> <span class="na">value=</span><span class="s">"%X{containerName} %X{jobName} %X{jobId} %d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n"</span> <span class="nt">/></span> <span class="nt"></layout></span> <span class="nt"></appender></span></code></pre></div> -<p>and add:</p> +<p>and</p> <div class="highlight"><pre><code class="xml"><span class="nt"><appender-ref</span> <span class="na">ref=</span><span class="s">"StreamAppender"</span><span class="nt">/></span></code></pre></div> +<p>to log4j.xml and define the system name by specifying the config:</p> + +<div class="highlight"><pre><code class="xml">task.log4j.system="<span class="nt"><system-name></span>"</code></pre></div> + <p>Configuring the StreamAppender will automatically encode messages using logstash’s <a href="https://github.com/logstash/log4j-jsonevent-layout";>Log4J JSON format</a>. Samza also supports pluggable serialization for those that prefer non-JSON logging events. This can be configured the same way other stream serializers are defined:</p> <div class="highlight"><pre><code class="jproperties"><span class="na">serializers.registry.log4j-string.class</span><span class="o">=</span><span class="s">org.apache.samza.logging.log4j.serializers.LoggingEventStringSerdeFactory</span> @@ -265,7 +272,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/jobs/logging.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/jobs/logging.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/jobs/packaging.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/jobs/packaging.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/jobs/packaging.html (original) +++ samza/site/learn/documentation/latest/jobs/packaging.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/jobs/packaging.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/jobs/packaging.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -179,7 +180,7 @@ lib/*.jar <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/jobs/packaging.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/jobs/packaging.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/jobs/reprocessing.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/jobs/reprocessing.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/jobs/reprocessing.html (original) +++ samza/site/learn/documentation/latest/jobs/reprocessing.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/jobs/reprocessing.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/jobs/reprocessing.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -230,7 +231,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/jobs/reprocessing.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/jobs/reprocessing.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/jobs/web-ui-rest-api.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/jobs/web-ui-rest-api.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/jobs/web-ui-rest-api.html (original) +++ samza/site/learn/documentation/latest/jobs/web-ui-rest-api.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/jobs/web-ui-rest-api.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/jobs/web-ui-rest-api.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -202,7 +203,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/jobs/web-ui-rest-api.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/jobs/web-ui-rest-api.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/jobs/yarn-jobs.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/jobs/yarn-jobs.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/jobs/yarn-jobs.html (original) +++ samza/site/learn/documentation/latest/jobs/yarn-jobs.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/jobs/yarn-jobs.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/jobs/yarn-jobs.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -176,7 +177,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/jobs/yarn-jobs.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/jobs/yarn-jobs.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/operations/kafka.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/operations/kafka.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/operations/kafka.html (original) +++ samza/site/learn/documentation/latest/operations/kafka.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/operations/kafka.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/operations/kafka.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -170,7 +171,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/operations/kafka.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/operations/kafka.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/operations/security.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/operations/security.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/operations/security.html (original) +++ samza/site/learn/documentation/latest/operations/security.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/operations/security.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/operations/security.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -212,7 +213,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/operations/security.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/operations/security.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/yarn/application-master.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/yarn/application-master.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/yarn/application-master.html (original) +++ samza/site/learn/documentation/latest/yarn/application-master.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/yarn/application-master.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/yarn/application-master.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -151,18 +152,19 @@ <p>When the Samza ApplicationMaster starts up, it does the following:</p> <ol> -<li>Receives configuration from YARN via the STREAMING_CONFIG environment variable.</li> +<li>Creates the <a href="../container/coordinator-stream.html#JobCoordinator">Job Coordinator</a> which bootstraps the Job Model and config from the <a href="../container/coordinator-stream.html">Coordinator Stream</a>.</li> <li>Starts a JMX server on a random port.</li> <li>Instantiates a metrics registry and reporters to keep track of relevant metrics.</li> <li>Registers the AM with YARN’s RM.</li> -<li>Get the total number of partitions for the Samza job using each input stream’s PartitionManager (see the <a href="../container/streams.html">Streams</a> page for details).</li> +<li>Gets the total number of partitions for the Samza job using each input stream’s PartitionManager (see the <a href="../container/streams.html">Streams</a> page for details).</li> <li>Read the total number of containers requested from the Samza job’s configuration.</li> <li>Assign each partition to a container (called a Task Group in Samza’s AM dashboard).</li> -<li>Make a <a href="http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/api/records/ResourceRequest.html";>ResourceRequest</a> to YARN for each container.</li> +<li>Make a <a href="http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/api/records/ResourceRequest.html";>ResourceRequest</a> to YARN for each container. If <a href="yarn-host-affinity.html">host-affinity</a> is enabled on the job, the AM uses the container locality information provided by the Job Coordinator and requests for the same host in the ResourceRequest.</li> +<li>Starts a ContainerAllocator thread that matches allocated containers and starts the container process.</li> <li>Poll the YARN RM every second to check for allocated and released containers.</li> </ol> -<p>From this point on, the ApplicationMaster just reacts to events from the RM.</p> +<p>From this point on, the ApplicationMaster just reacts to events from the RM and delegates it to the ContainerAllocator thread.</p> <h3 id="fault-tolerance">Fault Tolerance</h3> @@ -172,21 +174,6 @@ <p>If the AM, itself, fails, YARN will handle restarting the AM. When the AM is restarted, all containers that were running will be killed, and the AM will start from scratch. The same list of operations, shown above, will be executed. The AM will request new containers for its SamzaContainers, and proceed as though it has just started for the first time. YARN has a yarn.resourcemanager.am.max-retries configuration parameter that’s defined in <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml";>yarn-site.xml</a>. This configuration defaults to 1, which means that, by default, a single AM failure will cause your Samza job to stop running.</p> -<h3 id="dashboard">Dashboard</h3> - -<p>Samza’s ApplicationMaster comes with a dashboard to show useful information such as:</p> - -<ol> -<li>Where containers are located.</li> -<li>Links to logs.</li> -<li>The Samza job’s configuration.</li> -<li>Container failure count.</li> -</ol> - -<p>You can find this dashboard by going to your YARN grid’s ResourceManager page (usually something like <a href="http://localhost:8088/cluster";>http://localhost:8088/cluster</a>), and clicking on the “ApplicationMaster” link of a running Samza job.</p> - -<p><img src="/img/latest/learn/documentation/yarn/samza-am-dashboard.png" alt="Screenshot of ApplicationMaster dashboard" class="diagram-large"></p> - <h3 id="security">Security</h3> <p>The Samza dashboard’s HTTP access is currently un-secured, even when using YARN in secure-mode. This means that users with access to a YARN grid could port-scan a Samza ApplicationMaster’s HTTP server, and open the dashboard in a browser to view its contents. Sensitive configuration can be viewed by anyone, in this way, and care should be taken. There are plans to secure Samza’s ApplicationMaster using <a href="http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.0/bk_installing_manually_book/content/rpm-chap14-2-3-1.html";>Hadoop’s security</a> features (<a href="http://en.wikipedia.org/wiki/SPNEGO";>SPENAGO</a>).</p> @@ -211,7 +198,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/yarn/application-master.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/yarn/application-master.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } }); Modified: samza/site/learn/documentation/latest/yarn/isolation.html URL: http://svn.apache.org/viewvc/samza/site/learn/documentation/latest/yarn/isolation.html?rev=1721446&r1=1721445&r2=1721446&view=diff ============================================================================== --- samza/site/learn/documentation/latest/yarn/isolation.html (original) +++ samza/site/learn/documentation/latest/yarn/isolation.html Tue Dec 22 19:03:17 2015 @@ -46,7 +46,7 @@ - <a href="http://samza.apache.org/learn/documentation/0.9/yarn/isolation.html";><i id="switch-version-button"></i></a> + <a href="http://samza.apache.org/learn/documentation/0.10/yarn/isolation.html";><i id="switch-version-button"></i></a> <!-- links for the navigation bar --> @@ -99,6 +99,7 @@ <h1><i class="fa fa-history"></i> Archive</h1> <ul> <li><a href="/archive/index.html#latest">latest</a></li> + <li><a href="/archive/index.html#10">0.10</a></li> <li><a href="/archive/index.html#09">0.9</a></li> <li><a href="/archive/index.html#08">0.8</a></li> <li><a href="/archive/index.html#07">0.7</a></li> @@ -184,7 +185,7 @@ <script> $( document ).ready(function() { - if ( $.fn.urlExists( "/learn/documentation/0.9/yarn/isolation.html" ) ) { + if ( $.fn.urlExists( "/learn/documentation/0.10/yarn/isolation.html" ) ) { $("#switch-version-button").addClass("fa fa-history masthead-icon"); } });
