Repository: kafka Updated Branches: refs/heads/trunk 7d6f6f732 -> 3e2fe17c0
http://git-wip-us.apache.org/repos/asf/kafka/blob/3e2fe17c/docs/streams/developer-guide/running-app.html ---------------------------------------------------------------------- diff --git a/docs/streams/developer-guide/running-app.html b/docs/streams/developer-guide/running-app.html new file mode 100644 index 0000000..1bed1d0 --- /dev/null +++ b/docs/streams/developer-guide/running-app.html @@ -0,0 +1,197 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<script><!--#include virtual="../../js/templateData.js" --></script> + +<script id="content-template" type="text/x-handlebars-template"> + <!-- h1>Developer Guide for Kafka Streams</h1 --> + <div class="sub-nav-sticky"> + <div class="sticky-top"> + <!-- div style="height:35px"> + <a href="/{{version}}/documentation/streams/">Introduction</a> + <a class="active-menu-item" href="/{{version}}/documentation/streams/developer-guide">Developer Guide</a> + <a href="/{{version}}/documentation/streams/core-concepts">Concepts</a> + <a href="/{{version}}/documentation/streams/quickstart">Run Demo App</a> + <a href="/{{version}}/documentation/streams/tutorial">Tutorial: Write App</a> + </div --> + </div> + </div> + + <div class="section" id="running-streams-applications"> +<span id="streams-developer-guide-execution"></span><h1>Running Streams Applications<a class="headerlink" href="#running-streams-applications" title="Permalink to this headline"></a></h1> +<p>You can run Java applications that use the Kafka Streams library without any additional configuration or requirements.</p> +<div class="contents local topic" id="table-of-contents"> +<p class="topic-title first">Table of Contents</p> +<ul class="simple"> +<li><a class="reference internal" href="#starting-a-kafka-streams-application" id="id3">Starting a Kafka Streams application</a></li> +<li><a class="reference internal" href="#elastic-scaling-of-your-application" id="id4">Elastic scaling of your application</a><ul> +<li><a class="reference internal" href="#adding-capacity-to-your-application" id="id5">Adding capacity to your application</a></li> +<li><a class="reference internal" href="#removing-capacity-from-your-application" id="id6">Removing capacity from your application</a></li> +<li><a class="reference internal" href="#state-restoration-during-workload-rebalance" id="id7">State restoration during workload rebalance</a></li> +<li><a class="reference internal" href="#determining-how-many-application-instances-to-run" id="id8">Determining how many application instances to run</a></li> +</ul> +</li> +</ul> +</div> + <div class="section" id="running-streams-applications"> + <span id="streams-developer-guide-execution"></span><h1>Running Streams Applications<a class="headerlink" href="#running-streams-applications" title="Permalink to this headline"></a></h1> + <p>You can run Java applications that use the Kafka Streams library without any additional configuration or requirements. Kafka Streams + also provides the ability to receive notification of the various states of the application. The ability to monitor the runtime + status is discussed in <a class="reference internal" href="../monitoring.html#streams-monitoring"><span class="std std-ref">the monitoring guide</span></a>.</p> + <div class="contents local topic" id="table-of-contents"> + <p class="topic-title first">Table of Contents</p> + <ul class="simple"> + <li><a class="reference internal" href="#starting-a-kafka-streams-application" id="id3">Starting a Kafka Streams application</a></li> + <li><a class="reference internal" href="#elastic-scaling-of-your-application" id="id4">Elastic scaling of your application</a><ul> + <li><a class="reference internal" href="#adding-capacity-to-your-application" id="id5">Adding capacity to your application</a></li> + <li><a class="reference internal" href="#removing-capacity-from-your-application" id="id6">Removing capacity from your application</a></li> + <li><a class="reference internal" href="#state-restoration-during-workload-rebalance" id="id7">State restoration during workload rebalance</a></li> + <li><a class="reference internal" href="#determining-how-many-application-instances-to-run" id="id8">Determining how many application instances to run</a></li> + </ul> + </li> + </ul> + </div> + <div class="section" id="starting-a-kafka-streams-application"> + <span id="streams-developer-guide-execution-starting"></span><h2><a class="toc-backref" href="#id3">Starting a Kafka Streams application</a><a class="headerlink" href="#starting-a-kafka-streams-application" title="Permalink to this headline"></a></h2> + <p>You can package your Java application as a fat JAR file and then start the application like this:</p> + <div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># Start the application in class `com.example.MyStreamsApp`</span> +<span class="c1"># from the fat JAR named `path-to-app-fatjar.jar`.</span> +$ java -cp path-to-app-fatjar.jar com.example.MyStreamsApp +</pre></div> + </div> + <p>For more information about how you can package your application in this way, see the + <a class="reference internal" href="../code-examples.html#streams-code-examples"><span class="std std-ref">Streams code examples</span></a>.</p> + <p>When you start your application you are launching a Kafka Streams instance of your application. You can run multiple + instances of your application. A common scenario is that there are multiple instances of your application running in + parallel. For more information, see <a class="reference internal" href="../architecture.html#streams-architecture-parallelism-model"><span class="std std-ref">Parallelism Model</span></a>.</p> + <p>When the application instance starts running, the defined processor topology will be initialized as one or more stream tasks. + If the processor topology defines any state stores, these are also constructed during the initialization period. For + more information, see the <a class="reference internal" href="#streams-developer-guide-execution-scaling-state-restoration"><span class="std std-ref">State restoration during workload rebalance</span></a> section).</p> + </div> + <div class="section" id="elastic-scaling-of-your-application"> + <span id="streams-developer-guide-execution-scaling"></span><h2><a class="toc-backref" href="#id4">Elastic scaling of your application</a><a class="headerlink" href="#elastic-scaling-of-your-application" title="Permalink to this headline"></a></h2> + <p>Kafka Streams makes your stream processing applications elastic and scalable. You can add and remove processing capacity + dynamically during application runtime without any downtime or data loss. This makes your applications + resilient in the face of failures and for allows you to perform maintenance as needed (e.g. rolling upgrades).</p> + <p>For more information about this elasticity, see the <a class="reference internal" href="../architecture.html#streams-architecture-parallelism-model"><span class="std std-ref">Parallelism Model</span></a> section. Kafka Streams + leverages the Kafka group management functionality, which is built right into the <a class="reference external" href="https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol";>Kafka wire protocol</a>. It is the foundation that enables the + elasticity of Kafka Streams applications: members of a group coordinate and collaborate jointly on the consumption and + processing of data in Kafka. Additionally, Kafka Streams provides stateful processing and allows for fault-tolerant + state in environments where application instances may come and go at any time.</p> + <div class="section" id="adding-capacity-to-your-application"> + <h3><a class="toc-backref" href="#id5">Adding capacity to your application</a><a class="headerlink" href="#adding-capacity-to-your-application" title="Permalink to this headline"></a></h3> + <p>If you need more processing capacity for your stream processing application, you can simply start another instance of your stream processing application, e.g. on another machine, in order to scale out. The instances of your application will become aware of each other and automatically begin to share the processing work. More specifically, what will be handed over from the existing instances to the new instances is (some of) the stream tasks that have been run by the existing instances. Moving stream tasks from one instance to another results in moving the processing work plus any internal state of these stream tasks (the state of a stream task will be re-created in the target instance by restoring the state from its corresponding changelog topic).</p> + <p>The various instances of your application each run in their own JVM process, which means that each instance can leverage all the processing capacity that is available to their respective JVM process (minus the capacity that any non-Kafka-Streams part of your application may be using). This explains why running additional instances will grant your application additional processing capacity. The exact capacity you will be adding by running a new instance depends of course on the environment in which the new instance runs: available CPU cores, available main memory and Java heap space, local storage, network bandwidth, and so on. Similarly, if you stop any of the running instances of your application, then you are removing and freeing up the respective processing capacity.</p> + <div class="figure align-center" id="id1"> + <a class="reference internal image-reference" href="../../../images/streams-elastic-scaling-1.png"><img alt="../../../images/streams-elastic-scaling-1.png" src="../../../images/streams-elastic-scaling-1.png" style="width: 500pt; height: 400pt;" /></a> + <p class="caption"><span class="caption-text">Before adding capacity: only a single instance of your Kafka Streams application is running. At this point the corresponding Kafka consumer group of your application contains only a single member (this instance). All data is being read and processed by this single instance.</span></p> + </div> + <div class="figure align-center" id="id2"> + <a class="reference internal image-reference" href="../../../images/streams-elastic-scaling-2.png"><img alt="../../../images/streams-elastic-scaling-2.png" src="../../../images/streams-elastic-scaling-2.png" style="width: 500pt; height: 400pt;" /></a> + <p class="caption"><span class="caption-text">After adding capacity: now two additional instances of your Kafka Streams application are running, and they have automatically joined the application’s Kafka consumer group for a total of three current members. These three instances are automatically splitting the processing work between each other. The splitting is based on the Kafka topic partitions from which data is being read.</span></p> + </div> + </div> + <div class="section" id="removing-capacity-from-your-application"> + <h3><a class="toc-backref" href="#id6">Removing capacity from your application</a><a class="headerlink" href="#removing-capacity-from-your-application" title="Permalink to this headline"></a></h3> + <p>To remove processing capacity, you can stop running stream processing application instances (e.g., shut down two of + the four instances), it will automatically leave the applicationâs consumer group, and the remaining instances of + your application will automatically take over the processing work. The remaining instances take over the stream tasks that + were run by the stopped instances. Moving stream tasks from one instance to another results in moving the processing + work plus any internal state of these stream tasks. The state of a stream task is recreated in the target instance + from its changelog topic.</p> + <div class="figure align-center"> + <a class="reference internal image-reference" href="../../../images/streams-elastic-scaling-3.png"><img alt="../../../images/streams-elastic-scaling-3.png" src="../../../images/streams-elastic-scaling-3.png" style="width: 500pt; height: 400pt;" /></a> + </div> + </div> + <div class="section" id="state-restoration-during-workload-rebalance"> + <span id="streams-developer-guide-execution-scaling-state-restoration"></span><h3><a class="toc-backref" href="#id7">State restoration during workload rebalance</a><a class="headerlink" href="#state-restoration-during-workload-rebalance" title="Permalink to this headline"></a></h3> + <p>When a task is migrated, the task processing state is fully restored before the application instance resumes + processing. This guarantees the correct processing results. In Kafka Streams, state restoration is usually done by + replaying the corresponding changelog topic to reconstruct the state store. To minimize changelog-based restoration + latency by using replicated local state stores, you can specify <code class="docutils literal"><span class="pre">num.standby.replicas</span></code>. When a stream task is + initialized or re-initialized on the application instance, its state store is restored like this:</p> + <ul class="simple"> + <li>If no local state store exists, the changelog is replayed from the earliest to the current offset. This reconstructs the local state store to the most recent snapshot.</li> + <li>If a local state store exists, the changelog is replayed from the previously checkpointed offset. The changes are applied and the state is restored to the most recent snapshot. This method takes less time because it is applying a smaller portion of the changelog.</li> + </ul> + <p>For more information, see <a class="reference internal" href="config-streams.html#streams-developer-guide-standby-replicas"><span class="std std-ref">Standby Replicas</span></a>.</p> + </div> + <div class="section" id="determining-how-many-application-instances-to-run"> + <h3><a class="toc-backref" href="#id8">Determining how many application instances to run</a><a class="headerlink" href="#determining-how-many-application-instances-to-run" title="Permalink to this headline"></a></h3> + <p>The parallelism of a Kafka Streams application is primarily determined by how many partitions the input topics have. For + example, if your application reads from a single topic that has ten partitions, then you can run up to ten instances + of your applications. You can run further instances, but these will be idle.</p> + <p>The number of topic partitions is the upper limit for the parallelism of your Kafka Streams application and for the + number of running instances of your application.</p> + <p>To achieve balanced workload processing across application instances and to prevent processing hotpots, you should + distribute data and processing workloads:</p> + <ul class="simple"> + <li>Data should be equally distributed across topic partitions. For example, if two topic partitions each have 1 million messages, this is better than a single partition with 2 million messages and none in the other.</li> + <li>Processing workload should be equally distributed across topic partitions. For example, if the time to process messages varies widely, then it is better to spread the processing-intensive messages across partitions rather than storing these messages within the same partition.</li> + </ul> +</div> +</div> +</div> + + + </div> + </div> + <div class="pagination"> + <a href="/{{version}}/documentation/streams/developer-guide/memory-mgmt" class="pagination__btn pagination__btn__prev">Previous</a> + <a href="/{{version}}/documentation/streams/developer-guide/manage-topics" class="pagination__btn pagination__btn__next">Next</a> + </div> + </script> + + <!--#include virtual="../../../includes/_header.htm" --> + <!--#include virtual="../../../includes/_top.htm" --> + <div class="content documentation documentation--current"> + <!--#include virtual="../../../includes/_nav.htm" --> + <div class="right"> + <!--#include virtual="../../../includes/_docs_banner.htm" --> + <ul class="breadcrumbs"> + <li><a href="/documentation">Documentation</a></li> + <li><a href="/documentation/streams">Kafka Streams</a></li> + <li><a href="/documentation/streams/developer-guide/">Developer Guide</a></li> + </ul> + <div class="p-content"></div> + </div> + </div> + <!--#include virtual="../../../includes/_footer.htm" --> + <script> + $(function() { + // Show selected style on nav item + $('.b-nav__streams').addClass('selected'); + + //sticky secondary nav + var $navbar = $(".sub-nav-sticky"), + y_pos = $navbar.offset().top, + height = $navbar.height(); + + $(window).scroll(function() { + var scrollTop = $(window).scrollTop(); + + if (scrollTop > y_pos - height) { + $navbar.addClass("navbar-fixed") + } else if (scrollTop <= y_pos) { + $navbar.removeClass("navbar-fixed") + } + }); + + // Display docs subnav items + $('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded'); + }); + </script> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/kafka/blob/3e2fe17c/docs/streams/developer-guide/security.html ---------------------------------------------------------------------- diff --git a/docs/streams/developer-guide/security.html b/docs/streams/developer-guide/security.html new file mode 100644 index 0000000..92aa0f5 --- /dev/null +++ b/docs/streams/developer-guide/security.html @@ -0,0 +1,176 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<script><!--#include virtual="../../js/templateData.js" --></script> + +<script id="content-template" type="text/x-handlebars-template"> + <!-- h1>Developer Guide for Kafka Streams</h1 --> + <div class="sub-nav-sticky"> + <div class="sticky-top"> + <!-- div style="height:35px"> + <a href="/{{version}}/documentation/streams/">Introduction</a> + <a class="active-menu-item" href="/{{version}}/documentation/streams/developer-guide">Developer Guide</a> + <a href="/{{version}}/documentation/streams/core-concepts">Concepts</a> + <a href="/{{version}}/documentation/streams/quickstart">Run Demo App</a> + <a href="/{{version}}/documentation/streams/tutorial">Tutorial: Write App</a> + </div --> + </div> + </div> + + <div class="section" id="streams-security"> + <span id="streams-developer-guide-security"></span><h1>Streams Security<a class="headerlink" href="#streams-security" title="Permalink to this headline"></a></h1> + <div class="contents local topic" id="table-of-contents"> + <p class="topic-title first">Table of Contents</p> + <ul class="simple"> + <li><a class="reference internal" href="#required-acl-setting-for-secure-kafka-clusters" id="id1">Required ACL setting for secure Kafka clusters</a></li> + <li><a class="reference internal" href="#security-example" id="id2">Security example</a></li> + </ul> + </div> + <p>Kafka Streams natively integrates with the <a class="reference internal" href="../../kafka/security.html#kafka-security"><span class="std std-ref">Kafka’s security features</span></a> and supports all of the + client-side security features in Kafka. Streams leverages the <a class="reference internal" href="../../clients/index.html#kafka-clients"><span class="std std-ref">Java Producer and Consumer API</span></a>.</p> + <p>To secure your Stream processing applications, configure the security settings in the corresponding Kafka producer + and consumer clients, and then specify the corresponding configuration settings in your Kafka Streams application.</p> + <p>Kafka supports cluster encryption and authentication, including a mix of authenticated and unauthenticated, + and encrypted and non-encrypted clients. Using security is optional.</p> + <p>Here a few relevant client-side security features:</p> + <dl class="docutils"> + <dt>Encrypt data-in-transit between your applications and Kafka brokers</dt> + <dd>You can enable the encryption of the client-server communication between your applications and the Kafka brokers. + For example, you can configure your applications to always use encryption when reading and writing data to and from + Kafka. This is critical when reading and writing data across security domains such as internal network, public + internet, and partner networks.</dd> + <dt>Client authentication</dt> + <dd>You can enable client authentication for connections from your application to Kafka brokers. For example, you can + define that only specific applications are allowed to connect to your Kafka cluster.</dd> + <dt>Client authorization</dt> + <dd>You can enable client authorization of read and write operations by your applications. For example, you can define + that only specific applications are allowed to read from a Kafka topic. You can also restrict write access to Kafka + topics to prevent data pollution or fraudulent activities.</dd> + </dl> + <p>For more information about the security features in Apache Kafka, see <a class="reference internal" href="../../kafka/security.html#kafka-security"><span class="std std-ref">Kafka Security</span></a>.</p> + <div class="section" id="required-acl-setting-for-secure-kafka-clusters"> + <span id="streams-developer-guide-security-acls"></span><h2><a class="toc-backref" href="#id1">Required ACL setting for secure Kafka clusters</a><a class="headerlink" href="#required-acl-setting-for-secure-kafka-clusters" title="Permalink to this headline"></a></h2> + <p>When applications are run against a secured Kafka cluster, the principal running the application must have the ACL + <code class="docutils literal"><span class="pre">--cluster</span> <span class="pre">--operation</span> <span class="pre">Create</span></code> set so that the application has the permissions to create + <a class="reference internal" href="manage-topics.html#streams-developer-guide-topics-internal"><span class="std std-ref">internal topics</span></a>.</p> + </div> + <div class="section" id="security-example"> + <span id="streams-developer-guide-security-example"></span><h2><a class="toc-backref" href="#id2">Security example</a><a class="headerlink" href="#security-example" title="Permalink to this headline"></a></h2> + <p>The purpose is to configure a Kafka Streams application to enable client authentication and encrypt data-in-transit when + communicating with its Kafka cluster.</p> + <p>This example assumes that the Kafka brokers in the cluster already have their security setup and that the necessary SSL + certificates are available to the application in the local filesystem locations. For example, if you are using Docker + then you must also include these SSL certificates in the correct locations within the Docker image.</p> + <p>The snippet below shows the settings to enable client authentication and SSL encryption for data-in-transit between your + Kafka Streams application and the Kafka cluster it is reading and writing from:</p> + <div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># Essential security settings to enable client authentication and SSL encryption</span> +bootstrap.servers<span class="o">=</span>kafka.example.com:9093 +security.protocol<span class="o">=</span>SSL +ssl.truststore.location<span class="o">=</span>/etc/security/tls/kafka.client.truststore.jks +ssl.truststore.password<span class="o">=</span>test1234 +ssl.keystore.location<span class="o">=</span>/etc/security/tls/kafka.client.keystore.jks +ssl.keystore.password<span class="o">=</span>test1234 +ssl.key.password<span class="o">=</span>test1234 +</pre></div> + </div> + <p>Configure these settings in the application for your <code class="docutils literal"><span class="pre">StreamsConfig</span></code> instance. These settings will encrypt any + data-in-transit that is being read from or written to Kafka, and your application will authenticate itself against the + Kafka brokers that it is communicating with. Note that this example does not cover client authorization.</p> + <div class="highlight-java"><div class="highlight"><pre><span></span><span class="c1">// Code of your Java application that uses the Kafka Streams library</span> +<span class="n">Properties</span> <span class="n">settings</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Properties</span><span class="o">();</span> +<span class="n">settings</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">StreamsConfig</span><span class="o">.</span><span class="na">APPLICATION_ID_CONFIG</span><span class="o">,</span> <span class="s">"secure-kafka-streams-app"</span><span class="o">);</span> +<span class="c1">// Where to find secure Kafka brokers. Here, it's on port 9093.</span> +<span class="n">settings</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">StreamsConfig</span><span class="o">.</span><span class="na">BOOTSTRAP_SERVERS_CONFIG</span><span class="o">,</span> <span class="s">"kafka.example.com:9093"</span><span class="o">);</span> +<span class="c1">//</span> +<span class="c1">// ...further non-security related settings may follow here...</span> +<span class="c1">//</span> +<span class="c1">// Security settings.</span> +<span class="c1">// 1. These settings must match the security settings of the secure Kafka cluster.</span> +<span class="c1">// 2. The SSL trust store and key store files must be locally accessible to the application.</span> +<span class="n">settings</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">CommonClientConfigs</span><span class="o">.</span><span class="na">SECURITY_PROTOCOL_CONFIG</span><span class="o">,</span> <span class="s">"SSL"</span><span class="o">);</span> +<span class="n">settings</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">SslConfigs</span><span class="o">.</span><span class="na">SSL_TRUSTSTORE_LOCATION_CONFIG</span><span class="o">,</span> <span class="s">"/etc/security/tls/kafka.client.truststore.jks"</span><span class="o">);</span> +<span class="n">settings</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">SslConfigs</span><span class="o">.</span><span class="na">SSL_TRUSTSTORE_PASSWORD_CONFIG</span><span class="o">,</span> <span class="s">"test1234"</span><span class="o">);</span> +<span class="n">settings</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">SslConfigs</span><span class="o">.</span><span class="na">SSL_KEYSTORE_LOCATION_CONFIG</span><span class="o">,</span> <span class="s">"/etc/security/tls/kafka.client.keystore.jks"</span><span class="o">);</span> +<span class="n">settings</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">SslConfigs</span><span class="o">.</span><span class="na">SSL_KEYSTORE_PASSWORD_CONFIG</span><span class="o">,</span> <span class="s">"test1234"</span><span class="o">);</span> +<span class="n">settings</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">SslConfigs</span><span class="o">.</span><span class="na">SSL_KEY_PASSWORD_CONFIG</span><span class="o">,</span> <span class="s">"test1234"</span><span class="o">);</span> +<span class="n">StreamsConfig</span> <span class="n">streamsConfiguration</span> <span class="o">=</span> <span class="k">new</span> <span class="n">StreamsConfig</span><span class="o">(</span><span class="n">settings</span><span class="o">);</span> +</pre></div> + </div> + <p>If you incorrectly configure a security setting in your application, it will fail at runtime, typically right after you + start it. For example, if you enter an incorrect password for the <code class="docutils literal"><span class="pre">ssl.keystore.password</span></code> setting, an error message + similar to this would be logged and then the application would terminate:</p> + <div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># Misconfigured ssl.keystore.password</span> +Exception in thread <span class="s2">"main"</span> org.apache.kafka.common.KafkaException: Failed to construct kafka producer +<span class="o">[</span>...snip...<span class="o">]</span> +Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: + java.io.IOException: Keystore was tampered with, or password was incorrect +<span class="o">[</span>...snip...<span class="o">]</span> +Caused by: java.security.UnrecoverableKeyException: Password verification failed +</pre></div> + </div> + <p>Monitor your Kafka Streams application log files for such error messages to spot any misconfigured applications quickly.</p> +</div> +</div> + + + </div> + </div> + <div class="pagination"> + <a href="/{{version}}/documentation/streams/developer-guide/manage-topics" class="pagination__btn pagination__btn__prev">Previous</a> + <a href="/{{version}}/documentation/streams/developer-guide/app-reset-tool" class="pagination__btn pagination__btn__next">Next</a> + </div> + </script> + + <!--#include virtual="../../../includes/_header.htm" --> + <!--#include virtual="../../../includes/_top.htm" --> + <div class="content documentation documentation--current"> + <!--#include virtual="../../../includes/_nav.htm" --> + <div class="right"> + <!--#include virtual="../../../includes/_docs_banner.htm" --> + <ul class="breadcrumbs"> + <li><a href="/documentation">Documentation</a></li> + <li><a href="/documentation/streams">Kafka Streams</a></li> + <li><a href="/documentation/streams/developer-guide/">Developer Guide</a></li> + </ul> + <div class="p-content"></div> + </div> + </div> + <!--#include virtual="../../../includes/_footer.htm" --> + <script> + $(function() { + // Show selected style on nav item + $('.b-nav__streams').addClass('selected'); + + //sticky secondary nav + var $navbar = $(".sub-nav-sticky"), + y_pos = $navbar.offset().top, + height = $navbar.height(); + + $(window).scroll(function() { + var scrollTop = $(window).scrollTop(); + + if (scrollTop > y_pos - height) { + $navbar.addClass("navbar-fixed") + } else if (scrollTop <= y_pos) { + $navbar.removeClass("navbar-fixed") + } + }); + + // Display docs subnav items + $('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded'); + }); + </script> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/kafka/blob/3e2fe17c/docs/streams/developer-guide/write-streams.html ---------------------------------------------------------------------- diff --git a/docs/streams/developer-guide/write-streams.html b/docs/streams/developer-guide/write-streams.html new file mode 100644 index 0000000..f262b50 --- /dev/null +++ b/docs/streams/developer-guide/write-streams.html @@ -0,0 +1,198 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<script><!--#include virtual="../../js/templateData.js" --></script> + +<script id="content-template" type="text/x-handlebars-template"> + <!-- h1>Developer Guide for Kafka Streams</h1 --> + <div class="sub-nav-sticky"> + <!-- div class="sticky-top"> + <div style="height:35px"> + <a href="/{{version}}/documentation/streams/">Introduction</a> + <a class="active-menu-item" href="/{{version}}/documentation/streams/developer-guide">Developer Guide</a> + <a href="/{{version}}/documentation/streams/core-concepts">Concepts</a> + <a href="/{{version}}/documentation/streams/quickstart">Run Demo App</a> + <a href="/{{version}}/documentation/streams/tutorial">Tutorial: Write App</a> + </div> + </div --> + </div> + + <div class="section" id="writing-a-streams-application"> + <span id="streams-write-app"></span><h1>Writing a Streams Application<a class="headerlink" href="#writing-a-streams-application" title="Permalink to this headline"></a></h1> + <p class="topic-title first">Table of Contents</p> + <ul class="simple"> + <li><a class="reference internal" href="#libraries-and-maven-artifacts" id="id1">Libraries and Maven artifacts</a></li> + <li><a class="reference internal" href="#using-kafka-streams-within-your-application-code" id="id2">Using Kafka Streams within your application code</a></li> + </ul> + <p>Any Java application that makes use of the Kafka Streams library is considered a Kafka Streams application. + The computational logic of a Kafka Streams application is defined as a <a class="reference internal" href="../concepts.html#streams-concepts"><span class="std std-ref">processor topology</span></a>, + which is a graph of stream processors (nodes) and streams (edges).</p> + <p>You can define the processor topology with the Kafka Streams APIs:</p> + <dl class="docutils"> + <dt><a class="reference internal" href="dsl-api.html#streams-developer-guide-dsl"><span class="std std-ref">Kafka Streams DSL</span></a></dt> + <dd>A high-level API that provides provides the most common data transformation operations such as <code class="docutils literal"><span class="pre">map</span></code>, <code class="docutils literal"><span class="pre">filter</span></code>, <code class="docutils literal"><span class="pre">join</span></code>, and <code class="docutils literal"><span class="pre">aggregations</span></code> out of the box. The DSL is the recommended starting point for developers new to Kafka Streams, and should cover many use cases and stream processing needs.</dd> + <dt><a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a></dt> + <dd>A low-level API that lets you add and connect processors as well as interact directly with state stores. The Processor API provides you with even more flexibility than the DSL but at the expense of requiring more manual work on the side of the application developer (e.g., more lines of code).</dd> + </dl> + <div class="section" id="using-kafka-streams-within-your-application-code"> + <h2>Using Kafka Streams within your application code<a class="headerlink" href="#using-kafka-streams-within-your-application-code" title="Permalink to this headline"></a></h2> + <p>You can call Kafka Streams from anywhere in your application code, but usually these calls are made within the <code class="docutils literal"><span class="pre">main()</span></code> method of + your application, or some variant thereof. The basic elements of defining a processing topology within your application + are described below.</p> + <p>First, you must create an instance of <code class="docutils literal"><span class="pre">KafkaStreams</span></code>.</p> + <ul class="simple"> + <li>The first argument of the <code class="docutils literal"><span class="pre">KafkaStreams</span></code> constructor takes a topology (either <code class="docutils literal"><span class="pre">StreamsBuilder#build()</span></code> for the + <a class="reference internal" href="dsl-api.html#streams-developer-guide-dsl"><span class="std std-ref">DSL</span></a> or <code class="docutils literal"><span class="pre">Topology</span></code> for the + <a class="reference internal" href="processor-api.html#streams-developer-guide-processor-api"><span class="std std-ref">Processor API</span></a>) that is used to define a topology.</li> + <li>The second argument is an instance of <code class="docutils literal"><span class="pre">StreamsConfig</span></code>, which defines the configuration for this specific topology.</li> + </ul> + <p>Code example:</p> + <div class="highlight-java"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">org.apache.kafka.streams.KafkaStreams</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.apache.kafka.streams.StreamsConfig</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.apache.kafka.streams.kstream.StreamsBuilder</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.apache.kafka.streams.processor.Topology</span><span class="o">;</span> + +<span class="c1">// Use the builders to define the actual processing topology, e.g. to specify</span> +<span class="c1">// from which input topics to read, which stream operations (filter, map, etc.)</span> +<span class="c1">// should be called, and so on. We will cover this in detail in the subsequent</span> +<span class="c1">// sections of this Developer Guide.</span> + +<span class="n">StreamsBuilder</span> <span class="n">builder</span> <span class="o">=</span> <span class="o">...;</span> <span class="c1">// when using the DSL</span> +<span class="n">Topology</span> <span class="n">topology</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">build</span><span class="o">();</span> +<span class="c1">//</span> +<span class="c1">// OR</span> +<span class="c1">//</span> +<span class="n">Topology</span> <span class="n">topology</span> <span class="o">=</span> <span class="o">...;</span> <span class="c1">// when using the Processor API</span> + +<span class="c1">// Use the configuration to tell your application where the Kafka cluster is,</span> +<span class="c1">// which Serializers/Deserializers to use by default, to specify security settings,</span> +<span class="c1">// and so on.</span> +<span class="n">StreamsConfig</span> <span class="n">config</span> <span class="o">=</span> <span class="o">...;</span> + +<span class="n">KafkaStreams</span> <span class="n">streams</span> <span class="o">=</span> <span class="k">new</span> <span class="n">KafkaStreams</span><span class="o">(</span><span class="n">topology</span><span class="o">,</span> <span class="n">config</span><span class="o">);</span> +</pre></div> + </div> + <p>At this point, internal structures are initialized, but the processing is not started yet. + You have to explicitly start the Kafka Streams thread by calling the <code class="docutils literal"><span class="pre">KafkaStreams#start()</span></code> method:</p> + <div class="highlight-java"><div class="highlight"><pre><span></span><span class="c1">// Start the Kafka Streams threads</span> +<span class="n">streams</span><span class="o">.</span><span class="na">start</span><span class="o">();</span> +</pre></div> + </div> + <p>If there are other instances of this stream processing application running elsewhere (e.g., on another machine), Kafka + Streams transparently re-assigns tasks from the existing instances to the new instance that you just started. + For more information, see <a class="reference internal" href="../architecture.html#streams-architecture-tasks"><span class="std std-ref">Stream Partitions and Tasks</span></a> and <a class="reference internal" href="../architecture.html#streams-architecture-threads"><span class="std std-ref">Threading Model</span></a>.</p> + <p>To catch any unexpected exceptions, you can set an <code class="docutils literal"><span class="pre">java.lang.Thread.UncaughtExceptionHandler</span></code> before you start the + application. This handler is called whenever a stream thread is terminated by an unexpected exception:</p> + <div class="highlight-java"><div class="highlight"><pre><span></span><span class="c1">// Java 8+, using lambda expressions</span> +<span class="n">streams</span><span class="o">.</span><span class="na">setUncaughtExceptionHandler</span><span class="o">((</span><span class="n">Thread</span> <span class="n">thread</span><span class="o">,</span> <span class="n">Throwable</span> <span class="n">throwable</span><span class="o">)</span> <span class="o">-></span> <span class="o">{</span> + <span class="c1">// here you should examine the throwable/exception and perform an appropriate action!</span> +<span class="o">});</span> + + +<span class="c1">// Java 7</span> +<span class="n">streams</span><span class="o">.</span><span class="na">setUncaughtExceptionHandler</span><span class="o">(</span><span class="k">new</span> <span class="n">Thread</span><span class="o">.</span><span class="na">UncaughtExceptionHandler</span><span class="o">()</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">uncaughtException</span><span class="o">(</span><span class="n">Thread</span> <span class="n">thread</span><span class="o">,</span> <span class="n">Throwable</span> <span class="n">throwable</span><span class="o">)</span> <span class="o">{</span> + <span class="c1">// here you should examine the throwable/exception and perform an appropriate action!</span> + <span class="o">}</span> +<span class="o">});</span> +</pre></div> + </div> + <p>To stop the application instance, call the <code class="docutils literal"><span class="pre">KafkaStreams#close()</span></code> method:</p> + <div class="highlight-java"><div class="highlight"><pre><span></span><span class="c1">// Stop the Kafka Streams threads</span> +<span class="n">streams</span><span class="o">.</span><span class="na">close</span><span class="o">();</span> +</pre></div> + </div> + <p>To allow your application to gracefully shutdown in response to SIGTERM, it is recommended that you add a shutdown hook + and call <code class="docutils literal"><span class="pre">KafkaStreams#close</span></code>.</p> + <ul> + <li><p class="first">Here is a shutdown hook example in Java 8+:</p> + <blockquote> + <div><div class="highlight-java"><div class="highlight"><pre><span></span><span class="c1">// Add shutdown hook to stop the Kafka Streams threads.</span> +<span class="c1">// You can optionally provide a timeout to `close`.</span> +<span class="n">Runtime</span><span class="o">.</span><span class="na">getRuntime</span><span class="o">().</span><span class="na">addShutdownHook</span><span class="o">(</span><span class="k">new</span> <span class="n">Thread</span><span class="o">(</span><span class="n">streams</span><span class="o">::</span><span class="n">close</span><span class="o">));</span> +</pre></div> + </div> + </div></blockquote> + </li> + <li><p class="first">Here is a shutdown hook example in Java 7:</p> + <blockquote> + <div><div class="highlight-java"><div class="highlight"><pre><span></span><span class="c1">// Add shutdown hook to stop the Kafka Streams threads.</span> +<span class="c1">// You can optionally provide a timeout to `close`.</span> +<span class="n">Runtime</span><span class="o">.</span><span class="na">getRuntime</span><span class="o">().</span><span class="na">addShutdownHook</span><span class="o">(</span><span class="k">new</span> <span class="n">Thread</span><span class="o">(</span><span class="k">new</span> <span class="n">Runnable</span><span class="o">()</span> <span class="o">{</span> + <span class="nd">@Override</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">run</span><span class="o">()</span> <span class="o">{</span> + <span class="n">streams</span><span class="o">.</span><span class="na">close</span><span class="o">();</span> + <span class="o">}</span> +<span class="o">}));</span> +</pre></div> + </div> + </div></blockquote> + </li> + </ul> + <p>After an application is stopped, Kafka Streams will migrate any tasks that had been running in this instance to available remaining + instances.</p> +</div> +</div> + + + </div> + </div> + <div class="pagination"> + <a href="/{{version}}/documentation/streams/developer-guide/" class="pagination__btn pagination__btn__prev">Previous</a> + <a href="/{{version}}/documentation/streams/developer-guide/config-streams" class="pagination__btn pagination__btn__next">Next</a> + </div> +</script> + +<!--#include virtual="../../../includes/_header.htm" --> +<!--#include virtual="../../../includes/_top.htm" --> +<div class="content documentation documentation--current"> + <!--#include virtual="../../../includes/_nav.htm" --> + <div class="right"> + <!--#include virtual="../../../includes/_docs_banner.htm" --> + <ul class="breadcrumbs"> + <li><a href="/documentation">Documentation</a></li> + <li><a href="/documentation/streams">Kafka Streams</a></li> + <li><a href="/documentation/streams/developer-guide/">Developer Guide</a></li> + </ul> + <div class="p-content"></div> + </div> +</div> +<!--#include virtual="../../../includes/_footer.htm" --> +<script> + $(function() { + // Show selected style on nav item + $('.b-nav__streams').addClass('selected'); + + //sticky secondary nav + var $navbar = $(".sub-nav-sticky"), + y_pos = $navbar.offset().top, + height = $navbar.height(); + + $(window).scroll(function() { + var scrollTop = $(window).scrollTop(); + + if (scrollTop > y_pos - height) { + $navbar.addClass("navbar-fixed") + } else if (scrollTop <= y_pos) { + $navbar.removeClass("navbar-fixed") + } + }); + + // Display docs subnav items + $('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded'); + }); +</script> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/kafka/blob/3e2fe17c/docs/streams/index.html ---------------------------------------------------------------------- diff --git a/docs/streams/index.html b/docs/streams/index.html index ab72c87..cb25e29 100644 --- a/docs/streams/index.html +++ b/docs/streams/index.html @@ -16,12 +16,12 @@ <!--#include virtual="../js/templateData.js" --> </script> <script id="streams-template" type="text/x-handlebars-template"> - <h1>Kafka Streams API</h1> + <h1>Kafka Streams</h1> <div class="sub-nav-sticky"> <div class="sticky-top"> <div style="height:35px"> <a class="active-menu-item" href="/{{version}}/documentation/streams/">Introduction</a> - <a href="/{{version}}/documentation/streams/developer-guide">Developer Guide</a> + <a href="/{{version}}/documentation/streams/developer-guide/">Developer Guide</a> <a href="/{{version}}/documentation/streams/core-concepts">Concepts</a> <a href="/{{version}}/documentation/streams/quickstart">Run Demo App</a> <a href="/{{version}}/documentation/streams/tutorial">Tutorial: Write App</a> @@ -300,6 +300,7 @@ <!--#include virtual="../../includes/_docs_banner.htm" --> <ul class="breadcrumbs"> <li><a href="/documentation">Documentation</a> + <li><a href="/documentation/streams">Kafka Streams</a></li> </li> </ul> <div class="p-streams"></div> http://git-wip-us.apache.org/repos/asf/kafka/blob/3e2fe17c/docs/streams/quickstart.html ---------------------------------------------------------------------- diff --git a/docs/streams/quickstart.html b/docs/streams/quickstart.html index 314bce3..92ab885 100644 --- a/docs/streams/quickstart.html +++ b/docs/streams/quickstart.html @@ -23,7 +23,7 @@ <div class="sticky-top"> <div style="height:35px"> <a href="/{{version}}/documentation/streams/">Introduction</a> - <a href="/{{version}}/documentation/streams/developer-guide">Developer Guide</a> + <a href="/{{version}}/documentation/streams/developer-guide/">Developer Guide</a> <a href="/{{version}}/documentation/streams/core-concepts">Concepts</a> <a class="active-menu-item" href="/{{version}}/documentation/streams/quickstart">Run Demo App</a> <a href="/{{version}}/documentation/streams/tutorial">Tutorial: Write App</a> @@ -354,7 +354,7 @@ Looking beyond the scope of this concrete example, what Kafka Streams is doing h <!--#include virtual="../../includes/_docs_banner.htm" --> <ul class="breadcrumbs"> <li><a href="/documentation">Documentation</a></li> - <li><a href="/documentation/streams">Streams</a></li> + <li><a href="/documentation/streams">Kafka Streams</a></li> </ul> <div class="p-content"></div> </div> http://git-wip-us.apache.org/repos/asf/kafka/blob/3e2fe17c/docs/streams/tutorial.html ---------------------------------------------------------------------- diff --git a/docs/streams/tutorial.html b/docs/streams/tutorial.html index 71c9ca3..7d75ed6 100644 --- a/docs/streams/tutorial.html +++ b/docs/streams/tutorial.html @@ -22,7 +22,7 @@ <div class="sticky-top"> <div style="height:35px"> <a href="/{{version}}/documentation/streams/">Introduction</a> - <a href="/{{version}}/documentation/streams/developer-guide">Developer Guide</a> + <a href="/{{version}}/documentation/streams/developer-guide/">Developer Guide</a> <a href="/{{version}}/documentation/streams/core-concepts">Concepts</a> <a href="/{{version}}/documentation/streams/quickstart">Run Demo App</a> <a class="active-menu-item" href="/{{version}}/documentation/streams/tutorial">Tutorial: Write App</a> @@ -631,7 +631,7 @@ <!--#include virtual="../../includes/_docs_banner.htm" --> <ul class="breadcrumbs"> <li><a href="/documentation">Documentation</a></li> - <li><a href="/documentation/streams">Streams</a></li> + <li><a href="/documentation/streams">Kafka Streams</a></li> </ul> <div class="p-content"></div> </div> http://git-wip-us.apache.org/repos/asf/kafka/blob/3e2fe17c/docs/streams/upgrade-guide.html ---------------------------------------------------------------------- diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html index 2974058..d1726a7 100644 --- a/docs/streams/upgrade-guide.html +++ b/docs/streams/upgrade-guide.html @@ -373,7 +373,7 @@ <!--#include virtual="../../includes/_docs_banner.htm" --> <ul class="breadcrumbs"> <li><a href="/documentation">Documentation</a></li> - <li><a href="/documentation/streams">Kafka Streams API</a></li> + <li><a href="/documentation/streams">Kafka Streams</a></li> </ul> <div class="p-content"></div> </div>
