This is an automated email from the ASF dual-hosted git repository. guozhang pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push: new 27943e3 KAFKA-8219: add doc changes for static membership release (#6790) 27943e3 is described below commit 27943e39876efad87fce7b558af3419f1a7a025c Author: Boyang Chen <boy...@confluent.io> AuthorDate: Mon May 27 14:25:47 2019 -0700 KAFKA-8219: add doc changes for static membership release (#6790) * add doc changes for static membership release * address comments * address comments for a separate page * address Matthias' comment --- docs/design.html | 22 ++++++++++++++++++++++ docs/streams/upgrade-guide.html | 7 +++++++ docs/upgrade.html | 8 ++++++++ 3 files changed, 37 insertions(+) diff --git a/docs/design.html b/docs/design.html index eadf29d..4c4e796 100644 --- a/docs/design.html +++ b/docs/design.html @@ -216,6 +216,28 @@ In the case of Hadoop we parallelize the data load by splitting the load over individual map tasks, one for each node/topic/partition combination, allowing full parallelism in the loading. Hadoop provides the task management, and tasks which fail can restart without danger of duplicate data—they simply restart from their original position. + <h4><a id="static_membership" href="#static_membership">Static Membership</a></h4> + Static membership aims to improve the availability of stream applications, consumer groups and other applications built on top of the group rebalance protocol. + The rebalance protocol relies on the group coordinator to allocate entity ids to group members. These generated ids are ephemeral and will change when members restart and rejoin. + For consumer based apps, this "dynamic membership" can cause a large percentage of tasks re-assigned to different instances during administrative operations + such as code deploys, configuration updates and periodic restarts. For large state applications, shuffled tasks need a long time to recover their local states before processing + and cause applications to be partially or entirely unavailable. Motivated by this observation, Kafka’s group management protocol allows group members to provide persistent entity ids. + Group membership remains unchanged based on those ids, thus no rebalance will be triggered. + <p> + If you want to use static membership, + <ul> + <li>Upgrade both broker cluster and client apps to 2.3 or beyond, and also make sure the upgraded brokers are using <code>inter.broker.protocol.version</code> + of 2.3 or beyond as well.</li> + <li>Set the config <code>ConsumerConfig#GROUP_INSTANCE_ID_CONFIG</code> to a unique value for each consumer instance under one group.</li> + <li>For Kafka Streams applications, it is sufficient to set a unique <code>ConsumerConfig#GROUP_INSTANCE_ID_CONFIG</code> per KafkaStreams instance, + independent of the number of used threads for an instance.</li> + </ul> + If your broker is on an older version than 2.3, but you choose to set <code>ConsumerConfig#GROUP_INSTANCE_ID_CONFIG</code> on the client side, the application will detect + the broker version and then throws an UnsupportedException. If you accidentally configure duplicate ids for different instances, + a fencing mechanism on broker side will inform your duplicate client to shutdown immediately by triggering a <code>org.apache.kafka.common.errors.FencedInstanceIdException</code>. + For more details, see + <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances">KIP-345</a> + <h3><a id="semantics" href="#semantics">4.6 Message Delivery Semantics</a></h3> <p> Now that we understand a little about how producers and consumers work, let's discuss the semantic guarantees Kafka provides between producer and consumer. Clearly there are multiple possible message delivery diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html index f4c5d48..4b3dac2 100644 --- a/docs/streams/upgrade-guide.html +++ b/docs/streams/upgrade-guide.html @@ -241,6 +241,13 @@ details, see <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-312%3A+Add+Overloaded+StreamsBuilder+Build+Method+to+Accept+java.util.Properties">KIP-312</a> </p> + <p> + We are introducing static membership towards Kafka Streams user. This feature reduces unnecessary rebalances during normal application upgrades or rolling bounces. + For more details on how to use it, checkout <a href="/{{version}}/design/#static_membership">static membership design</a>. + Note, Kafka Streams uses the same <code>ConsumerConfig#GROUP_INSTANCE_ID_CONFIG</code>, and you only need to make sure it is uniquely defined across + different stream instances in one application. + </p> + <h3><a id="streams_api_changes_200" href="#streams_api_changes_200">Streams API changes in 2.0.0</a></h3> <p> In 2.0.0 we have added a few new APIs on the <code>ReadOnlyWindowStore</code> interface (for details please read <a href="#streams_api_changes_200">Streams API changes</a> below). diff --git a/docs/upgrade.html b/docs/upgrade.html index be9886c..d9232fa 100644 --- a/docs/upgrade.html +++ b/docs/upgrade.html @@ -71,6 +71,14 @@ </li> </ol> +<h5><a id="upgrade_230_notable" href="#upgrade_230_notable">Notable changes in 2.3.0</a></h5> +<ul> + <li> + We are introducing static membership towards consumer user. This feature reduces unnecessary rebalances during normal application upgrades or rolling bounces. + For more details on how to use it, checkout <a href="/{{version}}/design/#static_membership">static membership design</a>. + </li> +</ul> + <h5><a id="upgrade_220_notable" href="#upgrade_220_notable">Notable changes in 2.2.0</a></h5> <ul> <li>The default consumer group id has been changed from the empty string (<code>""</code>) to <code>null</code>. Consumers who use the new default group id will not be able to subscribe to topics,