divijvaidya commented on code in PR #668:
URL: https://github.com/apache/kafka-site/pull/668#discussion_r1994019813
##########
blog.html:
##########
@@ -22,6 +22,286 @@
<!--#include virtual="includes/_nav.htm" -->
<div class="right">
<h1 class="content-title">Blog</h1>
+ <article>
+ <h2 class="bullet">
+ <a id="apache_kafka_400_release_announcement"></a>
+ <a href="#apache_kafka_400_release_announcement">Apache
Kafka 4.0.0 Release Announcement</a>
+ </h2>
+ 17 March 2025 - David Jacot
+
+ <p>
+ We are proud to announce the release of Apache Kafka®
4.0 This release contains many new
+ features and improvements. This blog post will highlight
some of the more prominent features.
+ For a full list of changes, be sure to check the <a
href="https://archive.apache.org/dist/kafka/4.0.0/RELEASE_NOTES.html";>release
notes</a>.
+ </p>
+ <p>
+ Apache Kafka 4.0 is a significant milestone, marking the
first major release to operate entirely
+ without Apache ZooKeeper®. By running in KRaft mode
by default, Kafka simplifies
+ deployment and management, eliminating the complexity of
maintaining a separate ZooKeeper
+ ensemble. This change significantly reduces operational
overhead, enhances scalability, and
+ streamlines administrative tasks. We want to take this as
an opportunity to express our gratitude
+ to the ZooKeeper community and say, thank you! ZooKeeper
was the backbone of Kafka for
+ more than 10 years, and it did serve Kafka very well.
Kafka would most likely not be what it is
+ today without it. We don’t take this for granted and
highly appreciate all the hard work the
+ community invested to build ZooKeeper. Thank you!
+ </p>
+ <p>
+ Kafka 4.0 also brings the general availability of <a
href="https://cwiki.apache.org/confluence/x/HhD1D";>KIP-848</a> introducing a
powerful new consumer
+ group protocol designed to dramatically improve rebalance
performance. This optimization
+ significantly reduces downtime and latency, enhancing the
reliability and responsiveness of
+ consumer groups, especially in large-scale deployments.
+ </p>
+ <p>
+ Additionally, we are excited to offer <a
href="https://cwiki.apache.org/confluence/x/i4ogF";>early access</a> to
+ Queues for Kafka (<a
href="https://cwiki.apache.org/confluence/x/4hA0Dw";>KIP-932</a>), enabling
Kafka to support traditional queue semantics directly. This
+ feature extends Kafka’s versatility, making it an ideal
messaging platform for a wider range of
+ use cases, particularly those requiring point-to-point
messaging patterns.
+ </p>
+ <p>
+ Kafka's major releases, such as version 4.0, remove APIs
deprecated for at least 12 months to simplify the platform
+ and encourage the adoption of newer features. Notably, in
Kafka 4.0, Kafka Clients and Kafka Streams require Java 11,
+ while Kafka Brokers, Connect, and Tools now require Java
17. This release also updated the minimum supported client
+ and broker versions (<a
href="https://cwiki.apache.org/confluence/x/K5sODg";>KIP-896</a>) and defined
new baseline
+ requirements for supported upgrade paths. Further details
are provided in subsequent sections of this document.
+ </p>
+ <p>
+ These highlights illustrate Kafka's continued evolution to
meet the demands of modern,
+ scalable, and efficient data streaming and messaging
architectures.
+ </p>
+ <p>
+ See the <a
href="https://kafka.apache.org/documentation.html#upgrade_4_0_0";>Upgrading to
4.0</a> section in the
+ documentation for the list of notable changes and detailed
upgrade steps.
+ </p>
+
+ <h3>Kafka Broker, Controller, Producer, Consumer and Admin
Client</h3>
+
+ <ul>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/HhD1D";>KIP-848: The Next Generation
of the Consumer Rebalance Protocol</a>
+ <br>
+ Apache Kafka also says goodbye to “stop-the-world”
rebalances with the general availability of the
+ next generation of the consumer rebalance protocol. It
increases the stability and the performance
+ of consumer groups while simplifying clients. The new
protocol is enabled by default on the server
+ side. Consumers must opt-in by setting
<code>group.protocol=consumer</code>.
+ See <a
href="https://kafka.apache.org/40/documentation.html#consumer_rebalance_protocol";>here</a>
for details.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/B40ODg";>KIP-890: Transactions
Server-Side Defense</a>
+ <br>
+ The second phase of KIP-890 has been completed. It
reduces the chances of “zombie transactions”
+ during producer failures. See <a
href="https://kafka.apache.org/40/documentation.html#transaction_protocol";>here</a>
for details.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/4hA0Dw";>KIP-932: Queues for Kafka
(Early Access)</a>
+ <br>
+ This KIP introduces the concept of a share group as a
way of enabling cooperative consumption
+ using Kafka topics. It does not add the concept of a
“queue” to Kafka per se, but rather that
+ introduces cooperative consumption to accommodate
these queuing use-cases using regular Kafka
+ topics. Share groups make this possible. You can think
of a share group as roughly equivalent to a
+ “durable shared subscription” in existing systems.
+ Please see the <a
href="https://cwiki.apache.org/confluence/x/i4ogF";>Early Access Release
Notes</a>
+ for limitations and configurations.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/mpOzDw";>KIP-966: Eligible Leader
Replicas (Preview)</a>
+ <br>
+ KIP-966 introduces Eligible Leader Replicas (ELR) in
preview in 4.0. ELR is a subset of the ISR
+ replicas guaranteed to have complete data up to the
high-watermark. ELRs are safe for leader
+ election, preventing data loss. See <a
href="https://kafka.apache.org/40/documentation.html#eligible_leader_replicas";>here</a>
for details.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/MYtEE";>KIP-996: Pre-Vote</a>
+ <br>
+ KIP-996 introduces a "Pre-Vote" mechanism to reduce
unnecessary KRaft leader elections in Apache
+ Kafka clusters. By allowing nodes to check their
eligibility for leadership before initiating an election,
+ this approach minimizes disruptions caused by network
partitions or transient issues.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/XA-OEg";>KIP-1076: Metrics for
client applications KIP-714 extension</a>
+ <br>
+ With <a
href="https://cwiki.apache.org/confluence/x/XA-OEg";>KIP-714</a>, cluster
operators can now collect
+ client metrics directly from brokers using a plugin.
This makes it much easier to monitor client behavior.
+ However, KIP-714 only gathers metrics from Kafka
clients (admin, consumer, and producer) and doesn't cover
+ application-level metrics for embedded clients, such
as Kafka Streams. To fully understand performance,
+ it's important to collect both client metrics and
application-specific metrics. This KIP introduces a way
+ for applications that use Kafka clients to include
their own metrics alongside the existing client metrics.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/NIyMEw";>KIP-1106: Add duration
based offset reset option for consumer clients</a>
+ <br>
+ Kafka consumer supports <code>auto.offset.reset</code>
config option, which is used when there is no initial
+ offset in Kafka or if the current offset does not
exist any more on the server. This config currently
+ supports earliest/latest/none options. Currently
consumer resets might force applications to
+ reprocess large amounts of data from earlier offsets.
With infinite storage, it's beneficial to have a
+ duration based offset reset strategy. This KIP allows
applications to consume/initialise from a fixed
+ duration when there is no initial offset in Kafka.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/XoowEg";>KIP-1043: Administration of
groups</a>
+ <br>
+ KIP-848 and KIP-932 introduce new group types,
consumer and share respectively. The Admin Client
+ API has limitations when dealing with newer group
types, sometimes returning errors or incorrectly
+ stating that groups don't exist. To address this
challenge, a new command-line tool, <code>kafka-groups.sh</code>,
+ and updates to existing command line tools,
<code>kafka-consumer-groups.sh</code> and <code>kafka-share-groups.sh</code>,
+ were made in Kafka 4.0 to enable users to view all
groups in a cluster, along with their types and
+ protocols, providing accurate information even when
the Admin Client API fails.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/w4mMEw";>KIP-1099: Extend
kafka-consumer-groups command line tool to support new consumer group</a>
+ <br>
+ This KIP extends the
<code>kafka-consumer-groups.sh</code> and the
<code>kafka-share-groups.sh</code> command line tools to
+ provide more information related to consumer and share
groups. This helps when it comes to
+ troubleshooting consumer and share groups,
respectively introduced by KIP-848 and KIP-932.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/KYuMEw";>KIP-1102: Enable clients to
rebootstrap based on timeout or error code</a>
+ <br>
+ KIP-1102 enhances Kafka client resilience by
proactively triggering metadata rebootstrap when no
+ updates occur within a timeout period and allowing
servers to explicitly signal clients to rebootstrap.
+ This addresses previous limitations in KIP-899, where
clients could become stuck with outdated
+ metadata unless all brokers were unreachable.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/K5sODg";>KIP-896: Remove old client
protocol API versions in Kafka 4.0</a>
+ <br>
+ For the first time, old protocol API versions have
been removed. Users should ensure brokers are
+ version 2.1 or higher before upgrading Java clients
(including Connect and Streams) to 4.0. Similarly,
Review Comment:
Ah, I believe that we have no way to confirm if other clients follow the
same versioning, so we cannot say generic client version 2.1.
Perhaps, we can add something like, "Clients other than Java should use API
version 2.1 or higher to connect to 4.0 brokers"? Feel free to re-phrase and
not a blocking comment. Upto you to decide if we want to include this
information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
