This is an automated email from the ASF dual-hosted git repository. dajac pushed a commit to branch 4.0-blog in repository https://gitbox.apache.org/repos/asf/kafka-site.git
commit c125741d69af097f89f47994d8c95261aa7f9a7f Author: David Jacot <[email protected]> AuthorDate: Tue Mar 11 11:08:18 2025 +0100 MINOR: Apache Kafka 4.0.0 Blog --- blog.html | 280 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 280 insertions(+) diff --git a/blog.html b/blog.html index 12c9a717b..e11ae821c 100644 --- a/blog.html +++ b/blog.html @@ -22,6 +22,286 @@ <!--#include virtual="includes/_nav.htm" --> <div class="right"> <h1 class="content-title">Blog</h1> + <article> + <h2 class="bullet"> + <a id="apache_kafka_400_release_announcement"></a> + <a href="#apache_kafka_400_release_announcement">Apache Kafka 4.0.0 Release Announcement</a> + </h2> + 17 March 2025 - David Jacot + + <p> + We are proud to announce the release of Apache Kafka® 4.0 This release contains many new + features and improvements. This blog post will highlight some of the more prominent features. + For a full list of changes, be sure to check the <a href="https://archive.apache.org/dist/kafka/4.0.0/RELEASE_NOTES.html";>release notes</a>. + </p> + <p> + See the <a href="https://kafka.apache.org/documentation.html#upgrade_4_0_0";>Upgrading to 4.0 from any versions 0.8.x through 3.9.x</a> section in the + documentation for the list of notable changes and detailed upgrade steps. + </p> + <p> + Apache Kafka 4.0 is a significant milestone, marking the first major release to operate entirely + without Apache ZooKeeper®. By running in KRaft mode by default, Kafka simplifies + deployment and management, eliminating the complexity of maintaining a separate ZooKeeper + ensemble. This change significantly reduces operational overhead, enhances scalability, and + streamlines administrative tasks. We want to take this as an opportunity to express our gratitude + to the ZooKeeper community and say, thank you! ZooKeeper was the backbone of Kafka for + more than 10 years, and it did serve Kafka very well. Kafka would most likely not be what it is + today without it. We don’t take this for granted and highly appreciate all the hard work the + community invested to build ZooKeeper. Thank you! + </p> + <p> + Kafka 4.0 also brings the general availability of <a href="https://cwiki.apache.org/confluence/x/HhD1D";>KIP-848</a> introducing a powerful new consumer + group protocol designed to dramatically improve rebalance performance. This optimization + significantly reduces downtime and latency, enhancing the reliability and responsiveness of + consumer groups, especially in large-scale deployments. + </p> + <p> + Additionally, we are excited to offer <a href="https://cwiki.apache.org/confluence/x/i4ogF";>early access</a> to + Queues for Kafka (<a href="https://cwiki.apache.org/confluence/x/4hA0Dw";>KIP-932</a>), enabling Kafka to support traditional queue semantics directly. This + feature extends Kafka’s versatility, making it an ideal messaging platform for a wider range of + use cases, particularly those requiring point-to-point messaging patterns. + </p> + <p> + These highlights illustrate Kafka's continued evolution to meet the demands of modern, + scalable, and efficient data streaming and messaging architectures. + </p> + + <h3>Kafka Broker, Controller, Producer, Consumer and Admin Client</h3> + + <ul> + <li> + <a href="https://cwiki.apache.org/confluence/x/HhD1D";>KIP-848: The Next Generation of the Consumer Rebalance Protocol</a> + <br> + Apache Kafka also says goodbye to “stop-the-world” rebalances with the general availability of the + next generation of the consumer rebalance protocol. It increases the stability and the performance + of consumer groups while simplifying clients. The new protocol is enabled by default on the server + side. Consumers must opt-in by setting <code>group.protocol=consumer</code>. + See <a href="https://kafka.apache.org/40/documentation.html#consumer_rebalance_protocol";>here</a> for details. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/B40ODg";>KIP-890: Transactions Server-Side Defense</a> + <br> + The second phase of KIP-890 has been completed. It reduces the chances of “zombie transactions” + during producer failures. See <a href="https://kafka.apache.org/40/documentation.html#transaction_protocol";>here</a> for details. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/4hA0Dw";>KIP-932: Queues for Kafka (Early Access)</a> + <br> + This KIP introduces the concept of a share group as a way of enabling cooperative consumption + using Kafka topics. It does not add the concept of a “queue” to Kafka per se, but rather that + introduces cooperative consumption to accommodate these queuing use-cases using regular Kafka + topics. Share groups make this possible. You can think of a share group as roughly equivalent to a + “durable shared subscription” in existing systems. + Please see the <a href="https://cwiki.apache.org/confluence/x/i4ogF";>Early Access Release Notes</a> + for limitations and configurations. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/mpOzDw";>KIP-966: Eligible Leader Replicas (Preview)</a> + <br> + KIP-966 introduces Eligible Leader Replicas (ELR) in preview in 4.0. ELR is a subset of the ISR + replicas guaranteed to have complete data up to the high-watermark. ELRs are safe for leader + election, preventing data loss. See <a href="https://kafka.apache.org/40/documentation.html#eligible_leader_replicas";>here</a> for details. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/MYtEE";>KIP-996: Pre-Vote</a> + <br> + KIP-996 introduces a "Pre-Vote" mechanism to reduce unnecessary KRaft leader elections in Apache + Kafka clusters. By allowing nodes to check their eligibility for leadership before initiating an election, + this approach minimizes disruptions caused by network partitions or transient issues. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/joqzDw";>KIP-950: Tiered Storage Disablement</a> + <br> + This feature will allow customers to selectively disable Tiered Storage on a per-topic basis. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/XA-OEg";>KIP-1076: Metrics for client applications KIP-714 extension</a> + <br> + With KIP-714, cluster operators can now collect client metrics directly from brokers using a plugin. + This makes it much easier to monitor client behavior. However, KIP-714 only gathers metrics from + Kafka clients (admin, consumer, and producer) and doesn't cover application-level metrics for + embedded clients, such as Kafka Streams. To fully understand performance, it's important to collect + both client metrics and application-specific metrics. This KIP introduces a way for applications that + use Kafka clients to include their own metrics alongside the existing client metrics. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/NIyMEw";>KIP-1106: Add duration based offset reset option for consumer clients</a> + <br> + Kafka consumer supports auto.offset.reset config option, which is used when there is no initial + offset in Kafka or if the current offset does not exist any more on the server. This config currently + supports earliest/latest/none options. Currently consumer resets might force applications to + reprocess large amounts of data from earlier offsets. With infinite storage, its beneficial to have a + duration based offset reset strategy. This KIP allows applications to consume/initialise from a fixed + duration when there is no initial offset in Kafka. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/XoowEg";>KIP-1043: Administration of groups</a> + <br> + KIP-848 and KIP-932 introduce new group types, consumer and share respectively. The Admin Client + API has limitations when dealing with newer group types, sometimes returning errors or incorrectly + stating that groups don't exist. To address this challenge a new command-line tool, kafka-groups.sh, + and updates to existing command line tools, kafka-consumer-groups.sh and kafka-share-groups.sh, + were made in Kafka 4.0 to enable users to view all groups in a cluster, along with their types and + protocols, providing accurate information even when the Admin Client API fails. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/w4mMEw";>KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group</a> + <br> + This KIP extends the kafka-consumer-groups and the kafka-share-groups command line tools to + provide more information related to consumer and share groups. This helps when it comes to + troubleshooting consumer and share groups, respectively introduced by KIP-848 and KIP-932. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/KYuMEw";>KIP-1102: Enable clients to rebootstrap based on timeout or error code</a> + <br> + KIP-1102 enhances Kafka client resilience by proactively triggering metadata rebootstrap when no + updates occur within a timeout period and allowing servers to explicitly signal clients to rebootstrap. + This addresses previous limitations in KIP-899, where clients could become stuck with outdated + metadata unless all brokers were unreachable. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/K5sODg";>KIP-896: Remove old client protocol API versions in Kafka 4.0</a> + <br> + For the first time, old protocol API versions have been removed. Users should ensure brokers are + version 2.1 or higher before upgrading Java clients (including Connect and Streams to 4.0. Similarly, + users should ensure their Java clients version is 2.1 or higher before upgrading brokers to 4.0. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/y4kgF";>KIP-1124: Providing a clear Kafka Client upgrade path for 4.x</a> + <br> + This KIP outlines the upgrade patch for Kafka Clients, Streams and Connect to 4.0. It is a must read + before upgrading your clusters and clients. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/GCx4CQ";>KIP-653: Upgrade log4j to log4j2</a> + <br> + The logging framework has been migrated from Log4j to Log4j2. Users can use the + log4j-transform-cli tool to automatically convert their existing Log4j configuration files to Log4j2 + format. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/RomNCg";>KIP-724: Drop support for message formats v0 and v1</a> + <br> + The message formats v0 and v1 were deprecated in Apache Kafka 3.0. They were removed in 4.0. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/P4vOCg";>KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)</a> & + <a href="https://cwiki.apache.org/confluence/x/Bov5E";>KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)</a> + <br> + Kafka Clients and Kafka Streams require Java 11 while Kafka Brokers, Connect and Tools now requires Java 17. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/FAqpEQ";>KIP-1030: Change constraints and default values for various configurations</a> + <br> + KIP-1030 has changed the default values of some configurations in order to provide better default to + users starting using Apache Kafka. Check out the KIP for the details. + </li> + </ul> + + <h3>Kafka Streams</h3> + + <ul> + <li> + <a href="https://cwiki.apache.org/confluence/x/gIuMEw";>KIP-1104: Allow Foreign Key Extraction from Both Key and Value in KTable Joins</a> + <br> + KIP-1104 enhances Kafka Streams by allowing foreign keys to be extracted directly from both + record keys and values, removing the need to duplicate keys into values for foreign-key joins. + This improvement simplifies joins, reduces storage overhead, and provides a more intuitive + developer experience. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/TZCMEw";>KIP-1112: allow custom processor wrapping</a> + <br> + KIP-1112 simplifies applying cross-cutting logic in Kafka Streams by introducing the + ProcessorWrapper interface, enabling seamless injection of custom logic around Processor API + and DSL processors. This eliminates previous redundancy and reduces the maintenance + overhead caused by manually integrating logic into each processor individually. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/LQ6TEg";>KIP-1065: Add "retry" return-option to ProductionExceptionHandler</a> + <br> + KIP-1065 addresses persistent errors in Kafka Streams by allowing users to break retry loops + via a new "RETRY" option in the ProductionExceptionHandler. This enhancement provides + customizable error handling, enabling retries, graceful failures, or dropping problematic records + to continue processing efficiently. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/IgstEw";>KIP-1091: Improved Kafka Streams operator metrics</a> + <br> + Kafka Streams exposes its metrics with KIP-1076. The KIP especially adds a state metric for + each StreamThread and the client instance itself. It also provides detailed visibility into the + application state. + </li> + </ul> + + <h3>Kafka Connect</h3> + + <ul> + <li> + <a href="https://cwiki.apache.org/confluence/x/rZWzDw";>KIP-970: Deprecate and remove Connect's redundant task configurations endpoint</a> + <br> + The endpoint <code>GET /connectors/{connector}/tasks-config</code> added in KIP-661 was deprecated in + Apache Kafka 3.7 and is finally removed in 4.0. The endpoint + </code>GET /connectors/{connector}/tasks</code> must be used as a replacement. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/jA3OEg";>KIP-1074: Allow the replication of user internal topics</a> + <br> + Currently, Mirror Maker 2 automatically excludes topics ending in ".internal" or "-internal", + mistakenly identifying them as internal topics and preventing replication of legitimate business + topics unless a custom replication policy is implemented. This KIP introduces configurable + behavior, allowing users to replicate these topics without needing custom code. + </li> + <li> + <a href="https://cwiki.apache.org/confluence/x/UwotEw";>KIP-1089: Allow disabling heartbeats replication in MirrorSourceConnector</a> + <br> + Currently, MirrorSourceConnector always replicates heartbeat topics, which can cause issues + when multiple connectors with different configurations replicate topics between the same + clusters, resulting in duplicate heartbeat replication. This KIP introduces a configurable option to + disable heartbeat topic replication, allowing users greater flexibility, especially when using + multiple connectors with distinct configurations. + </li> + </ul> + + <h3>Summary</h3> + + <p>Ready to get started with Apache Kafka 4.0.0? Check out all the details in the <a href="https://downloads.apache.org/kafka/4.0.0/RELEASE_NOTES.html";>release notes</a> + and <a href="https://kafka.apache.org/downloads";>download</a> Apache Kafka 4.0.0.</p> + + <p>This was a community effort, so thank you to everyone who contributed to this release, including all our users and our 196 contributors:<br> + A. Sophie Blee-Goldman, abhi-ksolves, Abhijeet Kumar, Abhinav Dixit, Abhishek Giri, Alieh + Saeedi, Almog Gavra, Alyssa Huang, Andrew Schofield, Anna Sophie Blee-Goldman, Anshul + Goyal, Ao Li, Apoorv Mittal, Arnav Dadarya, Arpit Goyal, Artem Livshits, Ayoub Omari, + bachmanity1, bboyleonp666, Bill Bejeck, brenden20, Bruno Cadonna, Caio Guedes, Calvin Liu, + Chengyan, Cheryl Simmons, Chia Chuan Yu, Chia-Chuan Yu, Chia-Ping Tsai, Chirag Wadhwa, + Chris Egerton, Christo Lolov, Christopher L. Shannon, Chung, Ming-Yen, ClarkChen, Clay + Johnson, Colin P. McCabe, Colin Patrick McCabe, Colt McNealy, Danica Fine, Dániel Urbán, + David Arthur, David Jacot, David Mao, David Schlosnagle, Dejan Stojadinović, dengziming, + devanshikhatsuriya, Dimitar Dimitrov, Divij Vaidya, DL1231, Dmitry Werner, donaldzhu-cc, + Chia-Ping Tsai, Dongnuo Lyu, dujian0068, Edoardo Comar, Eric Chang, Federico Valeri, + Frederik Rouleau, GangHuo, Gantigmaa Selenge, Gaurav Narula, Greg Harris, Guozhang + Wang, Hailey Ni, Hongten, HYUNSANG HAN (한현상, Travis), Ian McDonald, Igor Soarez, + Ismael Juma, Ivan Yurchenko, Jakub Scholz, Jason Gustafson, Jason Taylor, Jeff Kim, + Jhen-Yung Hsu, Jim Galasyn, Joao Pedro Fonseca, Joao Pedro Fonseca Dantas, João Pedro + Fonseca Dantas, John Huang, JohnHuang, Jonah Hooper, José Armando García Sancio, Jose + Sancio, Josep Prat, Jun Rao, Justin Lee, Justine Olshan, Kamal Chandraprakash, + KApolinario1120, kartik-3513, Kaushik Raina, Ken Huang, kevin-wu24, Kirk True, Kondrat + Bertalan, Krishna Agarwal, Kuan-Po Tseng, kwonyonghyun, Laxman Ch, Liam Miller-Cushon, + Lianet Magrans, Linsiyuan9, Linu Shibu, Liu Zeyu, Logan Zhu, Loïc GREFFIER, Lucas + Brutschy, Luke Chen, lushilin, Mahsa Seifikar, Manikumar Reddy, mannoopj, Martin Sillence, + Mason Chen, Matthias J. Sax, Matthias Sax, Mehari Beyene, Mickael Maison, Ming-Yen Chung, + mingdaoy, msureshpro, Murali Basani, Nancy, Nick Guo, Nick Telford, Oleg Bonar, Oleksandr + K., Omnia Ibrahim, Parker Chang, Patrik Marton, PaulRMellor, Peter Lee, Philip Nee, PoAn + Yang, Rajini Sivaram, Ramin Gharib, Reviewers: José Armando García Sancio, Ritika Reddy, + Robert Young, Rohan, S.Y. Wang, Said Boudjelda, Sanskar Jhajharia, santhoshct, Sasaki Toru, + Satish Duggana, Saxon Chen, Scott Hendricks, Sean Quah, Sebastien Viale, Shivsundar R, + snehashisp, Stanislav Knot, Steven Xu, stevenbooke, Stig Døssing, Sushant Mahajan, Swikar + Patel, TaiJuWu, tall15421542-lab, TapDang, Ted Yan, TengYao Chi, Thomas Thornton, Tim Fox, + tkuramoto33, Tom Duckering, Vedarth Sharma, Vikas Singh, Viktor Somogyi-Vass, Vincent + Jiang, wperlichek, xijiu, Xuan-Zhang Gong, yangjf2019, Yaroslav Kutsela (@Serwios), Yash + Mayya, Yung, yungh, yx9o, Zhengke Zhou, Ziming Deng, 陳昱霖 (Yu-Lin Chen) + </p> + </article> <article> <h2 class="bullet"> <a id="apache_kafka_390_release_announcement"></a>
