The Apache Kafka community is pleased to announce the release for Apache Kafka 2.0.0.
This is a major release and includes significant new features from 40 KIPs. It contains fixes and improvements from 246 JIRAs, including a few critical bugs. Here is a summary of some notable changes: ** KIP-290 adds support for prefixed ACLs, simplifying access control management in large secure deployments. Bulk access to topics, consumer groups or transactional ids with a prefix can now be granted using a single rule. Access control for topic creation has also been improved to enable access to be granted to create specific topics or topics with a prefix. ** KIP-255 adds a framework for authenticating to Kafka brokers using OAuth2 bearer tokens. The SASL/OAUTHBEARER implementation is customizable using callbacks for token retrieval and validation. **Host name verification is now enabled by default for SSL connections to ensure that the default SSL configuration is not susceptible to man-in-the middle attacks. You can disable this verification for deployments where validation is performed using other mechanisms. ** You can now dynamically update SSL trust stores without broker restart. You can also configure security for broker listeners in ZooKeeper before starting brokers, including SSL key store and trust store passwords and JAAS configuration for SASL. With this new feature, you can store sensitive password configs in encrypted form in ZooKeeper rather than in cleartext in the broker properties file. ** The replication protocol has been improved to avoid log divergence between leader and follower during fast leader failover. We have also improved resilience of brokers by reducing the memory footprint of message down-conversions. By using message chunking, both memory usage and memory reference time have been reduced to avoid OutOfMemory errors in brokers. ** Kafka clients are now notified of throttling before any throttling is applied when quotas are enabled. This enables clients to distinguish between network errors and large throttle times when quotas are exceeded. ** We have added a configuration option for Kafka consumer to avoid indefinite blocking in the consumer. ** We have dropped support for Java 7 and removed the previously deprecated Scala producer and consumer. ** Kafka Connect includes a number of improvements and features. KIP-298 enables you to control how errors in connectors, transformations and converters are handled by enabling automatic retries and controlling the number of errors that are tolerated before the connector is stopped. More contextual information can be included in the logs to help diagnose problems and problematic messages consumed by sink connectors can be sent to a dead letter queue rather than forcing the connector to stop. ** KIP-297 adds a new extension point to move secrets out of connector configurations and integrate with any external key management system. The placeholders in connector configurations are only resolved before sending the configuration to the connector, ensuring that secrets are stored and managed securely in your preferred key management system and not exposed over the REST APIs or in log files. ** We have added a thin Scala wrapper API for our Kafka Streams DSL, which provides better type inference and better type safety during compile time. Scala users can have less boilerplate in their code, notably regarding Serdes with new implicit Serdes. ** Message headers are now supported in the Kafka Streams Processor API, allowing users to add and manipulate headers read from the source topics and propagate them to the sink topics. ** Windowed aggregations performance in Kafka Streams has been largely improved (sometimes by an order of magnitude) thanks to the new single-key-fetch API. ** We have further improved unit testibility of Kafka Streams with the kafka-streams-testutil artifact. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/2.0.0/RELEASE_NOTES.html You can download the source and binary release (Scala 2.11 and Scala 2.12) from: https://kafka.apache.org/downloads#2.0.0 <https://kafka.apache.org/downloads#2.0.0> --------------------------------------------------------------------------------------------------- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. ** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. With these APIs, Kafka can be used for two broad classes of application: ** Building real-time streaming data pipelines that reliably get data between systems or applications. ** Building real-time streaming applications that transform or react to the streams of data. Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 131 contributors to this release! Adem Efe Gencer, Alex D, Alex Dunayevsky, Allen Wang, Andras Beni, Andy Bryant, Andy Coates, Anna Povzner, Arjun Satish, asutosh936, Attila Sasvari, bartdevylder, Benedict Jin, Bill Bejeck, Blake Miller, Boyang Chen, cburroughs, Chia-Ping Tsai, Chris Egerton, Colin P. Mccabe, Colin Patrick McCabe, ConcurrencyPractitioner, Damian Guy, dan norwood, Daniel Shuy, Daniel Wojda, Dark, David Glasser, Debasish Ghosh, Detharon, Dhruvil Shah, Dmitry Minkovsky, Dong Lin, Edoardo Comar, emmanuel Harel, Eugene Sevastyanov, Ewen Cheslack-Postava, Fedor Bobin, fedosov-alexander, Filipe Agapito, Florian Hussonnois, fredfp, Gilles Degols, gitlw, Gitomain, Guangxian, Gunju Ko, Gunnar Morling, Guozhang Wang, hmcl, huxi, huxihx, Igor Kostiakov, Ismael Juma, Jacek Laskowski, Jagadesh Adireddi, Jarek Rudzinski, Jason Gustafson, Jeff Klukas, Jeremy Custenborder, Jiangjie (Becket) Qin, Jiangjie Qin, JieFang.He, Jimin Hsieh, Joan Goyeau, Joel Hamill, John Roesler, Jon Lee, Jorge Quilcate Otoya, Jun Rao, Kamal C, khairy, Koen De Groote, Konstantine Karantasis, Lee Dongjin, Liju John, Liquan Pei, lisa2lisa, Lucas Wang, Magesh Nandakumar, Magnus Edenhill, Magnus Reftel, Manikumar Reddy, Manikumar Reddy O, manjuapu, Mats Julian Olsen, Matthias J. Sax, Max Zheng, maytals, Michael Arndt, Michael G. Noll, Mickael Maison, nafshartous, Nick Travers, nixsticks, Paolo Patierno, parafiend, Patrik Erdes, Radai Rosenblatt, Rajini Sivaram, Randall Hauch, ro7m, Robert Yokota, Roman Khlebnov, Ron Dagostino, Sandor Murakozi, Sasaki Toru, Sean Glover, Sebastian Bauersfeld, Siva Santhalingam, Stanislav Kozlovski, Stephane Maarek, Stuart Perks, Surabhi Dixit, Sönke Liebau, taekyung, tedyu, Thomas Leplus, UVN, Vahid Hashemian, Valentino Proietti, Viktor Somogyi, Vitaly Pushkar, Wladimir Schmidt, wushujames, Xavier Léauté, xin, yaphet, Yaswanth Kumar, ying-zheng, Yu We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at https://kafka.apache.org/ Thank you! Regards, Rajini
