Re: Supported Kafka/Zookeeper Version with ELK 8.4.3
Hi, Sorry, we do not develop ELK. In fact, I'm not sure what that acronym refers to. I would suggest checking in with support for that product / project, since it is not part of Apache Kafka. best, Colin On Fri, Oct 28, 2022, at 06:23, Kumar, Sudip wrote: > Hi Team, > > We are still waiting for the reply. Please update we must know what version > of Kafka is compatible with ELK 8.4 version. > > Still, I can see no one replied on user and Dev community portal > > > > > Thanks > Sudip > > > *From:* Kumar, Sudip > *Sent:* Monday, October 17, 2022 5:23 PM > *To:* users@kafka.apache.org; d...@kafka.apache.org > *Cc:* Rajendra Bangal, Nikhil ; Verma, > Harshit ; Verma, Deepak Kumar > ; Arkal, Dinesh Balaji > ; Saurabh, Shobhit > > *Subject:* Supported Kafka/Zookeeper Version with ELK 8.4.3 > *Importance:* High > > > Hi Kafka Team, > > Currently we are planning to upgrade ELK 7.16 to 8.4.3 version. In our > ecosystem we are using Kafka as middleware which is ingesting data coming > from different sources where publisher (Logstash shipper) publishing data in > different Kafka Topics and subscriber (Logstash indexer) consuming the data. > > We have an integration of ELK 7.16 with Kafka V2.5.1 and zookeeper 3.5.8. > Please suggest if we upgrade on ELK 8.4.3 version which Kafka and Zookeeper > version will be supported? Provide us handful documents. > > Let me know if you any further questions. > > *Thanks* > *Sudip Kumar* > *Capgemini-India * > > > This message contains information that may be privileged or confidential and > is the property of the Capgemini Group. It is intended only for the person to > whom it is addressed. If you are not the intended recipient, you are not > authorized to read, print, retain, copy, disseminate, distribute, or use this > message or any part thereof. If you receive this message in error, please > notify the sender immediately and delete all copies of this message.
Re: [ANNOUNCE] Apache Kafka 3.2.1
Thanks, David, for running this release. C. On Tue, Aug 2, 2022, at 12:09, Matthew Benedict de Detrich wrote: > Thanks for organising the release! > > -- > Matthew de Detrich > Aiven Deutschland GmbH > Immanuelkirchstraße 26, 10405 Berlin > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > m: +491603708037 > w: aiven.io e: matthew.dedetr...@aiven.io > On 2. Aug 2022, 01:46 +0200, David Arthur , wrote: >> The Apache Kafka community is pleased to announce the release for >> Apache Kafka 3.2.1 >> >> This is a bugfix release with several fixes since the release of >> 3.2.0. A few of the major issues include: >> >> * KAFKA-14062 OAuth client token refresh fails with SASL extensions >> * KAFKA-14079 Memory leak in connectors using errors.tolerance=all >> * KAFKA-14024 Cooperative rebalance regression causing clients to get stuck >> >> >> All of the changes in this release can be found in the release notes: >> >> https://www.apache.org/dist/kafka/3.2.1/RELEASE_NOTES.html >> >> >> You can download the source and binary release (Scala 2.12 and 2.13) from: >> >> https://kafka.apache.org/downloads#3.2.1 >> >> --- >> >> >> Apache Kafka is a distributed streaming platform with four core APIs: >> >> ** The Producer API allows an application to publish a stream of >> records to one or more Kafka topics. >> >> ** The Consumer API allows an application to subscribe to one or more >> topics and process the stream of records produced to them. >> >> ** The Streams API allows an application to act as a stream processor, >> consuming an input stream from one or more topics and producing an >> output stream to one or more output topics, effectively transforming >> the input streams to output streams. >> >> ** The Connector API allows building and running reusable producers or >> consumers that connect Kafka topics to existing applications or data >> systems. For example, a connector to a relational database might >> capture every change to a table. >> >> >> With these APIs, Kafka can be used for two broad classes of application: >> >> ** Building real-time streaming data pipelines that reliably get data >> between systems or applications. >> >> ** Building real-time streaming applications that transform or react >> to the streams of data. >> >> >> Apache Kafka is in use at large and small companies worldwide, >> including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, >> Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and >> Zalando, among others. >> >> A big thank you for the following 19 contributors to this release! >> >> Akhilesh Chaganti, Bruno Cadonna, Christopher L. Shannon, David >> Arthur, Divij Vaidya, Eugene Tolbakov, Guozhang Wang, Ismael Juma, >> James Hughes, Jason Gustafson, Kirk True, Lucas Bradstreet, Luke Chen, >> Nicolas Guyomar, Niket Goel, Okada Haruki, Shawn Wang, Viktor >> Somogyi-Vass, Walker Carlson >> >> We welcome your help and feedback. For more information on how to >> report problems, and to get involved, visit the project website at >> https://kafka.apache.org/ >> >> >> Thank you! >> >> Regards, >> David Arthur
Re: [EXTERNAL] Re: Security vulnerabilities in kafka:2.13-2.6.0/2.7.0 docker image
It seems like your image does not show up on the mailing list. best, Colin On Wed, Sep 1, 2021, at 06:26, Ashish Patil wrote: > Hi Team > > I tried upgrading it to 2.13_2.8.0 but still have these vulnerabilities. > > > > What is your suggestion on this? > > Thanks > Ashish > > *From:* Jake Murphy Smith > *Sent:* 01 September 2021 09:31 > *To:* Ashish Patil > *Subject:* RE: [EXTERNAL] Re: Security vulnerabilities in > kafka:2.13-2.6.0/2.7.0 docker image > > > > *From:* Luke Chen > *Sent:* 01 September 2021 04:11 > *To:* Kafka Users > *Cc:* d...@kafka.apache.org; Jake Murphy Smith > *Subject:* [EXTERNAL] Re: Security vulnerabilities in kafka:2.13-2.6.0/2.7.0 > docker image > > *ATTENTION:* This email originated from outside of GM. > > > > Hi Ashish, > I suggested that you upgrade to V2.8. > I checked 2 of the CVEs, and are fixed (or not used, like libfetch) in V2.8. > If you still found the CVEs existed in V2.8, please raise it. > > Thank you. > Luke > > > > > On Wed, Sep 1, 2021 at 4:07 AM Ashish Patil wrote: >> Hi Team >> >> I wanted to use the 2.6.0 docker image for Kafka but It has lots of security >> vulnerabilities. >> Please find the below list of security vulnerabilities >> ** >> CVE-2021-36159 >> CVE-2020-25649 <https://github.com/advisories/GHSA-288c-cq4h-88gq> >> CVE-2021-22926 >> CVE-2021-22922 >> CVE-2021-22924 >> CVE-2021-22922 >> CVE-2021-22924 >> CVE-2021-31535 >> CVE-2019-17571 <https://github.com/advisories/GHSA-2qrg-x229-3v8q> >> ** >> >> I did raise this issue here >> https://github.com/wurstmeister/kafka-docker/issues/681 but it looks like >> the issue is within the Kafka binary. >> >> >> >> Do we have any plan to fix this in the coming version or any suggestions >> around this? >> >> Thanks >> Ashish
Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch
Congratulations, Randall! best, Colin On Wed, Apr 21, 2021, at 09:51, Mickael Maison wrote: > Congratulations Randall, well deserved! > > On Mon, Apr 19, 2021 at 10:55 PM Konstantine Karantasis > wrote: > > > > Congratulations Randall! > > > > Konstantine > > > > On Mon, Apr 19, 2021 at 1:14 AM Bruno Cadonna wrote: > > > > > Congrats Randall! Well deserved! > > > > > > Bruno > > > > > > On 17.04.21 01:43, Matthias J. Sax wrote: > > > > Hi, > > > > > > > > It's my pleasure to announce that Randall Hauch in now a member of the > > > > Kafka PMC. > > > > > > > > Randall has been a Kafka committer since Feb 2019. He has remained > > > > active in the community since becoming a committer. > > > > > > > > > > > > > > > > Congratulations Randall! > > > > > > > > -Matthias, on behalf of Apache Kafka PMC > > > > > > > >
Re: [kafka-clients] Re: [ANNOUNCE] Apache Kafka 2.8.0
Hi Ran, As part of the KRaft work outlined in KIP-500, we are planning on creating an authorizer that does not rely on ZooKeeper. This work is not included in the 2.8 release, however. regards, Colin On Mon, Apr 19, 2021, at 13:02, Ran Lupovich wrote: > Hello, Maybe I missed it in the documentations but where can I read about > what is the future plan for the zookeeper managed ACLs ? > > בתאריך יום ב׳, 19 באפר׳ 2021, 22:48, מאת Israel Ekpo >: > > > This is fantastic news! > > > > Thanks everyone for contributing and thanks John for managing the release. > > > > On Mon, Apr 19, 2021 at 1:10 PM Guozhang Wang wrote: > > > > > This is great! Thanks to everyone who has contributed to the release. > > > > > > On Mon, Apr 19, 2021 at 9:36 AM John Roesler > > wrote: > > > > > >> The Apache Kafka community is pleased to announce the > > >> release for Apache Kafka 2.8.0 > > >> > > >> Kafka 2.8.0 includes a number of significant new features. > > >> Here is a summary of some notable changes: > > >> > > >> * Early access of replace ZooKeeper with a self-managed > > >> quorum > > >> * Add Describe Cluster API > > >> * Support mutual TLS authentication on SASL_SSL listeners > > >> * JSON request/response debug logs > > >> * Limit broker connection creation rate > > >> * Topic identifiers > > >> * Expose task configurations in Connect REST API > > >> * Update Streams FSM to clarify ERROR state meaning > > >> * Extend StreamJoined to allow more store configs > > >> * More convenient TopologyTestDriver construtors > > >> * Introduce Kafka-Streams-specific uncaught exception > > >> handler > > >> * API to start and shut down Streams threads > > >> * Improve TimeWindowedDeserializer and TimeWindowedSerde to > > >> handle window size > > >> * Improve timeouts and retries in Kafka Streams > > >> > > >> All of the changes in this release can be found in the > > >> release notes: > > >> https://www.apache.org/dist/kafka/2.8.0/RELEASE_NOTES.html > > >> > > >> > > >> You can download the source and binary release (Scala 2.12 > > >> and 2.13) from: > > >> https://kafka.apache.org/downloads#2.8.0 > > >> > > >> > > >> --- > > >> > > >> > > >> Apache Kafka is a distributed streaming platform with four > > >> core APIs: > > >> > > >> > > >> ** The Producer API allows an application to publish a > > >> stream records to one or more Kafka topics. > > >> > > >> ** The Consumer API allows an application to subscribe to > > >> one or more topics and process the stream of records > > >> produced to them. > > >> > > >> ** The Streams API allows an application to act as a stream > > >> processor, consuming an input stream from one or more topics > > >> and producing an output stream to one or more output topics, > > >> effectively transforming the input streams to output > > >> streams. > > >> > > >> ** The Connector API allows building and running reusable > > >> producers or consumers that connect Kafka topics to existing > > >> applications or data systems. For example, a connector to a > > >> relational database might capture every change to a table. > > >> > > >> > > >> With these APIs, Kafka can be used for two broad classes of > > >> application: > > >> > > >> ** Building real-time streaming data pipelines that reliably > > >> get data between systems or applications. > > >> > > >> ** Building real-time streaming applications that transform > > >> or react to the streams of data. > > >> > > >> > > >> Apache Kafka is in use at large and small companies > > >> worldwide, including Capital One, Goldman Sachs, ING, > > >> LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York > > >> Times, Uber, Yelp, and Zalando, among others. > > >> > > >> A big thank you for the following 128 contributors to this > > >> release! > > >> > > >> 17hao, abc863377, Adem Efe Gencer, Alexander Iskuskov, Alok > >
Re: [ANNOUNCE] New committer: Boyang Chen
Congratulations, Boyang! cheers, Colin On Mon, Jun 22, 2020, at 16:26, Guozhang Wang wrote: > The PMC for Apache Kafka has invited Boyang Chen as a committer and we are > pleased to announce that he has accepted! > > Boyang has been active in the Kafka community more than two years ago. > Since then he has presented his experience operating with Kafka Streams at > Pinterest as well as several feature development including rebalance > improvements (KIP-345) and exactly-once scalability improvements (KIP-447) > in various Kafka Summit and Kafka Meetups. More recently he's also been > participating in Kafka broker development including post-Zookeeper > controller design (KIP-500). Besides all the code contributions, Boyang has > also helped reviewing even more PRs and KIPs than his own. > > Thanks for all the contributions Boyang! And look forward to more > collaborations with you on Apache Kafka. > > > -- Guozhang, on behalf of the Apache Kafka PMC >
Re: [kafka-clients] [VOTE] 2.5.0 RC3
+1 (binding) verified checksums ran unitTest ran check best, Colin On Tue, Apr 7, 2020, at 21:03, David Arthur wrote: > Hello Kafka users, developers and client-developers, > > This is the forth candidate for release of Apache Kafka 2.5.0. > > * TLS 1.3 support (1.2 is now the default) > * Co-groups for Kafka Streams > * Incremental rebalance for Kafka Consumer > * New metrics for better operational insight > * Upgrade Zookeeper to 3.5.7 > * Deprecate support for Scala 2.11 > > Release notes for the 2.5.0 release: > https://home.apache.org/~davidarthur/kafka-2.5.0-rc3/RELEASE_NOTES.html > > *** Please download, test and vote by Friday April 10th 5pm PT > > Kafka's KEYS file containing PGP keys we use to sign the release: > https://kafka.apache.org/KEYS > > * Release artifacts to be voted upon (source and binary): > https://home.apache.org/~davidarthur/kafka-2.5.0-rc3/ > > * Maven artifacts to be voted upon: > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > * Javadoc: > https://home.apache.org/~davidarthur/kafka-2.5.0-rc3/javadoc/ > > * Tag to be voted upon (off 2.5 branch) is the 2.5.0 tag: > https://github.com/apache/kafka/releases/tag/2.5.0-rc3 > > * Documentation: > https://kafka.apache.org/25/documentation.html > > * Protocol: > https://kafka.apache.org/25/protocol.html > > Successful Jenkins builds to follow > > Thanks! > David > > -- > You received this message because you are subscribed to the Google Groups > "kafka-clients" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kafka-clients+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/kafka-clients/CA%2B0Ze6rUxaPRvddHb50RfVxRtHHvnJD8j9Q9ni18Okc9s-_DSQ%40mail.gmail.com > > <https://groups.google.com/d/msgid/kafka-clients/CA%2B0Ze6rUxaPRvddHb50RfVxRtHHvnJD8j9Q9ni18Okc9s-_DSQ%40mail.gmail.com?utm_medium=email_source=footer>.
Re: Newbie Question
Thanks Hans - this makes sense, except for the debug messages give me exactly what I need without having to instrument any clients. It should be noted that for now, I am running a single server, so perhaps the messages change when I cluster? I maybe caused confusion by mentioning that I want to know where the messages go - that is not quite precise from an individual message perspective, but it is right enough for what I want to achieve (for now ;-) ). I just want a record of each IP Address and which topic (or something that can be traced back to a topic) they are connected to, from a high level, without having to instrument the clients (which can be upwards of 10,000, and I have no control or access over). Currently, as I mentioned, the debug messages have exactly what I need for this phase: [2020-03-28 20:32:23,901] DEBUG Principal = User:ANONYMOUS is Allowed Operation = Read from host = x.x.x.x on resource = Topic:LITERAL: (kafka.authorizer.logger) Just figuring there must be a better way of getting this info rather than turning on debug. On Sat, Mar 28, 2020 at 4:15 PM Hans Jespersen wrote: > I can tell from the terminology you use that you are familiar with > traditional message queue products. Kafka is very different. Thats what > makes it so interesting and revolutionary in my opinion. > > Clients do not connect to topics because kafka is a distributed and > clustered system where topics are sharded into pieces called partitions and > the topic partitions are spread out across all the kafka brokers in the > cluster (and also replicated several more times across the cluster for > fault tolerance). When a client logically connects to a topic, its actually > making many connections to many nodes in the kafka cluster which enables > both parallel processing and fault tolerance. > > Also when a client consumes a message, the message is not removed from a > queue, it remains in kafka for many days (sometimes months or years). It is > not “taken off the queue” it is rather “copied from the commit log”. It can > be consumed again and again if needed because it is an immutable record of > an event that happened. > > Now getting back to your question of how to see where messages get > consumed (copied). The reality is that they go many places and can be > consumed many times. This makes tracing and tracking message delivery more > difficult but not impossible. There are many tools both open source and > commercial that can track data from producer to kafka (with replication) to > multiple consumers. They typically involve taking telemetry from both > clients (producers and consumers) and brokers (all of them as they act as a > cluster) and aggregate all the data to see the full flow of messages in the > system. Thats why the logs may seem overwelming and you need to look at the > logs of all the broker (and perhaps all the clients as well) to get the > full picture. > > -hans > > > On Mar 28, 2020, at 4:50 PM, Colin Ross wrote: > > > > Hi All - just started to use Kafka. Just one thing driving me nuts. I > want > > to get logs of each time a publisher or subscriber connects. I am trying > to > > just get the IP that they connected from and the topic to which they > > connected. I have managed to do this through enabling debug in the > > kafka-authorizer, however, the number of logs are overwhelming as is the > > update rate (looks like 2 per second per client). > > > > What I am actually trying to achieve is to understand where messages go, > so > > I would be more than happy to just see notifications when messages are > > actually sent and actually taken off the queue. > > > > Is there a more efficient way of achieving my goal than turning on debug? > > > > Cheers > > Rossi >
Newbie Question
Hi All - just started to use Kafka. Just one thing driving me nuts. I want to get logs of each time a publisher or subscriber connects. I am trying to just get the IP that they connected from and the topic to which they connected. I have managed to do this through enabling debug in the kafka-authorizer, however, the number of logs are overwhelming as is the update rate (looks like 2 per second per client). What I am actually trying to achieve is to understand where messages go, so I would be more than happy to just see notifications when messages are actually sent and actually taken off the queue. Is there a more efficient way of achieving my goal than turning on debug? Cheers Rossi
Re: Kafka trunk vs master branch
+1. Colin On Fri, Dec 27, 2019 at 12:29 PM Guozhang Wang wrote: > Hmm, I think that's pushed by someone by mistake, we can delete it. > > > Guozhang > > > On Thu, Dec 26, 2019 at 12:18 PM Matthias J. Sax > wrote: > >> Should we delete `master` ? (Was not even aware it exists...) >> >> >> -Matthias >> >> On 12/25/19 9:50 AM, M. Manna wrote: >> > +1 with what John mentioned. >> > >> > Master is more like a template that gets created for new repo. It’s not >> in >> > use for any Kafka activities (not that we know of). >> > >> > Regards, >> > >> > >> > On Wed, 25 Dec 2019 at 17:04, John Roesler wrote: >> > >> >> Hi Sachin, >> >> >> >> Trunk is the basis for development. I’m not sure what master is for, if >> >> anything. I’ve never used it for anything or even checked it out. >> >> >> >> The numbered release branches are used to develop patch releases. >> >> >> >> Releases are created from trunk, PRs should be made against trunk, etc. >> >> >> >> Thanks for asking! >> >> John >> >> >> >> On Wed, Dec 25, 2019, at 08:54, Sachin Mittal wrote: >> >>> Hello Folks, >> >>> I just wanted to know what commits goes into what branch. >> >>> >> >>> I see trunk branch which seems default and latest. >> >>> I also see master branch which seems bit behind trunk. >> >>> I also see different versions branches like 2.2, 2.3 and 2.4 which are >> >> also >> >>> actively updated. >> >>> >> >>> I wanted to know when forking kafka repo, which is the branch one >> should >> >>> use base off to build from source or do any active development. >> >>> >> >>> What is the difference between between trunk and master branch? >> >>> Also release branches are created from trunk or master branch? >> >>> >> >>> Also when issuing a pull request which is the general branch one >> should >> >> use >> >>> as target? >> >>> >> >>> Thanks >> >>> Sachin >> >>> >> >> >> > >> >> > > -- > Thanks, > Guozhang > > *Guozhang Wang | Software Engineer | Confluent | +1 607.339.8352 > <607.339.8352> * >
Re: [ANNOUNCE] New committer: Mickael Maison
Congratulations, Mickael! cheers, Colin On Thu, Nov 7, 2019, at 13:53, Bill Bejeck wrote: > Congratulations Mickael! Well deserved! > > -Bill > > On Thu, Nov 7, 2019 at 4:38 PM Jun Rao wrote: > > > Hi, Everyone, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer > > Mickael > > Maison. > > > > Mickael has been contributing to Kafka since 2016. He proposed and > > implemented multiple KIPs. He has also been propomating Kafka through blogs > > and public talks. > > > > Congratulations, Mickael! > > > > Thanks, > > > > Jun (on behalf of the Apache Kafka PMC) > > >
Re: [VOTE] 2.3.1 RC2
+ d...@kafka.apache.org On Tue, Oct 22, 2019, at 15:48, Colin McCabe wrote: > +1. I ran the broker, producer, consumer, etc. > > best, > Colin > > On Tue, Oct 22, 2019, at 13:32, Guozhang Wang wrote: > > +1. I've ran the quick start and unit tests. > > > > > > Guozhang > > > > On Tue, Oct 22, 2019 at 12:57 PM David Arthur wrote: > > > > > Thanks, Jonathon and Jason. I've updated the release notes along with the > > > signature and checksums. KAFKA-9053 was also missing. > > > > > > On Tue, Oct 22, 2019 at 3:47 PM Jason Gustafson > > > wrote: > > > > > > > +1 > > > > > > > > I ran the basic quickstart on the 2.12 artifact and verified > > > > signatures/checksums. > > > > > > > > I also looked over the release notes. I see that KAFKA-8950 is included, > > > so > > > > maybe they just need to be refreshed. > > > > > > > > Thanks for running the release! > > > > > > > > -Jason > > > > > > > > On Fri, Oct 18, 2019 at 5:23 AM David Arthur wrote: > > > > > > > > > We found a few more critical issues and so have decided to do one more > > > RC > > > > > for 2.3.1. Please review the release notes: > > > > > > > > https://home.apache.org/~davidarthur/kafka-2.3.1-rc2/RELEASE_NOTES.html > > > > > > > > > > > > > > > *** Please download, test and vote by Tuesday, October 22, 9pm PDT > > > > > > > > > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > > > > > > > > > https://kafka.apache.org/KEYS > > > > > > > > > > > > > > > * Release artifacts to be voted upon (source and binary): > > > > > > > > > > https://home.apache.org/~davidarthur/kafka-2.3.1-rc2/ > > > > > > > > > > > > > > > * Maven artifacts to be voted upon: > > > > > > > > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > > > > > > > > > > > > * Javadoc: > > > > > > > > > > https://home.apache.org/~davidarthur/kafka-2.3.1-rc2/javadoc/ > > > > > > > > > > > > > > > * Tag to be voted upon (off 2.3 branch) is the 2.3.1 tag: > > > > > > > > > > https://github.com/apache/kafka/releases/tag/2.3.1-rc2 > > > > > > > > > > > > > > > * Documentation: > > > > > > > > > > https://kafka.apache.org/23/documentation.html > > > > > > > > > > > > > > > * Protocol: > > > > > > > > > > https://kafka.apache.org/23/protocol.html > > > > > > > > > > > > > > > * Successful Jenkins builds to follow > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > David > > > > > > > > > > > > > > > > > > -- > > > David Arthur > > > > > > > > > -- > > -- Guozhang > > >
Re: [VOTE] 2.3.1 RC2
+1. I ran the broker, producer, consumer, etc. best, Colin On Tue, Oct 22, 2019, at 13:32, Guozhang Wang wrote: > +1. I've ran the quick start and unit tests. > > > Guozhang > > On Tue, Oct 22, 2019 at 12:57 PM David Arthur wrote: > > > Thanks, Jonathon and Jason. I've updated the release notes along with the > > signature and checksums. KAFKA-9053 was also missing. > > > > On Tue, Oct 22, 2019 at 3:47 PM Jason Gustafson > > wrote: > > > > > +1 > > > > > > I ran the basic quickstart on the 2.12 artifact and verified > > > signatures/checksums. > > > > > > I also looked over the release notes. I see that KAFKA-8950 is included, > > so > > > maybe they just need to be refreshed. > > > > > > Thanks for running the release! > > > > > > -Jason > > > > > > On Fri, Oct 18, 2019 at 5:23 AM David Arthur wrote: > > > > > > > We found a few more critical issues and so have decided to do one more > > RC > > > > for 2.3.1. Please review the release notes: > > > > > > https://home.apache.org/~davidarthur/kafka-2.3.1-rc2/RELEASE_NOTES.html > > > > > > > > > > > > *** Please download, test and vote by Tuesday, October 22, 9pm PDT > > > > > > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > > > > > > > https://kafka.apache.org/KEYS > > > > > > > > > > > > * Release artifacts to be voted upon (source and binary): > > > > > > > > https://home.apache.org/~davidarthur/kafka-2.3.1-rc2/ > > > > > > > > > > > > * Maven artifacts to be voted upon: > > > > > > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > > > > > > > > > * Javadoc: > > > > > > > > https://home.apache.org/~davidarthur/kafka-2.3.1-rc2/javadoc/ > > > > > > > > > > > > * Tag to be voted upon (off 2.3 branch) is the 2.3.1 tag: > > > > > > > > https://github.com/apache/kafka/releases/tag/2.3.1-rc2 > > > > > > > > > > > > * Documentation: > > > > > > > > https://kafka.apache.org/23/documentation.html > > > > > > > > > > > > * Protocol: > > > > > > > > https://kafka.apache.org/23/protocol.html > > > > > > > > > > > > * Successful Jenkins builds to follow > > > > > > > > > > > > Thanks! > > > > > > > > David > > > > > > > > > > > > > -- > > David Arthur > > > > > -- > -- Guozhang >
Re: PR review
Hi M. Manna, I left a review. Take a look. Sorry for the delays. best, Colin On Mon, Jul 8, 2019, at 14:38, M. Manna wrote: > Hello, > > A few requests have been sent already. Could this please be reviewed ? Our > business implementation is holding due to this change. > > > > On Thu, 4 Jul 2019 at 13:33, M. Manna wrote: > > > https://github.com/apache/kafka/pull/6771 > > > > Could this be reviewed please ? > > > > On Wed, 3 Jul 2019 at 11:35, M. Manna wrote: > > > >> https://github.com/apache/kafka/pull/6771 > >> > >> Bouncing both users and dev to get some activity going. We are waiting > >> for a while to get this KIP pr merged. > >> > >> Could someone please review? > >> > >> Thanks, > >> > >> On Sun, 30 Jun 2019 at 08:59, M. Manna wrote: > >> > >>> https://github.com/apache/kafka/pull/6771 > >>> > >>> Hello, > >>> > >>> Could the above PR can be reviewed? This has been waiting for a long > >>> time. > >>> > >>> Just to mention, the package name should have "internal". Round-robin > >>> partitioning should have been supported without/without a key from the > >>> beginning. It provides user a guaranteed round-robin partitioning without > >>> having to regard for key values (e.g. null/not null). From our business > >>> side, this is a Kafka internal logic. Hence, the placement inside > >>> "internal" package. > >>> > >>> Thanks, > >>> > >> >
Re: [ANNOUNCE] Apache Kafka 2.3.0
Thanks to everyone who reviewed the Apache blog post about 2.3. It's live now at https://blogs.apache.org/kafka/date/20190624 Plus, Tim Berglund made a video about what's new in this release. https://www.youtube.com/watch?v=sNqwJT2WguQ Finally, check out Stéphane Maarek's video about 2.3 here: https://www.youtube.com/watch?v=YutjYKSGd64 cheers, Colin On Tue, Jun 25, 2019, at 09:40, Colin McCabe wrote: > The Apache Kafka community is pleased to announce the release for > Apache Kafka 2.3.0. > This release includes several new features, including: > > - There have been several improvements to the Kafka Connect REST API. > - Kafka Connect now supports incremental cooperative rebalancing. > - Kafka Streams now supports an in-memory session store and window > store. > - The AdminClient now allows users to determine what operations they > are authorized to perform on topics. > - There is a new broker start time metric. > - JMXTool can now connect to secured RMI ports. > - An incremental AlterConfigs API has been added. The old AlterConfigs > API has been deprecated. > - We now track partitions which are under their min ISR count. > - Consumers can now opt-out of automatic topic creation, even when it > is enabled on the broker. > - Kafka components can now use external configuration stores (KIP-421) > - We have implemented improved replica fetcher behavior when errors are > encountered > > All of the changes in this release can be found in the release notes: > https://www.apache.org/dist/kafka/2.3.0/RELEASE_NOTES.html > > > You can download the source and binary release (Scala 2.11 and 2.12) from: > https://kafka.apache.org/downloads#2.3.0 > > All of the changes in this release can be found in the release notes: > https://www.apache.org/dist/kafka/2.3.0/RELEASE_NOTES.html > > You can download the source and binary release (Scala 2.11 and 2.12) from: > https://kafka.apache.org/downloads#2.3.0 > > --- > > Apache Kafka is a distributed streaming platform with four core APIs: > > ** The Producer API allows an application to publish a stream records > to one or more Kafka topics. > > ** The Consumer API allows an application to subscribe to one or more > topics and process the stream of records produced to them. > > ** The Streams API allows an application to act as a stream processor, > consuming an input stream from one or more topics and producing an > output stream to one or more output topics, effectively transforming > the input streams to output streams. > > ** The Connector API allows building and running reusable producers or > consumers that connect Kafka topics to existing applications or data > systems. For example, a connector to a relational database might > capture every change to a table. > > > With these APIs, Kafka can be used for two broad classes of application: > ** Building real-time streaming data pipelines that reliably get data > between systems or applications. > ** Building real-time streaming applications that transform or react to > the streams of data. > > Apache Kafka is in use at large and small companies worldwide, > including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, > Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and > Zalando, among others. > > A big thank you for the following 101 contributors to this release! > > Aishwarya Gune, Alex Diachenko, Alex Dunayevsky, Anna Povzner, Arabelle > Hou, Arjun Satish, A. Sophie Blee-Goldman, asutosh936, Bill Bejeck, Bob > Barrett, Boyang Chen, Brian Bushree, cadonna, Casey Green, Chase > Walden, Chia-Ping Tsai, Chris Egerton, Chris Steingen, Colin Hicks, > Colin Patrick McCabe, commandini, cwildman, Cyrus Vafadari, Dan > Norwood, David Arthur, Dejan Stojadinović, Dhruvil Shah, Doroszlai, > Attila, Ewen Cheslack-Postava, Fangbin Sun, Filipe Agapito, Florian > Hussonnois, Gardner Vickers, Guozhang Wang, Gwen Shapira, Hai-Dang Dam, > highluck, huxi, huxihx, Ismael Juma, Ivan Yurchenko, Jarrod Urban, > Jason Gustafson, John Roesler, José Armando García Sancio, Joyce Fee, > Jun Rao, KartikVK, Kengo Seki, Kevin Lu, khairy, Konstantine > Karantasis, Kristian Aurlien, Kyle Ambroff-Kao, lambdaliu, Lee Dongjin, > Lifei Chen, Lucas Bradstreet, Lysss, lzh3636, Magesh Nandakumar, > Manikumar Reddy, Mark Cho, Massimo Siani, Matthias J. Sax, Michael > Gruben Trejo, Mickael Maison, Murad, Nicholas Parker, opera443399, Paul > Davidson, pierDipi, pkleindl, Radai Rosenblatt, Rajini Sivaram, Randall > Hauch, Rohan, Rohan Desai, Ron Dagostino, Ryan Chen, saisandeep, > sandmannn, sdreynolds, Sebastián Ortega, Shaobo Liu, Sönke L
[ANNOUNCE] Apache Kafka 2.3.0
The Apache Kafka community is pleased to announce the release for Apache Kafka 2.3.0. This release includes several new features, including: - There have been several improvements to the Kafka Connect REST API. - Kafka Connect now supports incremental cooperative rebalancing. - Kafka Streams now supports an in-memory session store and window store. - The AdminClient now allows users to determine what operations they are authorized to perform on topics. - There is a new broker start time metric. - JMXTool can now connect to secured RMI ports. - An incremental AlterConfigs API has been added. The old AlterConfigs API has been deprecated. - We now track partitions which are under their min ISR count. - Consumers can now opt-out of automatic topic creation, even when it is enabled on the broker. - Kafka components can now use external configuration stores (KIP-421) - We have implemented improved replica fetcher behavior when errors are encountered All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/2.3.0/RELEASE_NOTES.html You can download the source and binary release (Scala 2.11 and 2.12) from: https://kafka.apache.org/downloads#2.3.0 All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/2.3.0/RELEASE_NOTES.html You can download the source and binary release (Scala 2.11 and 2.12) from: https://kafka.apache.org/downloads#2.3.0 --- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. ** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. With these APIs, Kafka can be used for two broad classes of application: ** Building real-time streaming data pipelines that reliably get data between systems or applications. ** Building real-time streaming applications that transform or react to the streams of data. Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 101 contributors to this release! Aishwarya Gune, Alex Diachenko, Alex Dunayevsky, Anna Povzner, Arabelle Hou, Arjun Satish, A. Sophie Blee-Goldman, asutosh936, Bill Bejeck, Bob Barrett, Boyang Chen, Brian Bushree, cadonna, Casey Green, Chase Walden, Chia-Ping Tsai, Chris Egerton, Chris Steingen, Colin Hicks, Colin Patrick McCabe, commandini, cwildman, Cyrus Vafadari, Dan Norwood, David Arthur, Dejan Stojadinović, Dhruvil Shah, Doroszlai, Attila, Ewen Cheslack-Postava, Fangbin Sun, Filipe Agapito, Florian Hussonnois, Gardner Vickers, Guozhang Wang, Gwen Shapira, Hai-Dang Dam, highluck, huxi, huxihx, Ismael Juma, Ivan Yurchenko, Jarrod Urban, Jason Gustafson, John Roesler, José Armando García Sancio, Joyce Fee, Jun Rao, KartikVK, Kengo Seki, Kevin Lu, khairy, Konstantine Karantasis, Kristian Aurlien, Kyle Ambroff-Kao, lambdaliu, Lee Dongjin, Lifei Chen, Lucas Bradstreet, Lysss, lzh3636, Magesh Nandakumar, Manikumar Reddy, Mark Cho, Massimo Siani, Matthias J. Sax, Michael Gruben Trejo, Mickael Maison, Murad, Nicholas Parker, opera443399, Paul Davidson, pierDipi, pkleindl, Radai Rosenblatt, Rajini Sivaram, Randall Hauch, Rohan, Rohan Desai, Ron Dagostino, Ryan Chen, saisandeep, sandmannn, sdreynolds, Sebastián Ortega, Shaobo Liu, Sönke Liebau, Stanislav Kozlovski, Suman BN, tadsul, Tcsalist, Ted Yu, Vahid Hashemian, Victoria Bialas, Viktor Somogyi, Viktor Somogyi-Vass, Vito Jeng, wenhoujx, Xiongqi Wu, Yaroslav Klymko, Zhanxiang (Patrick) Huang We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at https://kafka.apache.org/ Thank you! Regards, Colin
[RESULT] [VOTE] 2.3.0 RC3
Hi all, This vote passes with 6 +1 votes (3 of which are binding) and no 0 or -1 votes. Thanks to everyone who voted. +1 votes PMC Members: * Ismael Juma * Guozhang Wang * Gwen Shapira Community: * Kamal Chandraprakash * Jakub Scholz * Mickael Maison 0 votes * No votes -1 votes * No votes Vote thread: https://www.mail-archive.com/dev@kafka.apache.org/msg98814.html I'll continue with the release process and the release announcement will follow in the next few days. thanks, Colin On Mon, Jun 24, 2019, at 01:17, Mickael Maison wrote: > Thanks Colin for making a new RC for KAFKA-8564. > +1 (non binding) > I checked signatures and ran quickstart on the 2.12 binary > > On Mon, Jun 24, 2019 at 6:03 AM Gwen Shapira wrote: > > > > +1 (binding) > > Verified signatures, verified good build on jenkins, built from > > sources anyway and ran quickstart on the 2.11 binary. > > > > Looks good! > > > > On Sun, Jun 23, 2019 at 3:06 PM Jakub Scholz wrote: > > > > > > +1 (non-binding). I used the binaries and run some of my tests against > > > them. > > > > > > On Thu, Jun 20, 2019 at 12:03 AM Colin McCabe wrote: > > > > > > > Hi all, > > > > > > > > We discovered some problems with the second release candidate (RC2) of > > > > 2.3.0. Specifically, KAFKA-8564. I've created a new RC which includes > > > > the > > > > fix for this issue. > > > > > > > > Check out the release notes for the 2.3.0 release here: > > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/RELEASE_NOTES.html > > > > > > > > The vote will go until Saturday, June 22nd, or until we create another > > > > RC. > > > > > > > > * Kafka's KEYS file containing PGP keys we use to sign the release can > > > > be > > > > found here: > > > > https://kafka.apache.org/KEYS > > > > > > > > * The release artifacts to be voted upon (source and binary) are here: > > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/ > > > > > > > > * Maven artifacts to be voted upon: > > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > > > > > * Javadoc: > > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/javadoc/ > > > > > > > > * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag: > > > > https://github.com/apache/kafka/releases/tag/2.3.0-rc3 > > > > > > > > best, > > > > Colin > > > > > > > > C. > > > > > > > > > > > > -- > > Gwen Shapira > > Product Manager | Confluent > > 650.450.2760 | @gwenshap > > Follow us: Twitter | blog >
[VOTE] 2.3.0 RC3
Hi all, We discovered some problems with the second release candidate (RC2) of 2.3.0. Specifically, KAFKA-8564. I've created a new RC which includes the fix for this issue. Check out the release notes for the 2.3.0 release here: https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/RELEASE_NOTES.html The vote will go until Saturday, June 22nd, or until we create another RC. * Kafka's KEYS file containing PGP keys we use to sign the release can be found here: https://kafka.apache.org/KEYS * The release artifacts to be voted upon (source and binary) are here: https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/javadoc/ * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag: https://github.com/apache/kafka/releases/tag/2.3.0-rc3 best, Colin C.
Re: [kafka-clients] [VOTE] 2.3.0 RC1
On Tue, Jun 4, 2019, at 23:17, Colin McCabe wrote: > Hi all, > > This is the first candidate for the release of Apache Kafka 2.3.0. > > This release includes many new features, including: > * Support for incremental cooperative rebalancing > * An in-memory session store and window store for Kafka Streams > * An API for allowing users to determine what operations they are authorized > to perform on topics. > * A new broker start time metric. > * The ability for JMXTool to connect to secured RMI ports. > * A new and improved API for setting topic and broker configurations. > * Support for non-key joining in KTable One small correction here: support for non-key joining (KIP-213) slipped from 2.3 due to time constraints. Regards, Colin > * The ability to track the number of partitions which are under their min ISR > count. > * The ability for consumers to opt out of automatic topic creation, even when > it is enabled on the broker. > * The ability to use external configuration stores. > * Improved replica fetcher behavior when errors are encountered. > > Check out the release notes for the 2.3.0 release here: > https://home.apache.org/~cmccabe/kafka-2.3.0-rc1/RELEASE_NOTES.html > > The vote will go until Friday, June 7th, or until we create another RC. > > * Kafka's KEYS file containing PGP keys we use to sign the release can be > found here: > https://kafka.apache.org/KEYS > > * The release artifacts to be voted upon (source and binary) are here: > https://home.apache.org/~cmccabe/kafka-2.3.0-rc1/ > > * Maven artifacts to be voted upon: > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > * Javadoc: > https://home.apache.org/~cmccabe/kafka-2.3.0-rc1/javadoc/ > > * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag: > https://github.com/apache/kafka/releases/tag/2.3.0-rc1 > > thanks, > Colin > > > -- > You received this message because you are subscribed to the Google Groups > "kafka-clients" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kafka-clients+unsubscr...@googlegroups.com. > To post to this group, send email to kafka-clie...@googlegroups.com. > Visit this group at https://groups.google.com/group/kafka-clients. > To view this discussion on the web visit > https://groups.google.com/d/msgid/kafka-clients/461015c6-d018-40f6-a018-eaadf5c25f23%40www.fastmail.com > > <https://groups.google.com/d/msgid/kafka-clients/461015c6-d018-40f6-a018-eaadf5c25f23%40www.fastmail.com?utm_medium=email_source=footer>. > For more options, visit https://groups.google.com/d/optout.
[VOTE] 2.3.0 RC1
Hi all, This is the first candidate for the release of Apache Kafka 2.3.0. This release includes many new features, including: * Support for incremental cooperative rebalancing * An in-memory session store and window store for Kafka Streams * An API for allowing users to determine what operations they are authorized to perform on topics. * A new broker start time metric. * The ability for JMXTool to connect to secured RMI ports. * A new and improved API for setting topic and broker configurations. * Support for non-key joining in KTable. * The ability to track the number of partitions which are under their min ISR count. * The ability for consumers to opt out of automatic topic creation, even when it is enabled on the broker. * The ability to use external configuration stores. * Improved replica fetcher behavior when errors are encountered. Check out the release notes for the 2.3.0 release here: https://home.apache.org/~cmccabe/kafka-2.3.0-rc1/RELEASE_NOTES.html The vote will go until Friday, June 7th, or until we create another RC. * Kafka's KEYS file containing PGP keys we use to sign the release can be found here: https://kafka.apache.org/KEYS * The release artifacts to be voted upon (source and binary) are here: https://home.apache.org/~cmccabe/kafka-2.3.0-rc1/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~cmccabe/kafka-2.3.0-rc1/javadoc/ * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag: https://github.com/apache/kafka/releases/tag/2.3.0-rc1 thanks, Colin
Re: [ANNOUNCE] New Kafka PMC member: Matthias J. Sax
Congratulations, Matthias! best, Colin On Fri, Apr 19, 2019, at 14:55, James Cheng wrote: > Congrats!! > > -James > > Sent from my iPhone > > > On Apr 18, 2019, at 2:35 PM, Guozhang Wang wrote: > > > > Hello Everyone, > > > > I'm glad to announce that Matthias J. Sax is now a member of Kafka PM > > > > Matthias has been a committer since Jan. 2018, and since then he continued > > to be active in the community and made significant contributions the > > project. > > > > > > Congratulations to Matthias! > > > > -- Guozhang >
[ANNOUNCE] Apache Kafka 2.1.1
The Apache Kafka community is pleased to announce the release for Apache Kafka 2.1.1. This is a bugfix release for Kafka 2.1.0. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/2.1.1/RELEASE_NOTES.html You can download the source and binary release from: https://kafka.apache.org/downloads#2.1.1 --- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. ** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. With these APIs, Kafka can be used for two broad classes of application: ** Building real-time streaming data pipelines that reliably get data between systems or applications. ** Building real-time streaming applications that transform or react to the streams of data. Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 46 contributors to this release! Alex Diachenko, Andras Katona, Anna Povzner, Arjun Satish, Bill Bejeck, Bob Barrett, cadonna, Chia-Ping Tsai, Chris Egerton, Colin P. McCabe, cwildman, Cyrus Vafadari, Dhruvil Shah, Dong Lin, Flavien Raynaud, Guozhang Wang, hackerwin7, huxi, Ismael Juma, Jacek Laskowski, Jakub Scholz, Jarek Rudzinski, Jason Gustafson, John Roesler, Jonathan Santilli, Konstantine Karantasis, lambdaliu, layfe, Lee Dongjin, linyli001, Lucas Bradstreet, Magesh Nandakumar, Mark Cho, Matthias J. Sax, mingaliu, Pasquale Vazzana, Rajini Sivaram, Randall Hauch, Renato Mefi, Robert Yokota, Ron Dagostino, slim, Stig Rohde Døssing, Vahid Hashemian, and Xi Yang. We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at https://kafka.apache.org/ Thank you! Regards, Colin McCabe
Re: [kafka-clients] [VOTE] 2.1.1 RC2
P.S. I have added KAFKA-7897 to the release notes. Good catch, Jason. best, Colin On Fri, Feb 15, 2019, at 00:49, Colin McCabe wrote: > Hi all, > > With 7 non-binding +1 votes, 3 binding +1 votes, no +0 votes, and no -1 > votes, the vote passes. > > Thanks, all! > > cheers, > Colin > > > On Fri, Feb 15, 2019, at 00:07, Jonathan Santilli wrote: >> >> >> Hello, >> >> I have downloaded the source and executed integration and unit tests >> successfully. >> Ran kafka-monitor for about 1 hour without any issues. >> >> +1 >> >> Thanks for the release Colin. >> -- >> Jonathan Santilli >> >> >> >> On Fri, Feb 15, 2019 at 6:16 AM Jason Gustafson wrote: >>> Ran the quickstart against the 2.11 artifact and checked the release notes. >>> For some reason, KAFKA-7897 is not included in the notes, though I >>> definitely see it in the tagged version. The RC was probably created before >>> the JIRA was resolved. I think we can regenerate without another RC, so +1 >>> from me. >>> >>> Thanks Colin! >>> >>> On Thu, Feb 14, 2019 at 3:32 PM Jun Rao wrote: >>> >>> > Hi, Colin, >>> > >>> > Thanks for running the release. Verified the quickstart for 2.12 binary. >>> > +1 >>> > from me. >>> > >>> > Jun >>> > >>> > On Fri, Feb 8, 2019 at 12:02 PM Colin McCabe wrote: >>> > >>> > > Hi all, >>> > > >>> > > This is the third candidate for release of Apache Kafka 2.1.1. This >>> > > release includes many bug fixes for Apache Kafka 2.1. >>> > > >>> > > Compared to rc1, this release includes the following changes: >>> > > * MINOR: release.py: fix some compatibility problems. >>> > > * KAFKA-7897; Disable leader epoch cache when older message formats are >>> > > used >>> > > * KAFKA-7902: Replace original loginContext if SASL/OAUTHBEARER refresh >>> > > login fails >>> > > * MINOR: Fix more places where the version should be bumped from 2.1.0 >>> > > -> >>> > > 2.1.1 >>> > > * KAFKA-7890: Invalidate ClusterConnectionState cache for a broker if >>> > > the >>> > > hostname of the broker changes. >>> > > * KAFKA-7873; Always seek to beginning in KafkaBasedLog >>> > > * MINOR: Correctly set dev version in version.py >>> > > >>> > > Check out the release notes here: >>> > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/RELEASE_NOTES.html >>> > > >>> > > The vote will go until Wednesday, February 13st. >>> > > >>> > > * Release artifacts to be voted upon (source and binary): >>> > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/ >>> > > >>> > > * Maven artifacts to be voted upon: >>> > > https://repository.apache.org/content/groups/staging/ >>> > > >>> > > * Javadoc: >>> > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/javadoc/ >>> > > >>> > > * Tag to be voted upon (off 2.1 branch) is the 2.1.1 tag: >>> > > https://github.com/apache/kafka/releases/tag/2.1.1-rc2 >>> > > >>> > > * Jenkins builds for the 2.1 branch: >>> > > Unit/integration tests: https://builds.apache.org/job/kafka-2.1-jdk8/ >>> > > >>> > > Thanks to everyone who tested the earlier RCs. >>> > > >>> > > cheers, >>> > > Colin >>> > > >>> > > -- >>> > > You received this message because you are subscribed to the Google >>> > > Groups >>> > > "kafka-clients" group. >>> > > To unsubscribe from this group and stop receiving emails from it, send >>> > > an >>> > > email to kafka-clients+unsubscr...@googlegroups.com >>> > > <mailto:kafka-clients%2bunsubscr...@googlegroups.com>. >>> > > To post to this group, send email to kafka-clie...@googlegroups.com. >>> > > Visit this group at https://groups.google.com/group/kafka-clients. >>> > > To view this discussion on the web visit >>> > > >>> > https://groups.google.com/d/msgid/kafka-clients/ea314ca1-d23a-47c4-8fc7-83b9b1c792db%40www.fastmail.com >>> > > . >>> > > For more options, visit https://groups.google.com/d/optout. >>> > > >>> > >> >> >> -- >> Santilli Jonathan > > > -- > You received this message because you are subscribed to the Google Groups > "kafka-clients" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kafka-clients+unsubscr...@googlegroups.com. > To post to this group, send email to kafka-clie...@googlegroups.com. > Visit this group at https://groups.google.com/group/kafka-clients. > To view this discussion on the web visit > https://groups.google.com/d/msgid/kafka-clients/dcfd1bcc-8960-4c7f-b95c-57e9a6aae0b7%40www.fastmail.com > > <https://groups.google.com/d/msgid/kafka-clients/dcfd1bcc-8960-4c7f-b95c-57e9a6aae0b7%40www.fastmail.com?utm_medium=email_source=footer>. > For more options, visit https://groups.google.com/d/optout.
Re: [kafka-clients] [VOTE] 2.1.1 RC2
Hi all, With 7 non-binding +1 votes, 3 binding +1 votes, no +0 votes, and no -1 votes, the vote passes. Thanks, all! cheers, Colin On Fri, Feb 15, 2019, at 00:07, Jonathan Santilli wrote: > > > Hello, > > I have downloaded the source and executed integration and unit tests > successfully. > Ran kafka-monitor for about 1 hour without any issues. > > +1 > > Thanks for the release Colin. > -- > Jonathan Santilli > > > > On Fri, Feb 15, 2019 at 6:16 AM Jason Gustafson wrote: >> Ran the quickstart against the 2.11 artifact and checked the release notes. >> For some reason, KAFKA-7897 is not included in the notes, though I >> definitely see it in the tagged version. The RC was probably created before >> the JIRA was resolved. I think we can regenerate without another RC, so +1 >> from me. >> >> Thanks Colin! >> >> On Thu, Feb 14, 2019 at 3:32 PM Jun Rao wrote: >> >> > Hi, Colin, >> > >> > Thanks for running the release. Verified the quickstart for 2.12 binary. +1 >> > from me. >> > >> > Jun >> > >> > On Fri, Feb 8, 2019 at 12:02 PM Colin McCabe wrote: >> > >> > > Hi all, >> > > >> > > This is the third candidate for release of Apache Kafka 2.1.1. This >> > > release includes many bug fixes for Apache Kafka 2.1. >> > > >> > > Compared to rc1, this release includes the following changes: >> > > * MINOR: release.py: fix some compatibility problems. >> > > * KAFKA-7897; Disable leader epoch cache when older message formats are >> > > used >> > > * KAFKA-7902: Replace original loginContext if SASL/OAUTHBEARER refresh >> > > login fails >> > > * MINOR: Fix more places where the version should be bumped from 2.1.0 -> >> > > 2.1.1 >> > > * KAFKA-7890: Invalidate ClusterConnectionState cache for a broker if the >> > > hostname of the broker changes. >> > > * KAFKA-7873; Always seek to beginning in KafkaBasedLog >> > > * MINOR: Correctly set dev version in version.py >> > > >> > > Check out the release notes here: >> > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/RELEASE_NOTES.html >> > > >> > > The vote will go until Wednesday, February 13st. >> > > >> > > * Release artifacts to be voted upon (source and binary): >> > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/ >> > > >> > > * Maven artifacts to be voted upon: >> > > https://repository.apache.org/content/groups/staging/ >> > > >> > > * Javadoc: >> > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/javadoc/ >> > > >> > > * Tag to be voted upon (off 2.1 branch) is the 2.1.1 tag: >> > > https://github.com/apache/kafka/releases/tag/2.1.1-rc2 >> > > >> > > * Jenkins builds for the 2.1 branch: >> > > Unit/integration tests: https://builds.apache.org/job/kafka-2.1-jdk8/ >> > > >> > > Thanks to everyone who tested the earlier RCs. >> > > >> > > cheers, >> > > Colin >> > > >> > > -- >> > > You received this message because you are subscribed to the Google Groups >> > > "kafka-clients" group. >> > > To unsubscribe from this group and stop receiving emails from it, send an >> > > email to kafka-clients+unsubscr...@googlegroups.com >> > > <mailto:kafka-clients%2bunsubscr...@googlegroups.com>. >> > > To post to this group, send email to kafka-clie...@googlegroups.com. >> > > Visit this group at https://groups.google.com/group/kafka-clients. >> > > To view this discussion on the web visit >> > > >> > https://groups.google.com/d/msgid/kafka-clients/ea314ca1-d23a-47c4-8fc7-83b9b1c792db%40www.fastmail.com >> > > . >> > > For more options, visit https://groups.google.com/d/optout. >> > > >> > > > > -- > Santilli Jonathan
Re: [VOTE] 2.1.1 RC2
Thanks, everyone, for taking a look! Just as a reminder, the vote will run until Wednesday. best, Colin On Mon, Feb 11, 2019, at 05:17, Satish Duggana wrote: > +1 (non-binding) > > - Ran testAll/releaseTarGzAll successfully with NO failures. > - Ran through quickstart of core/streams on builds generated from tag > - Ran few internal apps targeting to topics on 3 node cluster. > > Thanks for running the release Colin! > > Satish. > > On Mon, Feb 11, 2019 at 2:29 PM Mickael Maison > wrote: > > > > +1 (non binding) > > I built from source and ran Quickstart. All good. Thanks Colin > > > > > > On Mon, Feb 11, 2019 at 12:43 AM Vahid Hashemian > > wrote: > > > > > > +1 > > > > > > I built from the source, and ran the Quickstart on Ubuntu 18.04 with Java > > > 8. > > > Some unit tests fail for me but none consistently across several runs. > > > > > > Thanks for running the release Colin. > > > > > > --Vahid > > > > > > On Sat, Feb 9, 2019 at 2:24 PM Jakub Scholz wrote: > > > > > > > +1 (non-binding). I built it from source and run my tests. Everything > > > > seems > > > > to be fine. > > > > > > > > On Sat, Feb 9, 2019 at 12:10 AM Magnus Edenhill > > > > wrote: > > > > > > > > > +1 > > > > > > > > > > Passes librdkafka test suite. > > > > > > > > > > Den fre 8 feb. 2019 kl 21:02 skrev Colin McCabe : > > > > > > > > > > > Hi all, > > > > > > > > > > > > This is the third candidate for release of Apache Kafka 2.1.1. This > > > > > > release includes many bug fixes for Apache Kafka 2.1. > > > > > > > > > > > > Compared to rc1, this release includes the following changes: > > > > > > * MINOR: release.py: fix some compatibility problems. > > > > > > * KAFKA-7897; Disable leader epoch cache when older message formats > > > > > > are > > > > > > used > > > > > > * KAFKA-7902: Replace original loginContext if SASL/OAUTHBEARER > > > > > > refresh > > > > > > login fails > > > > > > * MINOR: Fix more places where the version should be bumped from > > > > > > 2.1.0 > > > > -> > > > > > > 2.1.1 > > > > > > * KAFKA-7890: Invalidate ClusterConnectionState cache for a broker > > > > > > if > > > > the > > > > > > hostname of the broker changes. > > > > > > * KAFKA-7873; Always seek to beginning in KafkaBasedLog > > > > > > * MINOR: Correctly set dev version in version.py > > > > > > > > > > > > Check out the release notes here: > > > > > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/RELEASE_NOTES.html > > > > > > > > > > > > The vote will go until Wednesday, February 13st. > > > > > > > > > > > > * Release artifacts to be voted upon (source and binary): > > > > > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/ > > > > > > > > > > > > * Maven artifacts to be voted upon: > > > > > > https://repository.apache.org/content/groups/staging/ > > > > > > > > > > > > * Javadoc: > > > > > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/javadoc/ > > > > > > > > > > > > * Tag to be voted upon (off 2.1 branch) is the 2.1.1 tag: > > > > > > https://github.com/apache/kafka/releases/tag/2.1.1-rc2 > > > > > > > > > > > > * Jenkins builds for the 2.1 branch: > > > > > > Unit/integration tests: > > > > > > https://builds.apache.org/job/kafka-2.1-jdk8/ > > > > > > > > > > > > Thanks to everyone who tested the earlier RCs. > > > > > > > > > > > > cheers, > > > > > > Colin > > > > > > > > > > > > > > > >
[VOTE] 2.1.1 RC2
Hi all, This is the third candidate for release of Apache Kafka 2.1.1. This release includes many bug fixes for Apache Kafka 2.1. Compared to rc1, this release includes the following changes: * MINOR: release.py: fix some compatibility problems. * KAFKA-7897; Disable leader epoch cache when older message formats are used * KAFKA-7902: Replace original loginContext if SASL/OAUTHBEARER refresh login fails * MINOR: Fix more places where the version should be bumped from 2.1.0 -> 2.1.1 * KAFKA-7890: Invalidate ClusterConnectionState cache for a broker if the hostname of the broker changes. * KAFKA-7873; Always seek to beginning in KafkaBasedLog * MINOR: Correctly set dev version in version.py Check out the release notes here: http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/RELEASE_NOTES.html The vote will go until Wednesday, February 13st. * Release artifacts to be voted upon (source and binary): http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/ * Javadoc: http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/javadoc/ * Tag to be voted upon (off 2.1 branch) is the 2.1.1 tag: https://github.com/apache/kafka/releases/tag/2.1.1-rc2 * Jenkins builds for the 2.1 branch: Unit/integration tests: https://builds.apache.org/job/kafka-2.1-jdk8/ Thanks to everyone who tested the earlier RCs. cheers, Colin
Re: [VOTE] 2.1.1 RC1
(+all lists) Hi Eno, Thanks for testing this. Those tests passed in the Jenkins build we did here: https://builds.apache.org/job/kafka-2.1-jdk8/118/ Perhaps there is an environment issue at play here? Do you get the same failures running those tests on the 2.1 release? Best, Colin On Wed, Jan 30, 2019, at 09:11, Eno Thereska wrote: > Hi Colin, > > I've been running the tests and so far I get the following failures. Are > they known? > > kafka.server.ReplicaManagerQuotasTest > shouldGetBothMessagesIfQuotasAllow > FAILED > kafka.server.ReplicaManagerQuotasTest > > testCompleteInDelayedFetchWithReplicaThrottling FAILED > kafka.server.ReplicaManagerQuotasTest > > shouldExcludeSubsequentThrottledPartitions FAILED > kafka.server.ReplicaManagerQuotasTest > > shouldGetNoMessagesIfQuotasExceededOnSubsequentPartitions FAILED > kafka.server.ReplicaManagerQuotasTest > > shouldIncludeInSyncThrottledReplicas FAILED > > Thanks > Eno > > On Sun, Jan 27, 2019 at 9:46 PM Colin McCabe wrote: > > > Hi all, > > > > This is the second candidate for release of Apache Kafka 2.1.1. This > > release includes many bug fixes for Apache Kafka 2.1. > > > > Compared to rc0, this release includes the following changes: > > * MINOR: Upgrade ducktape to 0.7.5 (#6197) > > * KAFKA-7837: Ensure offline partitions are picked up as soon as possible > > when shrinking ISR > > * tests/kafkatest/__init__.py now contains __version__ = '2.1.1' rather > > than '2.1.1.dev0' > > * Maven artifacts should be properly staged this time > > * I have added my GPG key to https://kafka.apache.org/KEYS > > > > Check out the release notes here: > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/RELEASE_NOTES.html > > > > The vote will go until Friday, February 1st. > > > > * Release artifacts to be voted upon (source and binary): > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/ > > > > * Maven artifacts to be voted upon: > > https://repository.apache.org/content/groups/staging/ > > > > * Javadoc: > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/javadoc/ > > > > * Tag to be voted upon (off 2.1 branch) is the 2.1.1 tag: > > https://github.com/apache/kafka/releases/tag/2.1.1-rc1 > > > > * Successful Jenkins builds for the 2.1 branch: > > Unit/integration tests: https://builds.apache.org/job/kafka-2.1-jdk8/118/ > > > > thanks, > > Colin > > >
Re: [VOTE] 2.1.1 RC1
Hi Eno, Thanks for testing this. Those tests passed in the Jenkins build we did here: https://builds.apache.org/job/kafka-2.1-jdk8/118/ Perhaps there is an environment issue at play here? Do you get the same failures running those tests on the 2.1 release? Best, Colin On Wed, Jan 30, 2019, at 09:11, Eno Thereska wrote: > Hi Colin, > > I've been running the tests and so far I get the following failures. Are > they known? > > kafka.server.ReplicaManagerQuotasTest > shouldGetBothMessagesIfQuotasAllow > FAILED > kafka.server.ReplicaManagerQuotasTest > > testCompleteInDelayedFetchWithReplicaThrottling FAILED > kafka.server.ReplicaManagerQuotasTest > > shouldExcludeSubsequentThrottledPartitions FAILED > kafka.server.ReplicaManagerQuotasTest > > shouldGetNoMessagesIfQuotasExceededOnSubsequentPartitions FAILED > kafka.server.ReplicaManagerQuotasTest > > shouldIncludeInSyncThrottledReplicas FAILED > > Thanks > Eno > > On Sun, Jan 27, 2019 at 9:46 PM Colin McCabe wrote: > > > Hi all, > > > > This is the second candidate for release of Apache Kafka 2.1.1. This > > release includes many bug fixes for Apache Kafka 2.1. > > > > Compared to rc0, this release includes the following changes: > > * MINOR: Upgrade ducktape to 0.7.5 (#6197) > > * KAFKA-7837: Ensure offline partitions are picked up as soon as possible > > when shrinking ISR > > * tests/kafkatest/__init__.py now contains __version__ = '2.1.1' rather > > than '2.1.1.dev0' > > * Maven artifacts should be properly staged this time > > * I have added my GPG key to https://kafka.apache.org/KEYS > > > > Check out the release notes here: > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/RELEASE_NOTES.html > > > > The vote will go until Friday, February 1st. > > > > * Release artifacts to be voted upon (source and binary): > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/ > > > > * Maven artifacts to be voted upon: > > https://repository.apache.org/content/groups/staging/ > > > > * Javadoc: > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/javadoc/ > > > > * Tag to be voted upon (off 2.1 branch) is the 2.1.1 tag: > > https://github.com/apache/kafka/releases/tag/2.1.1-rc1 > > > > * Successful Jenkins builds for the 2.1 branch: > > Unit/integration tests: https://builds.apache.org/job/kafka-2.1-jdk8/118/ > > > > thanks, > > Colin > > >
[VOTE] 2.1.1 RC1
Hi all, This is the second candidate for release of Apache Kafka 2.1.1. This release includes many bug fixes for Apache Kafka 2.1. Compared to rc0, this release includes the following changes: * MINOR: Upgrade ducktape to 0.7.5 (#6197) * KAFKA-7837: Ensure offline partitions are picked up as soon as possible when shrinking ISR * tests/kafkatest/__init__.py now contains __version__ = '2.1.1' rather than '2.1.1.dev0' * Maven artifacts should be properly staged this time * I have added my GPG key to https://kafka.apache.org/KEYS Check out the release notes here: http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/RELEASE_NOTES.html The vote will go until Friday, February 1st. * Release artifacts to be voted upon (source and binary): http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/ * Javadoc: http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/javadoc/ * Tag to be voted upon (off 2.1 branch) is the 2.1.1 tag: https://github.com/apache/kafka/releases/tag/2.1.1-rc1 * Successful Jenkins builds for the 2.1 branch: Unit/integration tests: https://builds.apache.org/job/kafka-2.1-jdk8/118/ thanks, Colin
[VOTE] 2.1.1 RC0
Hi all, This is the first candidate for release of Apache Kafka 2.1.1. This release includes many bug fixes for Apache Kafka 2.1. Check out the release notes here: http://home.apache.org/~cmccabe/kafka-2.1.1-rc0/RELEASE_NOTES.html The vote will go until Wednesday, January 30th. * Release artifacts to be voted upon (source and binary): http://home.apache.org/~cmccabe/kafka-2.1.1-rc0/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/ * Javadoc: http://home.apache.org/~cmccabe/kafka-2.1.1-rc0/javadoc/ * Tag to be voted upon (off 2.1 branch) is the 2.1.1 tag: https://github.com/apache/kafka/releases/tag/2.1.1-rc0 * Successful Jenkins builds for the 2.1 branch: Unit/integration tests: https://builds.apache.org/job/kafka-2.1-jdk8/116/ Note that we are still investigating two recent system test failures: one in StreamsUpgradeTest, and another in TestUpgrade. We believe that the failures are spurious, and are re-running the tests. Also, I am working on getting my PGP key added to http://kafka.apache.org/KEYS. thanks, Colin
Packaging jks certificates into an uber jar and accessing them for kafka client or otherwise?
I've been trying to read from kafka via a spark streaming client. I found out spark cluster doesn't have certificates deployed. Then I tried using the same local certificates I've been testing with by packing them in an uber jar and getting a File handle from the Classloader resource. But I'm getting a File Not Found exception. These are jks certificates. Is anybody aware how to package certificates in a jar with a kafka client preferably the spark one?
Re: ConsumerGroupCommand tool improvement?
On Mon, Aug 6, 2018, at 12:04, Vahid S Hashemian wrote:
> Hi Colin,
>
> Thanks for considering the idea and sharing your feedback.
>
> The improvements I proposed can be achieved, to some extend, using the
> AdminClient API and the Consumer Group CLI tool. But they won't fully
> support the proposal.
>
> For example,
> Regular expressions are not supported on the groups
> Topic / Client filtering is not supported across all groups
>
> So the reason for proposing the idea was to see if other Kafka users are
> also interested in some of these features so we can remove the burden of
> them writing custom code around existing consumer group features, and make
> those features built into Kafka Consumer Group Command and AdminClient
> API.
Hmm. If you're writing Java code that calls the APIs, though, this is easy,
right? You can filter the groups in the cluster with a regular expression with
just a single line of code in Java 8.
> adminClient.listConsumerGroups().all().stream().
>filter(listing -> listing.groupId().matches(myRegex)).
>collect(Collectors.toList());
An option to filter ListConsumerGroups by a regular expression might actually
be less easy-to-use for most users than this simple filter, since users would
have to read the JavaDoc for our APIs.
Maybe there are some use-cases where it makes sense to add regex support to the
command-line tools, though.
I guess the reason why I am pushing back on this is that regular expressions
add a lot of complexity to the API and make it harder for us to meet our
backwards compatibility guarantees. For example, Java regular expressions
changed slightly between Java 7 and Java 8
best,
Colin
>
> Thanks again!
> --Vahid
>
>
>
> From: Colin McCabe
> To: users@kafka.apache.org
> Date: 08/03/2018 04:16 PM
> Subject:Re: ConsumerGroupCommand tool improvement?
>
>
>
> Hi Vahid,
>
> Interesting idea.
>
> It seems like if you're using the AdminClient APIs programmatically, you
> can just do the filtering yourself in a more flexible way than what we
> could provide.
>
> On the other hand, if you're using the ./bin/consumer-groups.sh
> command-line tool, why not use grep or a similar tool to filter the
> output? Maybe there is some additional functionality in supporting
> regexes in the command-line tool, but it also seems like it might be kind
> of complex as well. Do you have some examples where having regex support
> int the tool would be much easier than the traditional way of piping the
> output to grep, awk, and sed?
>
> best,
> Colin
>
>
> On Thu, Aug 2, 2018, at 14:23, Vahid S Hashemian wrote:
> > Hi all,
> >
> > A requirement has been raised by a colleague and I wanted to see if
> there
> > is any interest in the community in adding the functionality to Apache
> > Kafka.
> >
> > ConsumerGroupCommand tool in describe ('--describe' or '--describe
> > --offsets') mode currently lists all topics the group has consumed from
> > and all consumers with assigned partitions for a single group.
> > The idea is to allow filtering of topics, consumers (client ids), and
> even
> > groups using regular expressions. This will allow the tool to handle use
>
> > cases such as
> > What's the status of a particular consumer (or consumers) in all the
> > groups they are consuming from? (for example to check if they are
> lagging
> > behind in all groups)
> > What consumer groups are consuming from a topic (or topics) and what's
> the
> > lag for each group?
> > Limit the existing result to the topics/consumers of interest (for
> groups
> > with several topics/consumers)
> > ...
> >
> > This would potentially lead to enhancing the AdminClient API as well.
> >
> > If the community also sees a value in this, I could start drafting a
> KIP.
> >
> > Thanks for your feedback.
> > --Vahid
> >
>
>
>
>
>
Re: ConsumerGroupCommand tool improvement?
Hi Vahid,
Interesting idea.
It seems like if you're using the AdminClient APIs programmatically, you can
just do the filtering yourself in a more flexible way than what we could
provide.
On the other hand, if you're using the ./bin/consumer-groups.sh command-line
tool, why not use grep or a similar tool to filter the output? Maybe there is
some additional functionality in supporting regexes in the command-line tool,
but it also seems like it might be kind of complex as well. Do you have some
examples where having regex support int the tool would be much easier than the
traditional way of piping the output to grep, awk, and sed?
best,
Colin
On Thu, Aug 2, 2018, at 14:23, Vahid S Hashemian wrote:
> Hi all,
>
> A requirement has been raised by a colleague and I wanted to see if there
> is any interest in the community in adding the functionality to Apache
> Kafka.
>
> ConsumerGroupCommand tool in describe ('--describe' or '--describe
> --offsets') mode currently lists all topics the group has consumed from
> and all consumers with assigned partitions for a single group.
> The idea is to allow filtering of topics, consumers (client ids), and even
> groups using regular expressions. This will allow the tool to handle use
> cases such as
> What's the status of a particular consumer (or consumers) in all the
> groups they are consuming from? (for example to check if they are lagging
> behind in all groups)
> What consumer groups are consuming from a topic (or topics) and what's the
> lag for each group?
> Limit the existing result to the topics/consumers of interest (for groups
> with several topics/consumers)
> ...
>
> This would potentially lead to enhancing the AdminClient API as well.
>
> If the community also sees a value in this, I could start drafting a KIP.
>
> Thanks for your feedback.
> --Vahid
>
Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
Thanks, Matthias, this looks great. It seems like these APIs could either be used against mock objects, or against real brokers running in the same process. Is there a way for the user to select which they want when using the API? Sorry if it's in the KIP and I missed it. cheers, Colin On Thu, Jan 11, 2018, at 18:06, Matthias J. Sax wrote: > Dear Kafka community, > > I want to propose KIP-247 to add public test utils to the Streams API. > The goal is to simplify testing of Kafka Streams applications. > > Please find details in the wiki: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams > > This is an initial KIP, and we hope to add more utility functions later. > Thus, this KIP is not comprehensive but a first step. Of course, we can > enrich this initial KIP if we think it falls too short. But we should > not aim to be comprehensive to keep the scope manageable. > > In fact, I think we should add some more helpers to simplify result > verification. I will update the KIP with this asap. Just wanted to start > the discussion early on. > > An initial WIP PR can be found here: > https://github.com/apache/kafka/pull/4402 > > I also included the user-list (please hit "reply-all" to include both > lists in this KIP discussion). > > Thanks a lot. > > > -Matthias > > > Email had 1 attachment: > + signature.asc > 1k (application/pgp-signature)
Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch
Thanks for the explanation. I still think it would be better to have the mutation operations require write ACLs, though. It might not be 100% intuitive for novice users, but the current split between Describe and Read is not intuitive for either novice or experienced users. In any case, I am +1 on the incremental improvement discussed in KIP-163. cheers, Colin On Sat, Jun 17, 2017, at 11:11, Hans Jespersen wrote: > > Offset commit is something that is done in the act of consuming (or > reading) Kafka messages. > Yes technically it is a write to the Kafka consumer offset topic but it's > much easier for > administers to think of ACLs in terms of whether the user is allowed to > write (Produce) or > read (Consume) messages and not the lower level semantics that are that > consuming is actually > reading AND writing (albeit only to the offset topic). > > -hans > > > > > > On Jun 17, 2017, at 10:59 AM, Viktor Somogyi <viktor.somo...@cloudera.com> > > wrote: > > > > Hi Vahid, > > > > +1 for OffsetFetch from me too. > > > > I also wanted to ask the strangeness of the permissions, like why is > > OffsetCommit a Read operation instead of Write which would intuitively make > > more sense to me. Perhaps any expert could shed some light on this? :) > > > > Viktor > > > > On Tue, Jun 13, 2017 at 2:38 PM, Vahid S Hashemian < > > vahidhashem...@us.ibm.com <mailto:vahidhashem...@us.ibm.com>> wrote: > > > >> Hi Michal, > >> > >> Thanks a lot for your feedback. > >> > >> Your statement about Heartbeat is fair and makes sense. I'll update the > >> KIP accordingly. > >> > >> --Vahid > >> > >> > >> > >> > >> From:Michal Borowiecki <michal.borowie...@openbet.com> > >> To:users@kafka.apache.org, Vahid S Hashemian < > >> vahidhashem...@us.ibm.com>, d...@kafka.apache.org > >> Date:06/13/2017 01:35 AM > >> Subject:Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL > >> Permission of OffsetFetch > >> -- > >> > >> > >> > >> Hi Vahid, > >> > >> +1 wrt OffsetFetch. > >> > >> The "Additional Food for Thought" mentions Heartbeat as a non-mutating > >> action. I don't think that's true as the GroupCoordinator updates the > >> latestHeartbeat field for the member and adds a new object to the > >> heartbeatPurgatory, see completeAndScheduleNextHeartbeatExpiration() > >> called from handleHeartbeat() > >> > >> NB added dev mailing list back into CC as it seems to have been lost along > >> the way. > >> > >> Cheers, > >> > >> Michał > >> > >> > >> On 12/06/17 18:47, Vahid S Hashemian wrote: > >> Hi Colin, > >> > >> Thanks for the feedback. > >> > >> To be honest, I'm not sure either why Read was selected instead of Write > >> for mutating APIs in the initial design (I asked Ewen on the corresponding > >> JIRA and he seemed unsure too). > >> Perhaps someone who was involved in the design can clarify. > >> > >> Thanks. > >> --Vahid > >> > >> > >> > >> > >> From: Colin McCabe *<cmcc...@apache.org <mailto:cmcc...@apache.org>>* > >> <cmcc...@apache.org <mailto:cmcc...@apache.org>> > >> To: *users@kafka.apache.org <mailto:users@kafka.apache.org>* > >> <users@kafka.apache.org <mailto:users@kafka.apache.org>> > >> Date: 06/12/2017 10:11 AM > >> Subject:Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL > >> Permission of OffsetFetch > >> > >> > >> > >> Hi Vahid, > >> > >> I think you make a valid point that the ACLs controlling group > >> operations are not very intuitive. > >> > >> This is probably a dumb question, but why are we using Read for mutating > >> APIs? Shouldn't that be Write? > >> > >> The distinction between Describe and Read makes a lot of sense for > >> Topics. A group isn't really something that you "read" from in the same > >> way as a topic, so it always felt kind of weird there. > >> > >> best, > >> Colin > >> > >> > >> On Thu, Jun 8, 2017, at 11:29, Vahid S Hashemian wrote: > >> > >> Hi all, > >
Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch
Hi Vahid, I think you make a valid point that the ACLs controlling group operations are not very intuitive. This is probably a dumb question, but why are we using Read for mutating APIs? Shouldn't that be Write? The distinction between Describe and Read makes a lot of sense for Topics. A group isn't really something that you "read" from in the same way as a topic, so it always felt kind of weird there. best, Colin On Thu, Jun 8, 2017, at 11:29, Vahid S Hashemian wrote: > Hi all, > > I'm resending my earlier note hoping it would spark some conversation > this > time around :) > > Thanks. > --Vahid > > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > To: dev <d...@kafka.apache.org>, "Kafka User" <users@kafka.apache.org> > Date: 05/30/2017 08:33 AM > Subject:KIP-163: Lower the Minimum Required ACL Permission of > OffsetFetch > > > > Hi, > > I started a new KIP to improve the minimum required ACL permissions of > some of the APIs: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-163%3A+Lower+the+Minimum+Required+ACL+Permission+of+OffsetFetch > > The KIP is to address KAFKA-4585. > > Feedback and suggestions are welcome! > > Thanks. > --Vahid > > > > >
Re: Kafka exception
Can you reproduce this with the latest version of Kafka?
best,
Colin
On Wed, Feb 15, 2017, at 22:29, 揣立武 wrote:
>
> Hi,all! Our program uses the high level consumer api(the version is
> 0.8.x). Sometimes the program will throw an exception in the 42th row in
> kafka.utils.IteratorTemplate class，the content is "throw new
> IllegalStateException("Expected item but none found.")".
>
> I think it is a race condition problem between the close thread and the
> consume thread. When the close thread calling the method
> ConsumerConnector.shutdown(), it will set ConsumerIterator's state is
> NOT_READY. But at the same time, the consume thread calls the method
> ConsumerIterator.hasNext() and goes to the 67th row in
> kafka.utils.IteratorTemplate class，the content is "if(state == DONE) {",
> the if will be false that means has a item. And when calling the
> ConsumerIterator.next(), it will throw that exception.
>
> Have you ever had this problem? Please tell me how to deal with it,
> thanks!
>
>
>
Re: Reg: Kafka HDFS Connector with (HDFS SSL enabled)
Hi, Just to be clear, HDFS doesn't use HTTP or HTTPS as its primary transport mechanism. Instead, HDFS uses the Hadoop RPC transport mechanism. So in general, it should not be necessary to configure SSL to connect a client to HDFS. HDFS does "support SSL" in the sense that the HDFS web UI can be configured to use it. You can also use HttpFS or WebHDFS to access HDFS, which might motivate you to configure SSL. Are you trying to configure one of these? best, Colin On Wed, Feb 15, 2017, at 11:03, BigData dev wrote: > Hi, > > Does Kafka HDFS Connect work with HDFS (SSL). As I see only properties in > security is > hdfs.authentication.kerberos, connect.hdfs.keytab,hdfs.namenode.principal > as these properties are all related to HDFS Kerberos. > > As from the configuration and code I see we pass only Kerberos > parameters, > not seen SSL configuration, so want to confirm will the Kafka HDFS > Connector works with HDFS (SSL enabled)? > > Could you please provide any information on this. > > > Thanks
Re: log.flush.interval.messages setting of Kafka 0.9.0.0
Hi, Based on a quick look at the code, the log.flush.interval.messages setting affects how frequently we call fsync on the underlying filesystem. As you mentioned, by default this is set to Long.MAX_VALUE, which effectively disables it. Keep in mind that the underlying operating system will flush data to disk even without the use of fsync. There are a few kernel parameters that control this on Linux: vm.dirty_bytes, vm.dirty_background_bytes, vm.dirty_writeback_centisecs, vm.dirty_expire_centisecs, and so forth. In other words, the kernel will not buffer an indefinite amount of data, even if it has a huge amount of memory. It will eventually be written to disk. I agree that having 3x replication can reduce the negative impact of disabling fsync this way. However, keep in mind that if all machines lose power at once, you will still lose the data in memory, even with 3x replication. This is one reason why users who require the highest level of data durability run with battery backup for their clusters. Another reason is because some hard drives on the market are not totally reliable about flushing the data to the platter when they claim to have done so. best, Colin On Sat, Dec 17, 2016, at 19:06, +8618611626818 wrote: > someone can help explain it？ > > 发自我的超级手机 > > 2016年12月16日 18:17于 Json Tu <kafka...@126.com>写道： > > > > Hi all, > > we have a cluster of 0.9.0.0 with 3 nodes, we have a topic with 3 replicas, > > and send it with ack -1, our sending latency is avg 7ms. I prepare to > > optimize performance of cluster through adjusting some params. > > we find our brokers has set config item as below, > > log.flush.interval.messages=1 > > and other relevant parameter is default, and I find the default value of > > log.flush.interval.messages is LONG.MAX_VALUE, because of setting this > > config will flush intiative that may affect performace . I wonder can I > > cancel this config item’s setting, and use default value. > > > > I think use default value may have two drawback as below. > > 1.recovery checkpoint can not be updated，so when load segments，it will scan > > from begin to end. > > 2.it may lose data when leader partition’s broker’s vm is restart，but I > > think 3 replicas can remedy this drawback if the network between them is > > good. > > > > any suggestions? thank you
Re: KIP-119: Drop Support for Scala 2.10 in Kafka 0.11
+1 (non-binding) for KIP-119 I agree that supporting the 2 most recently release versions of Java and Scala is a reasonable guideline. However, I think it makes sense to have a brief discussion when dropping support for an old Java or Scala version. It's a fairly infrequent event, and it's worth considering the situation at the time. Each release brings its own set of benefits, and people don't always adopt new releases with the same speed. For example, it took people a long time to move from JDK6, but JDK7 seems to have taken a shorter amount of time. If the upstream Java/Scala release cadences speed up enough, we might want to support more than 2 versions. And so forth. best, Colin On Fri, Feb 3, 2017, at 08:10, Ismael Juma wrote: > Hi Grant, > > That's an interesting point. It would be good to hear what others' think > of > making that the official policy instead of starting a discussion/vote > each > time. If there is consensus, I am happy to revise the KIPs. Otherwise, we > keep them as they are and discuss/vote on this instance only. > > Ismael > > On Fri, Feb 3, 2017 at 3:41 PM, Grant Henke <ghe...@cloudera.com> wrote: > > > Thanks for proposing this Ismael. This makes sense to me. > > > > In this KIP and the java KIP you state: > > > > A reasonable policy is to support the 2 most recently released versions so > > > that we can strike a good balance between supporting older versions, > > > maintainability and taking advantage of language and library > > improvements. > > > > > > What do you think about adjusting the KIP to instead vote on that as a > > standard policy for Java and Scala going forward? Something along the lines > > of: > > > > "Kafka's policy is to support the 2 most recently released versions of Java > > and Scala at a given time. When a new version becomes available, the > > supported versions will be updated in the next major release of Kafka." > > > > > > On Fri, Feb 3, 2017 at 8:30 AM, Ismael Juma <ism...@juma.me.uk> wrote: > > > > > Hi all, > > > > > > I have posted a KIP for dropping support for Scala 2.10 in Kafka 0.11: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 119%3A+Drop+Support+for+Scala+2.10+in+Kafka+0.11 > > > > > > Please take a look. Your feedback is appreciated. > > > > > > Thanks, > > > Ismael > > > > > > > > > > > -- > > Grant Henke > > Software Engineer | Cloudera > > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke > >
Re: If you run Kafka in AWS or Docker, how do you persist data?
Hello, We use docker for kafka on vm's with both nas and local disk. We mount the volumes externally. We havent had many problems at all, and a restart has cleared any issue. We are on .8.1 We are also started to deploy to aws. -- Colin +1 612 859 6129 Skype colin.p.clark On Mar 4, 2015, at 10:46 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, On Fri, Feb 27, 2015 at 1:36 AM, James Cheng jch...@tivo.com wrote: Hi, I know that Netflix might be talking about Kafka on AWS at the March meetup, but I wanted to bring up the topic anyway. I'm sure that some people are running Kafka in AWS. I'd say most, not some :) Is anyone running Kafka within docker in production? How does that work? Not us. When I was at DevOps Days in NYC last year, everyone was talking about Docker, but only about 2.5 people in the room actually really used it. For both of these, how do you persist data? If on AWS, do you use EBS? Do you use ephemeral storage and then rely on replication? And if using docker, do you persist data outside the docker container and on the host machine? We've used both EBD and local disks in AWS. We don't have Kafka replication, as far as I know. And related, how do you deal with broker failure? Do you simply replace it, and repopulate a new broker via replication? Or do you bring back up the broker with the persisted files? We monitor all Kafka pieces - producers, consumer, and brokers with SPM. We have alerts and anomaly detection enabled for various Kafka metrics (yeah, consumer lag being one of them). Broker failures have been very rare (we've used 0.7.2, 0.8.1.x, and are now on 0.8.2). When they happened a restart was typically enough. I can recall one instance where segments recovery tool a long time (minutes, maybe more than an hour), but this was 6 months ago. Trying to learn about what people are doing, beyond on premises and dedicated hardware. In my world almost everyone I talk to is in AWS. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Poll RESULTS: Producer/Consumer languages
Digital River will be releasing one soon as well that is integrated with the netflix stack for discovery, load balancing, metrics,etc. -- Colin Clark +1 612 859 6129 Skype colin.p.clark On Feb 11, 2015, at 7:20 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey Justin, I don't think LinkedIn is, but Confluent has made a pretty complete producer and consumer REST proxy that we will be releasing quite soon. -Jay On Wed, Feb 11, 2015 at 2:40 PM, Justin Maltat justin.mal...@gmail.com wrote: Hi, Is Linkedin planning to release its REST proxy server as an official release? This would be quite interesting for using Kafka in some existing heterogeneous environment. Justin On Thu, Jan 29, 2015 at 3:58 PM, Guozhang Wang wangg...@gmail.com wrote: Thanks for sharing this Otis, I am also quite surprised about Python / Go popularity. At LinkedIn we use a REST proxy server for our non-java clients, but introducing a second hop will also bring more overhead as well as complexities, such as producer acking and offset committing, etc. So I think by the end of the day non-java clients will still be around. Guozhang On Wed, Jan 28, 2015 at 8:54 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, I promised to share the results of this poll, and here they are: http://blog.sematext.com/2015/01/28/kafka-poll-results-producer-consumer/ List of surprises is there. I wonder if anyone else is surprised by any aspect of the breakdown, or is the breakdown just as you expected? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ -- -- Guozhang
Re: Resilient Producer
Logstash -- Colin Clark +1 612 859 6129 Skype colin.p.clark On Jan 28, 2015, at 10:47 AM, Gwen Shapira gshap...@cloudera.com wrote: It sounds like you are describing Flume, with SpoolingDirectory source (or exec source running tail) and Kafka channel. On Wed, Jan 28, 2015 at 10:39 AM, Fernando O. fot...@gmail.com wrote: Hi all, I'm evaluating using Kafka. I liked this thing of Facebook scribe that you log to your own machine and then there's a separate process that forwards messages to the central logger. With Kafka it seems that I have to embed the publisher in my app, and deal with any communication problem managing that on the producer side. I googled quite a bit trying to find a project that would basically use daemon that parses a log file and send the lines to the Kafka cluster (something like a tail file.log but instead of redirecting the output to the console: send it to kafka) Does anyone knows about something like that? Thanks! Fernando.
Re: Stale TopicMetadata
: [
1
],
partitionErrorCode: 0,
partitionId: 3,
leader: 1,
byteLength: 26
},
{
replicas: [
0
],
isr: [],
partitionErrorCode: 5,
partitionId: 4,
leader: -1,
byteLength: 22
}
],
byteLength: 133
}
],
responseSize: 188,
correlationId: -1000
}
Well, I can see from partition metadata that some partions have no leader
(-1), but my problem is that I actually rely on the brokers list to create
a pool of connections. And even when the broker 0 is down, I still get it
back from the metadata. Is this what is expected? I know it could be that
brokers are only a list of all places where that topic could be found, but
in that case couldn't we at least have a flag indicating wether that broker
is online or not?
Regards
--
The intuitive mind is a sacred gift and the
rational mind is a faithful servant. We have
created a society that honors the servant and
has forgotten the gift.
--
*Colin Blower*
/Software Engineer/
Barracuda Networks Inc.
+1 408-342-5576 (o)
Re: 0.8 protocol exception
I had the exact same problem when I started writing code for the new
protocol. This is an oddity with the way the protocol spec uses EBNF to
specify arrays.
Checkout the section on protocol primitives, especially arrays.
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProtocolPrimitiveTypes
Essentially, each array is preceded by its length as an int32.
With regards to your specific buffer:
Buffer 00 00 00 16 00 03 00 00 00 00 00 00 00 03 66 6f 6f 00 07 6d 79
54 6f 70 69 63
You seem to be missing the clientId as well as the array length. For
comparison the buffer, with the client I wrote, for clientId
perl-kafka and topics foo and myTopic:
00 00 00 26 00 03 00 00 00 00 00 2a 00 0a 70 65 72 6c 2d 6b 61 66 6b 61
00 00 00 02 00 03 66 6f 6f 00 07 6d 79 54 6f 70 69 63
On 07/08/2013 04:12 AM, Vinicius Carvalho wrote:
Ok, so I've found out the error: The documentation is outdated, the
MetadataRequest BNF should be:
NumberOfTopics [TopicList]
Had to check the scala source code for that.
Is there a place with a most to date doc?
Regards
On Mon, Jul 8, 2013 at 6:42 AM, Vinicius Carvalho
viniciusccarva...@gmail.com wrote:
Hi there. I'm building the 0.8 version of a client to nodejs. I never
coded for node and most of my code is following what the prozees guys did
(I'm talking to them on updating the lib)
But, I'm facing some errors when I test a very simple metadata request.
I'm getting this exception on kafka:
java.nio.BufferUnderflowException
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:145)
at java.nio.ByteBuffer.get(ByteBuffer.java:694)
at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:38)
at
kafka.api.TopicMetadataRequest$$anonfun$readFrom$1.apply(TopicMetadataRequest.scala:44)
at
kafka.api.TopicMetadataRequest$$anonfun$readFrom$1.apply(TopicMetadataRequest.scala:43)
at scala.collection.immutable.Range$ByOne$class.foreach(Range.scala:282)
at scala.collection.immutable.Range$$anon$2.foreach(Range.scala:265)
at
kafka.api.TopicMetadataRequest$.readFrom(TopicMetadataRequest.scala:43)
at kafka.api.RequestKeys$$anonfun$4.apply(RequestKeys.scala:37)
at kafka.api.RequestKeys$$anonfun$4.apply(RequestKeys.scala:37)
at kafka.network.RequestChannel$Request.init(RequestChannel.scala:49)
at kafka.network.Processor.read(SocketServer.scala:345)
at kafka.network.Processor.run(SocketServer.scala:245)
at java.lang.Thread.run(Thread.java:722)
My test sends a metadatarequest (api code 3) using a clientId foo,
correlationId 0 and topicName myTopic
I don't know where I'm missing something. Please find the js code bellow
and also a java counter version that I've created just for the sake of my
lack of experience with node and js. I get the same error with the java
version:
Request.prototype.toBytes = function() {
var encoded = new BufferMaker()
.Int16BE(this.apiKey)
.Int16BE(API_VERSION)
.Int32BE(this.correlationId)
.Int16BE(Buffer.byteLength(this.clientId,'UTF-8'))
.string(this.clientId)
.string(this.requestMessage.toBytes()).make();
var bytes = new
BufferMaker().Int32BE(encoded.length).string(encoded).make();
MetadataRequest.prototype.toBytes = function () {
var bytes = new
BufferMaker().Int16BE(Buffer.byteLength(this.topicName,'UTF-8')).string(this.topicName).make();
return bytes;
}
var req = new Request(3,0,foo,new MetadataRequest(myTopic));
var status = connection.write(req.toBytes());
Here's the content of the buffer:
Buffer 00 00 00 16 00 03 00 00 00 00 00 00 00 03 66 6f 6f 00 07 6d 79 54
6f 70 69 63
And the java version (using vertx netclient):
Buffer writeBuffer = new Buffer()
.appendShort((short) 3) //metadataRequest
.appendShort((short) 0) //ApiVersion
.appendInt(0) //Correlation
.appendShort((short) foo.getBytes(UTF-8).length).appendString(
foo, UTF-8) //ShortString clientId
.appendShort((short) myTopic.getBytes(UTF-8).length).appendString
(myTopic, UTF-8); //ShortString topicName
Buffer message = new Buffer();
message.appendInt(writeBuffer.length()); //add total Message size to
buffer
System.out.println(writeBuffer.length());
message.appendBuffer(writeBuffer);
socket.write(message);
--
The intuitive mind is a sacred gift and the
rational mind is a faithful servant. We have
created a society that honors the servant and
has forgotten the gift.
--
*Colin Blower*
/Software Engineer/
Barracuda Networks Inc.
+1 408-342-5576 (o)
Re: 0.8 protocol exception
Hey Vinicius, Feel free to contact me if you would like help with the protocol stuff. I'm sure you will be able to figure it out from the wiki and scala code, but it would be nice to make the wiki as detailed as possible for the next set of developers. Thanks for working on a new Kafka client. I would love to see the final product. On 07/08/2013 02:44 PM, Vinicius Carvalho wrote: Thanks Colin, yep you are right I was missing the ShortString way to represent arrays. All is fine now, I'm able to connect and execute MetadataRequest/Responses. Moving to ProducerRequest/Responses now :) On Mon, Jul 8, 2013 at 5:13 PM, Colin Blower cblo...@barracuda.com wrote: I had the exact same problem when I started writing code for the new protocol. This is an oddity with the way the protocol spec uses EBNF to specify arrays. Checkout the section on protocol primitives, especially arrays. https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProtocolPrimitiveTypes Essentially, each array is preceded by its length as an int32. With regards to your specific buffer: Buffer 00 00 00 16 00 03 00 00 00 00 00 00 00 03 66 6f 6f 00 07 6d 79 54 6f 70 69 63 You seem to be missing the clientId as well as the array length. For comparison the buffer, with the client I wrote, for clientId perl-kafka and topics foo and myTopic: 00 00 00 26 00 03 00 00 00 00 00 2a 00 0a 70 65 72 6c 2d 6b 61 66 6b 61 00 00 00 02 00 03 66 6f 6f 00 07 6d 79 54 6f 70 69 63 On 07/08/2013 04:12 AM, Vinicius Carvalho wrote: Ok, so I've found out the error: The documentation is outdated, the MetadataRequest BNF should be: NumberOfTopics [TopicList] -- *Colin Blower* /Software Engineer/ Barracuda Networks Inc. +1 408-342-5576 (o)
Re: 0.8.0 HEAD 3/4/2013 performance jump?
I vote for ack=1. It is a reasonable tradeoff between performance and reliability. On 03/05/2013 08:13 AM, Jun Rao wrote: Chris, Joe, Yes, the default ack is currently 0. Let me explain the ack mode a bit more so that we are on the same page (details are covered in my ApachCon presentation http://www.slideshare.net/junrao/kafka-replication-apachecon2013) . There are only 3 ack modes that make sense. ack=0: producer waits until the message is in the producer's socket buffer ack=1: producer waits until the message is received by the leader ack=-1: producer waits until the message is committed The tradeoffs are: ack=0: lowest latency; some data loss during broker failure ack=1: lower latency; a few data loss during broker failure ack=-1: low latency; no data loss during broker failure All cases work with replication factor 1, which is the default setting out of box. With ack=1/-1, the producer may see some error when the leader hasn't been being elected. However, the number of errors should be small since typically leaders are elected very quickly. The argument for making the default ack 0 is that (1) this is the same behavior you get in 0.7 and (2) the producer runs fastest in this mode. The argument for making the default ack 1 or -1 is that they gave you better reliability. I am not sure what's the best thing to do that here since correct setting really depends on the application. What do people feel? Thanks, Jun -- *Colin Blower* /Software Engineer/ Barracuda Networks Inc. +1 408-342-5576 (o) Copy, by Barracuda, helps you store, protect, and share all your amazing things. Start today: www.copy.com.
