Re: Frequent consumer offset commit failures

2020-06-24 Thread James Olsen
Ricardo, Thanks for your response. I've been looking over your advice and examining the Kafka client code and our logs. I'm still thinking that the issue is simply that the Broker can't keep up with the offset commit rate. The reasons for this are: * The first message in any sequence is

Re:Re: [ANNOUNCE] New committer: Xi Hu

2020-06-24 Thread Roc Marshal
Congratulations ! Xi Hu. Best, Roc Marshal. At 2020-06-25 01:30:33, "Boyang Chen" wrote: >Congratulations Xi! Well deserved. > >On Wed, Jun 24, 2020 at 10:10 AM AJ Chen wrote: > >> Congratulations, Xi. >> -aj >> >> >> >> On Wed, Jun 24, 2020 at 9:27 AM Guozhang Wang wrote: >> >>

Re: _consumer_offsets topic produce rate is dam high

2020-06-24 Thread Liam Clarke-Hutchinson
...the classpath is required to make the formatter available to the console producer consumer... On Thu, Jun 25, 2020 at 9:44 AM Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > To investigate, you can read the messages on the topic to figure out how > consumer groups are committing.

Re: _consumer_offsets topic produce rate is dam high

2020-06-24 Thread Liam Clarke-Hutchinson
To investigate, you can read the messages on the topic to figure out how consumer groups are committing. They're stored in a binary format, but you can make them human readable using a formatter. Here's what I did last time to do this - the classpath is required to make the formatter available to

Re: [ANNOUNCE] New committer: Xi Hu

2020-06-24 Thread Boyang Chen
Congratulations Xi! Well deserved. On Wed, Jun 24, 2020 at 10:10 AM AJ Chen wrote: > Congratulations, Xi. > -aj > > > > On Wed, Jun 24, 2020 at 9:27 AM Guozhang Wang wrote: > > > The PMC for Apache Kafka has invited Xi Hu as a committer and we are > > pleased to announce that he has accepted!

Re: [ANNOUNCE] New committer: Xi Hu

2020-06-24 Thread AJ Chen
Congratulations, Xi. -aj On Wed, Jun 24, 2020 at 9:27 AM Guozhang Wang wrote: > The PMC for Apache Kafka has invited Xi Hu as a committer and we are > pleased to announce that he has accepted! > > Xi Hu has been actively contributing to Kafka since 2016, and is well > recognized especially

Re: [ANNOUNCE] New committer: Xi Hu

2020-06-24 Thread Tom Bentley
Congratulations Xi! On Wed, Jun 24, 2020 at 5:34 PM Guozhang Wang wrote: > The PMC for Apache Kafka has invited Xi Hu as a committer and we are > pleased to announce that he has accepted! > > Xi Hu has been actively contributing to Kafka since 2016, and is well > recognized especially for his

[ANNOUNCE] New committer: Xi Hu

2020-06-24 Thread Guozhang Wang
The PMC for Apache Kafka has invited Xi Hu as a committer and we are pleased to announce that he has accepted! Xi Hu has been actively contributing to Kafka since 2016, and is well recognized especially for his non-code contributions: he maintains a tech blog post evangelizing Kafka in the

Re: _consumer_offsets topic produce rate is dam high

2020-06-24 Thread Karolis Pocius
Check if any of your consumers have auto commit turned off and instead commit after processing each message. Also, even if all consumers are using auto commit, maybe some of them have the interval set to something crazy low like 1 ms. On Sat, Jun 20, 2020 at 8:31 PM Ashutosh singh wrote: > Hi

Re: Kafka still writable when only one process available

2020-06-24 Thread Can Zhang
Hi Liam, Thanks for your quick reply, and great detailed explanation! When starting console producer with "--request-required-acks all", I do see "Messages are rejected since there are fewer in-sync replicas than required" in log. Best, Can Zhang On Wed, Jun 24, 2020 at 11:33 AM Liam