Re: Kafka consumer ensure offsets committed before partition rebalance to avoid duplicates

2022-09-20 Thread Pushkar Deole
Hi Luke, Thanks for the details... so from explanation above, it seems that both of these scenarios, I won't be able to avoid duplicates processing, which is main purpose that I was looking to achieve scenario 1: consumer shuts down, and doesn't commit offsets of already polled and processed

Re: Kafka consumer ensure offsets committed before partition rebalance to avoid duplicates

2022-09-20 Thread Luke Chen
Hi 1. I was under impression, from documentation, that close method waits for 30 seconds to complete processing of any in-flight events and then commits offsets of last poll. Isn't that true? what does timeout of 30 seconds mean? -> 30 seconds timeout is to have a buffer for graceful closing,

[VOTE] 3.3.0 RC2

2022-09-20 Thread David Arthur
Hello Kafka users, developers and client-developers, This is the second release candidate for Apache Kafka 3.3.0. Many new features and bug fixes are included in this major release of Kafka. A significant number of the issues in this release are related to KRaft, which will be considered

Re: Help with log.dirs please

2022-09-20 Thread Chris Peart
Thanks Peter, I’ll give this a go tomorrow and let you know how I get on. Many Thanks, Chris > On 20 Sep 2022, at 9:32 pm, Peter Bukowinski wrote: > > Hi Chris, > > If the configs are correct and the permissions on all the /data/X/kafka/data > directories are correct, then kafka should use

Re: Help with log.dirs please

2022-09-20 Thread Peter Bukowinski
Hi Chris, If the configs are correct and the permissions on all the /data/X/kafka/data directories are correct, then kafka should use all of the log dirs when creating topics. Remember that kafka will not automatically move any existing topic data when the cluster configs change. I’d test by

Re: Help with log.dirs please

2022-09-20 Thread Chris Peart
Thinking about this, as this is not in production it might be easier just reset everything. Would it be something like: Stopping Kafka Deleting all files and folders in /Kafka/data Changing the zookeeper setting to point to a different path in the zookeeper cluster Start Kafka Some help on

Re: Help with log.dirs please

2022-09-20 Thread Chris Peart
Hi Peter, I have checked the logs on all 4 brokers and could only see /data/1/data/kafka being used, log.dirs config in the logs showed all the disks but no errors. I managed to get the other 5 disks working by adding the path /data/[2-6]/data/kafka and setting the owner as kafka and

Re: Kafka consumer ensure offsets committed before partition rebalance to avoid duplicates

2022-09-20 Thread Pushkar Deole
Thanks Luke.. 1. I was under impression, from documentation, that close method waits for 30 seconds to complete processing of any in-flight events and then commits offsets of last poll. Isn't that true? what does timeout of 30 seconds mean? 2. How does CoperativeStickyAssignor solve my problem

Re: CVE-2022-34917: Unauthenticated clients may cause OutOfMemoryError on Apache Kafka Brokers

2022-09-20 Thread Luke Chen
What a great finding! Thanks Mickael Maison, Tom Bentley and Daniel Collins! And thanks for all the release managers who help drive all these security patch releases! Luke On Mon, Sep 19, 2022 at 11:53 PM Manikumar wrote: > Severity: High > > Description: > > A security vulnerability has