Re: KafkaConsumer refactor proposal

Hi Erik, Regarding the consumer refactor project, we’re in the process of converting Philip’s design to a “proper” KIP here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-945%3A+Update+threading+model+for+Consumer It’s still very much a draft and not ready for a formal DISCUSS

Re: KafkaConsumer refactor proposal

Hi Erik, Thanks for your interest in the client world :) I agree that the current implementation of how we handle callbacks is problematic in that it introduces race conditions and/or bottlenecks. I don’t have as much experience with the callbacks from an application developer standpoint. Is

Re: KafkaConsumer refactor proposal

Hi Philip, I have been scanning through https://cwiki.apache.org/confluence/display/KAFKA/Consumer+threading+refactor+design and KIP-848 and from this I understand that the kafka consumer API will not change. Perhaps the refactoring and/or KIP-848 is a good opportunity to improve the API

Re: KafkaConsumer refactor proposal

Hey Divij, Sorry for the late reply - I was away for some extended period of time, but I'm back to address some of the questions here. To your point, we will incorporate your suggestions to the 1-pager, and well as the future KIP-945 (we decided to create a kip for this work for better

Re: KafkaConsumer refactor proposal

Thank you Philip for the response. I will post more comments about this in the upcoming days but meanwhile here are my initial thoughts. Please let me know what you think about them. Addition to terminology Let's divide the API calls from the consumer into two categories. System API calls and

Re: KafkaConsumer refactor proposal

Hi all! I've added the "KIP-848" and "ctr" tags to the relevant PRs. see: https://github.com/apache/kafka/pulls?q=label%3Actr+sort%3Acreated-asc On Tue, May 9, 2023 at 10:29 AM Philip Nee wrote: > Hey Divij, > > Firstly, THANK YOU for reaching out and providing the feedback. At this >

Re: KafkaConsumer refactor proposal

Hey Divij, Firstly, THANK YOU for reaching out and providing the feedback. At this moment, I don't have all of the answers to your questions at the moment, but I hope my responses could be helpful in moving the project forward. 1. There isn't a prioritization strategy, and the network request

Re: KafkaConsumer refactor proposal

Hey Philip Thank you for writing this down. Overall, the ideal of decoupling the responsibility amongst threads and communication using queues looks good. I have a few questions associated with handling failure scenarios. 1. I observed that there is no concept of prioritization in the

Re: KafkaConsumer refactor proposal

On Sun, Sep 18, 2022 at 6:03 AM Luke Chen wrote: > Hi Philip, > > Thanks for the write-up. > Also thank you for taking the time to read the proposal. Very grateful. > Some questions: > > 1. You said: If we don't need a coordinator, the background thread will > stay in the *initialized* state.

Re: KafkaConsumer refactor proposal

Hi Philip, Thanks for the write-up. Some questions: 1. You said: If we don't need a coordinator, the background thread will stay in the *initialized* state. But in the definition of *initialized, *it said: *initialized*: The background thread completes initialization, and the loop has started.

Re: KafkaConsumer refactor proposal

On Fri, Sep 16, 2022 at 3:01 PM Guozhang Wang wrote: > Hi Philip, > > On Fri, Sep 16, 2022 at 1:20 PM Philip Nee wrote: > > > Hi Guozhang, > > > > Thank you for the reviews, and I agree with your suggestions: If we find > > any behavior changes, we will issue a KIP. Is it possible to first >

Re: KafkaConsumer refactor proposal

Hi Philip, On Fri, Sep 16, 2022 at 1:20 PM Philip Nee wrote: > Hi Guozhang, > > Thank you for the reviews, and I agree with your suggestions: If we find > any behavior changes, we will issue a KIP. Is it possible to first publish > the new implementation in a separate class so that we don't

Re: KafkaConsumer refactor proposal

Hi Guozhang, Thank you for the reviews, and I agree with your suggestions: If we find any behavior changes, we will issue a KIP. Is it possible to first publish the new implementation in a separate class so that we don't disrupt the existing implementation? Then we can issue a KIP to address the

Re: KafkaConsumer refactor proposal

Thank you, Kirk! Your feedback is very much appreciated; I'd like to try to address them in the email and the document. Here are my responses: 1. I removed that point because I think we should still try to maintain the current behavior to avoid regressions 2. I think I misworded it. I think

Re: KafkaConsumer refactor proposal

Hello Philip, Thanks for writing down the 1-pager. Just to clarify, the reason we wrote this as a 1-pager instead of a KIP is that so far all the implementations should be internal, and hence not impacting anything on the public APIs. If, e.g. we found along with the implementation that some

Re: KafkaConsumer refactor proposal

Hi Philip! Thanks for the write-up. On Tue, Sep 13, 2022, at 2:13 PM, Philip Nee wrote: > Hi all, > > Here is the proposal to refactor the Kafka Consumer > . > The 1-pager is on the wiki, so please

KafkaConsumer refactor proposal

Hi all, Here is the proposal to refactor the Kafka Consumer . The 1-pager is on the wiki, so please take a look at it. Also, this is a prerequisite for KIP-848 (the next gen rebalance protocol).