[jira] [Created] (KAFKA-14977) testDescribeStateOfExistingGroupWithRoundRobinAssignor is flaky

Philip Nee created KAFKA-14977: -- Summary: testDescribeStateOfExistingGroupWithRoundRobinAssignor is flaky Key: KAFKA-14977 URL: https://issues.apache.org/jira/browse/KAFKA-14977 Project: Kafka

Re: Kafka client needs KAFKA-10337 to cover async commit use case

orkaround. > > I guess we can fix this by adding `client.pollNoWakeup()`, e.g. at the > start of `ConsumerCoordinator.commitOffsetsAsync`. Is that an acceptable > change? > > Kind regards, > Erik. > > > Op 05-05-2023 om 20:20 schreef Philip Nee: > > Hey Eric, &g

Re: Kafka client needs KAFKA-10337 to cover async commit use case

vious async > >> commits > >> to complete if it tries to commit an empty offset map. > > > > Indeed! I am assuming the same is true for commitAsync. The important > > thing is that we need something to get those callbacks. I would prefer > > commitAsync but i

[jira] [Resolved] (KAFKA-13668) Failed cluster authorization should not be fatal for producer

[ https://issues.apache.org/jira/browse/KAFKA-13668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Nee resolved KAFKA-13668. Fix Version/s: 3.6.0 Resolution: Fixed > Failed cluster authorization should not be fa

[jira] [Resolved] (KAFKA-14016) Revoke more partitions than expected in Cooperative rebalance

[ https://issues.apache.org/jira/browse/KAFKA-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Nee resolved KAFKA-14016. Fix Version/s: 3.5.0 3.4.1 Assignee: Philip Nee Resolution

[jira] [Resolved] (KAFKA-13891) sync group failed with rebalanceInProgress error cause rebalance many rounds in coopeartive

[ https://issues.apache.org/jira/browse/KAFKA-13891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Nee resolved KAFKA-13891. Fix Version/s: 3.5.0 3.4.1 (was: 3.6.0

[jira] [Created] (KAFKA-14960) Metadata Request Manager and listTopics/partitionsFor API

Philip Nee created KAFKA-14960: -- Summary: Metadata Request Manager and listTopics/partitionsFor API Key: KAFKA-14960 URL: https://issues.apache.org/jira/browse/KAFKA-14960 Project: Kafka Issue

Re: Kafka client needs KAFKA-10337 to cover async commit use case

Sorry - I dug a bit into the old PR. Seems like the issue is there's broken contract that if the commitSync won't wait for the previous async commits to complete if it tries to commit an empty offset map. On Tue, May 2, 2023 at 12:49 PM Philip Nee wrote: > Hey Erik, > > Just

Re: Kafka client needs KAFKA-10337 to cover async commit use case

Hey Erik, Just a couple of questions to you: Firstly, could you explain the situation in that you would prefer to invoke commitAsync over commitSync in the rebalance listener? Typically we would use the synchronized method to ensure the commits are completed before moving on with the

Re: [DISCUSS] Recommendations for managing long-running projects

Hi Kirk and David, Thanks for raising this, Kirk. David, to your questions, as I'm also contributing to the project: Do you expect the changes to come in incrementally, or as a few large patches? We try to limit the PR size as much as possible, but sometimes it is inevitably large... We could

Re: [DISCUSS] Adding non-committers as Github collaborators

Hi all, opinion from a regular person here: I think being able to re-trigger jenkin's test and tagging can be very helpful for a lot of times. On Fri, Apr 28, 2023 at 9:31 AM John Roesler wrote: > Hi all, > > This is a great suggestion! It seems like a really good way to make the > Apache Kafka

[jira] [Created] (KAFKA-14950) Implement assign() and assignment()

Philip Nee created KAFKA-14950: -- Summary: Implement assign() and assignment() Key: KAFKA-14950 URL: https://issues.apache.org/jira/browse/KAFKA-14950 Project: Kafka Issue Type: Sub-task

Re: [VOTE] KIP-909: DNS Resolution Fallures Should Not Fail the Client

> > +1. Thanks for the design. Looking forward to the implementation. > > > > On Tue, Apr 25, 2023 at 10:49 AM Jason Gustafson > > wrote: > > > > > > +1 Thanks Philip! > > > > > > > > > On Thu, Apr 13, 2023 at 7:49 AM Kirk True wrote:

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

Hey Jason, Thanks again. I've updated the KIP. P On Fri, Apr 21, 2023 at 9:56 AM Jason Gustafson wrote: > Hey Philip, that sounds good to me. > > -Jason > > > > On Thu, Apr 20, 2023 at 1:20 PM Philip Nee wrote: > > > Hey Jason, > > > > Thanks for t

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

ctions fail. I know I'm the > one that suggested generalizing the configuration, but I feel some > hesitation after thinking about it more. > > Thanks, > Jason > > On Wed, Apr 19, 2023 at 8:10 PM Philip Nee wrote: > > > Hey Jason, > > > > Thanks for your revie

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

Hey Colin, I still need to finish reading and understanding the KIP, but I have a couple of comments despite being ignorant of most of the KRaft stuff. (Sorry!) Firstly, does it make sense to create an extension of the current AdminClient only to handle these specific KRaft use cases? It seems

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

rying to reach the > bootstrap servers perhaps? That would be closer to behavior today which > only treats DNS resolution failures as fatal. What do you think? > > Best, > Jason > > On Mon, Apr 10, 2023 at 1:53 PM Philip Nee wrote: > > > Thanks, everyone: I'm starting a v

[VOTE] KIP-909: DNS Resolution Fallures Should Not Fail the Client

Hey everyone! I'm starting a vote for KIP-909: DNS Resolution Fallures Should Not Fail the Client Please refer to the discussion thread here: https://lists.apache.org/thread/st84zzwnq5m3pkzd1r7jk9lmqdt9m98s Thanks! P

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

L606 > [2] - > > https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L439 > > Cheers, > > Chris > > On Fri, Feb 24, 2023 at 4:59 PM Philip Nee wrote: > > > Hey Chris, > > >

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

or so. Link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-909%3A+DNS+Resolution+Failure+Should+Not+Fail+the+Clients Thanks, P On Thu, Mar 23, 2023 at 12:59 PM Philip Nee wrote: > Hey Kirk, > > Sorry about omitting your response; it slipped through the cracks... > >

[jira] [Created] (KAFKA-14875) Implement Wakeup()

Philip Nee created KAFKA-14875: -- Summary: Implement Wakeup() Key: KAFKA-14875 URL: https://issues.apache.org/jira/browse/KAFKA-14875 Project: Kafka Issue Type: Task Components

Re: Should the mock consumer call the consumer rebalance listener on rebalance?

Awesome! On Thu, Mar 23, 2023 at 12:31 PM Dan S wrote: > I'll pick it up, thanks! > > On Thu, Mar 23, 2023, 19:27 Philip Nee wrote: > > > Hey Dan, > > > > Your analysis looks right. I do see TODO item there to implement the > > rebalance callback. Would you

Re: Should the mock consumer call the consumer rebalance listener on rebalance?

and then adding them > back. We're noticing our custom listener is never called, which was > unexpected, but based on the code analysis in the original email seems to > be the current implementation. The question is whether this is indeed > desired behavior. > > On Thu, Mar 23, 2023,

Re: Should the mock consumer call the consumer rebalance listener on rebalance?

Hey Dan, Thanks for looking into this. Are you talking about MockConsumer? If you invoke subscribe(Collection topics), it actually registers a Noop callback. Perhaps this is what you are seeing? P On Thu, Mar 23, 2023 at 11:11 AM Dan S wrote: > Hello all, > > It seems to me based on reading

[jira] [Resolved] (KAFKA-14754) improve consumer under example package

[ https://issues.apache.org/jira/browse/KAFKA-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Nee resolved KAFKA-14754. Resolution: Fixed > improve consumer under example pack

[jira] [Resolved] (KAFKA-14753) improve producer under example package

[ https://issues.apache.org/jira/browse/KAFKA-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Nee resolved KAFKA-14753. Resolution: Fixed > improve producer under example pack

[jira] [Reopened] (KAFKA-14752) improve kafka examples under examples package

[ https://issues.apache.org/jira/browse/KAFKA-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Nee reopened KAFKA-14752: > improve kafka examples under examples pack

[jira] [Resolved] (KAFKA-14752) improve kafka examples under examples package

[ https://issues.apache.org/jira/browse/KAFKA-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Nee resolved KAFKA-14752. Resolution: Fixed > improve kafka examples under examples pack

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

lures will be helpful in any case. > > -Jason > > > > > > > On Tue, Feb 28, 2023 at 11:47 AM Philip Nee wrote: > > > Jason: > > Thanks for the feedback. Now giving it a second thought, I think your > > suggestion of logging the error might make sense, as I cou

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

t; >> > >> I was toying with the idea of a NetworkException subclass too. It's a > >> simpler API, but it doesn't allow for preflight validation, which can be > >> useful in scenarios where submitting new configurations for client > >> applications is expensive

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

https://github.com/apache/kafka/blob/5f9d01668cae64b2cacd7872d82964fa78862aaf/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L606 > [2] - > > https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Ab

[jira] [Created] (KAFKA-14761) Integration Tests for the New Consumer Implementation

Philip Nee created KAFKA-14761: -- Summary: Integration Tests for the New Consumer Implementation Key: KAFKA-14761 URL: https://issues.apache.org/jira/browse/KAFKA-14761 Project: Kafka Issue Type

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

-facing behavior. It also seems like > that detail might impact how intertwined this and KIP-899 are, though the > similarity could still be superficial either way. > > Cheers, > > Chris > > On Thu, Feb 23, 2023 at 9:21 PM Philip Nee wrote: > > > Hey Ismael, >

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

PM Ismael Juma wrote: > Thanks for the KIP. Not sure if I missed it, but how long will we retry for > and when do we give up and propagate the failure to the user? > > Ismael > > On Thu, Feb 23, 2023 at 9:30 AM Philip Nee wrote: > > > Hi all! > > > > I want

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

olution to poll, then KIP-899 becomes just a question > of whether we should call poll at other points after the first resolution. > It seems like these could potentially be merged into one proposal, or you > and Ivan could coordinate on symbiotic KIPs. > > Thanks again, > -John > &

[DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

Hi all! I want to start a discussion thread about how we can handle client bootstrap failure due DNS lookup. This requires a bit of behavioral change, so a KIP is proposed and attached to this email. Let me know what you think! *A small remark here*: *As the title of this KIP might sound

Re: Dead letter management

Hey Viram, AFAIK, I don't think Kafka supports retries on dead messages. Your best bet would be to handle these cases by your application. Here's a useful article, in case you haven't read it:

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

Hey Ivan, Thanks for the KIP. Some questions for clarification: It seems like the main problem is that if we don't poll frequently enough, the cluster topology can change entirely before the metadata is refreshed and thus causing staled clients. My question is: if you don't use the client for a

Re: Pull request regarding to consumer metrics cleanup issue after shutdown

Thanks Feiyan! Will spend some time reviewing the PR :) On Tue, Jan 17, 2023 at 7:08 PM Feiyan Yu wrote: > Howdy! > > I found that there is no logic to remove metrics registered in > `ConsumerCoordinatorMetrics` when `ConsumerCoordinator` is closed, which > might > > I created an issue related

Reviews for KAFKA-13668

Hey committers, Trying to get some attention to the KAFKA-13668. Is anyone familiar with the producer code able to provide feedback for the issue KAFKA-13668 ? In a nutshell, it allows producers to retry upon receiving authorization errors w/o needing

Re: [ANNOUNCE] New committer: Justine Olshan

wow congrats! On Thu, Dec 29, 2022 at 1:05 PM Chris Egerton wrote: > Congrats, Justine! > > On Thu, Dec 29, 2022, 15:58 David Jacot wrote: > > > Hi all, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer > > Justine > > Olshan. > > > > Justine has been contributing to

[jira] [Created] (KAFKA-14468) Refactor Commit Logic

Philip Nee created KAFKA-14468: -- Summary: Refactor Commit Logic Key: KAFKA-14468 URL: https://issues.apache.org/jira/browse/KAFKA-14468 Project: Kafka Issue Type: Sub-task Components

[jira] [Created] (KAFKA-14438) Stop supporting empty consumer groupId

Philip Nee created KAFKA-14438: -- Summary: Stop supporting empty consumer groupId Key: KAFKA-14438 URL: https://issues.apache.org/jira/browse/KAFKA-14438 Project: Kafka Issue Type: Task

Re: Regarding an issue with message distributions in kafka.

Hello Puneet, Why do you need multiple consumers from the same group consuming from the same partition? I am trying to think of a compiling reason for such a use case... It makes more sense to me to have a thread pool to process the records fetched from a single consumer. This is a great

Re: Regarding Producer Metric - "request-latency-avg"

Hey Amrit, Actually, I think the confluent guide is correct because it seems like the request is created here . and the queue time

Re: Regarding Producer Metric - "request-latency-avg"

Hey Amrit, Could you elaborate on "this is not updated in producer metrics documentation in confluent"? The confluent documentation states, "The average request latency in ms", which seems right to

[jira] [Resolved] (KAFKA-14252) Create background thread skeleton

[ https://issues.apache.org/jira/browse/KAFKA-14252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Nee resolved KAFKA-14252. Resolution: Fixed > Create background thread skele

[jira] [Created] (KAFKA-14274) Implement fetching logic

Philip Nee created KAFKA-14274: -- Summary: Implement fetching logic Key: KAFKA-14274 URL: https://issues.apache.org/jira/browse/KAFKA-14274 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-14264) Create a coordinator class for the new implementation

Philip Nee created KAFKA-14264: -- Summary: Create a coordinator class for the new implementation Key: KAFKA-14264 URL: https://issues.apache.org/jira/browse/KAFKA-14264 Project: Kafka Issue Type

[jira] [Created] (KAFKA-14252) Create background thread skeleton

Philip Nee created KAFKA-14252: -- Summary: Create background thread skeleton Key: KAFKA-14252 URL: https://issues.apache.org/jira/browse/KAFKA-14252 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-14247) Implement EventHandler interface and DefaultEventHandler

Philip Nee created KAFKA-14247: -- Summary: Implement EventHandler interface and DefaultEventHandler Key: KAFKA-14247 URL: https://issues.apache.org/jira/browse/KAFKA-14247 Project: Kafka Issue

[jira] [Created] (KAFKA-14246) Refactor KafkaConsumer Threading Model

Philip Nee created KAFKA-14246: -- Summary: Refactor KafkaConsumer Threading Model Key: KAFKA-14246 URL: https://issues.apache.org/jira/browse/KAFKA-14246 Project: Kafka Issue Type: Improvement

Re: KafkaConsumer refactor proposal

he state machine, and here are the following scenario: > c. Poll the networkClient. > d. Backoff for retryBackoffMs milliseconds. > > Thank you. > Again, really grateful for you to review the proposal, so thank you! P > Luke > > On Sat, Sep 17, 2022 at 6:29 AM

Re: KafkaConsumer refactor proposal

On Fri, Sep 16, 2022 at 3:01 PM Guozhang Wang wrote: > Hi Philip, > > On Fri, Sep 16, 2022 at 1:20 PM Philip Nee wrote: > > > Hi Guozhang, > > > > Thank you for the reviews, and I agree with your suggestions: If we find > > any behavior changes, we will

Re: KafkaConsumer refactor proposal

. > > 2. I'd suggest for new modules like `ConsumerBackgroundThread`, we first > defines an interface in the `internals` package, e.g. `RequestEventHandler` > (assuming the previous rename suggestion), and then have a > `DefaultRequestEventHandler` implementation class which encapsulate the >

Re: KafkaConsumer refactor proposal

lip! > > Thanks for the write-up. > > On Tue, Sep 13, 2022, at 2:13 PM, Philip Nee wrote: > > Hi all, > > > > Here is the proposal to refactor the Kafka Consumer > > < > https://cwiki.apache.org/confluence/display/KAFKA/Proposal%3A+Consumer+Threading+Model+Ref

KafkaConsumer refactor proposal

Hi all, Here is the proposal to refactor the Kafka Consumer . The 1-pager is on the wiki, so please take a look at it. Also, this is a prerequisite for KIP-848 (the next gen rebalance protocol).

[jira] [Created] (KAFKA-14196) Flaky OffsetValidationTest seems to indicate potential duplication issue during rebalance

Philip Nee created KAFKA-14196: -- Summary: Flaky OffsetValidationTest seems to indicate potential duplication issue during rebalance Key: KAFKA-14196 URL: https://issues.apache.org/jira/browse/KAFKA-14196

Requesting permission to the apache kafka project

Hello, Would like to gain permission to the AK for contribution. wikiID: pnee jiraID: pnee Thanks, P

<    1   2