Re versions: fair enough, I think it's okay to keep it as strings. Just to
clarify the concern I had is that if we do want to augment it in the
future, it will be harder to change a `string` field into a `struct` :P
Re onAssignment: actually even with the current proposal, since we are
giving the
Regarding the version, I would rather add this later when we have a
clear use case for it. It seems to me that we are speculating here. I
understand your point but I am not fully convinced by the solution at
the moment. Would you agree with this?
Regarding the onAssignment, I was thinking about
Regarding the version, what I was thinking is that in the HB request, for
"serverAssignor" field, instead of just having it as a single string field,
maybe we could consider also making it a structure that includes: name,
minimumVersion, maximumVersion. Where the minimumVersion/maximumVersion
Hi Guozhang,
Regarding the version, my understanding is that the version would be
either the client software version or the request version, is this
correct? If so, we could indeed pass this information down to the
assignor via the interface. One way would be to pass a "server
context" to the
Hello David,
On Fri, Sep 23, 2022 at 2:00 AM David Jacot
wrote:
> Hey,
>
> > Just to clarify I was asking about the `version` of the assignor (i.e. up
> to what version that the client would support), and I do agree we would not
> need metadata. What I have in mind is that, for some specific
Hey,
> Just to clarify I was asking about the `version` of the assignor (i.e. up
to what version that the client would support), and I do agree we would not
need metadata. What I have in mind is that, for some specific built-in
broker-assignors, e.g. rack-aware assignors, if it's possible that in
P-795. We can table this discussion for after
> > the Connect discussion, as there will be more clarity on how extending the
> > new protocol will look like.
> >
> > From: dev@kafka.apache.org At: 09/12/22 07:58:32 UTC-4:00To:
> > dev@kafka.apache.org
> > Subject
cussion, as there will be more clarity on how extending the
> new protocol will look like.
>
> From: dev@kafka.apache.org At: 09/12/22 07:58:32 UTC-4:00To:
> dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-848: The Next Generation of the Consumer
> Rebalance Protocol
>
> Hi
: Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance
Protocol
Hi Hector,
We definitely need to share internals with Connect APIs. That model
would not scale otherwise.
Regarding the schema registry, I wonder if we could just use the new
protocol. At the end of the day
Hi Guozhang,
1. I have added a reference to the relevant chapter instead of
repeating the whole thing. Does that work for you?
2. The "Rebalance Triggers" section you are referring to is about when
a rebalance should be triggered for the non-upgraded members using the
old protocol. The section
Schema Registry, and so on).
>
>
> From: dev@kafka.apache.org At: 08/12/22 09:31:36 UTC-4:00To:
> dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance
> Protocol
>
> Thank you Guozhang/David for the feedback. Looks like there's ag
Hello David,
Alright I think that's sufficient. Just to make that clear in the doc,
could we update:
1) the heartbeat request handling section, stating when coordinator will
trigger rebalance based on the HB's member metadata / reason?
2) the "Rebalance Triggers" section to include what we
in the case of Schema Registry, and so on).
From: dev@kafka.apache.org At: 08/12/22 09:31:36 UTC-4:00To:
dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance
Protocol
Thank you Guozhang/David for the feedback. Looks like there's agreement on
using separate
Hi Guozhang,
I thought that the assignor will always be consulted when the next
heartbeat request is constructed. In other words,
`PartitionAssignor#metadata` will be called for every heartbeat. This
gives the opportunity for the assignor to enforce a rebalance by
setting the reason to a non-zero
Hello David,
One of Jun's comments make me thinking:
```
In this case, a new assignment is triggered by the client side
assignor. When constructing the HB, the consumer will always consult
the client side assignor and propagate the information to the group
coordinator. In other words, we don't
Hi Jun,
I have updated the KIP to include your feedback. I have also tried to
clarify the parts which were not cleared.
Best,
David
On Fri, Sep 2, 2022 at 4:18 PM David Jacot wrote:
>
> Hi Jun,
>
> Thanks for your feedback. Let me start by answering your questions
> inline and I will update
Hi Jun,
Thanks for your feedback. Let me start by answering your questions
inline and I will update the KIP next week.
> Thanks for the KIP. Overall, the main benefits of the KIP seem to be fewer
> RPCs during rebalance and more efficient support of wildcard. A few
> comments below.
I would
Hi, David,
Thanks for the KIP. Overall, the main benefits of the KIP seem to be fewer
RPCs during rebalance and more efficient support of wildcard. A few
comments below.
30. ConsumerGroupHeartbeatRequest
30.1 ServerAssignor is a singleton. Do we plan to support rolling changing
of the partition
Hi all,
The KIP states that we will re-implement the coordinator in Java. I
discussed this offline with a few folks and folks are concerned that
we could introduce many regressions in the old protocol if we do so.
Therefore, I am going to remove this statement from the KIP. It is an
Hi Luke,
> 1.1. I think the state machine are: "Empty, assigning, reconciling, stable,
> dead" mentioned in Consumer Group States section, right?
This sentence does not refer to those group states but rather to a
state machine replication (SMR). This refers to the entire state of
group
Hi All,
As per the suggestions from David/Guozhang above, I updated the page for
Connect to have it's own set of APIs and not extend the ones from consumer.
Plz review again.
@Luke,
Thank you. I am actually looking for review comments/thoughts/suggestions
as Connect changes are still in draft
Hi David,
Thanks for the update.
Some more questions:
1. In Group Coordinator section, you mentioned:
> The new group coordinator will have a state machine per
*__consumer_offsets* partitions, where each state machine is modelled as an
event loop. Those state machines will be executed in
Hi Sagar,
I have some thoughts about Kafka Connect integrating with KIP-848, but I
think we should have a separate discussion thread for the Kafka Connect
KIP: Integrating Kafka Connect With New Consumer Rebalance Protocol [1],
and let this discussion thread focus on consumer rebalance protocol,
Thank you Guozhang/David for the feedback. Looks like there's agreement on
using separate APIs for Connect. I would revisit the doc and see what
changes are to be made.
Thanks!
Sagar.
On Tue, Aug 9, 2022 at 7:11 PM David Jacot
wrote:
> Hi Sagar,
>
> Thanks for the feedback and the document.
Hi all,
Thanks again for all the very good feedback so far. I have addressed,
I think, all the outstanding points. Here are the main updates:
* I have introduced STALE_MEMBER_EPOCH to differentiate it from the
fatal FENCED_MEMBER_EPOCH error.
* I have introduced topic ids in the
Hello Greg,
Thanks for reviewing the KIP! I hope David has addressed your questions
above, and I'd like to just emphasize one thing regarding your question 1):
one principle of the reconciliation is that, for a single client, before it
has completely revoked all the partitions that it should
Thanks Sagar, I made a quick read on your doc, and am also leaning towards
having a separate API for connect. Would also love to hear @Randall Hauch
@Konstantin 's opinion about that.
One thing is that for Kafka Connect, their compatibility story, especially
downgrading is more flexible than
Hi Gregory,
Thanks for your feedback. Please find my answers below.
> 1. The 'Rejected Alternatives' section describes how member epoch should
> advance in step with the group epoch and assignment epoch values. I think
> that this is a good idea for the reasons described in the KIP. When the
>
Hi Luke,
Thanks for your comments. Please find my answers below.
> 1. Will the member reconciling jump to the latest target assignment? For
> example, member epoch are A:1, B:2, but now the group is in target
> assignment epoch 4. Will the members jump to the target assignment result
> in epoch
Hi Sagar,
Thanks for the feedback and the document. That's really helpful. I
will take a look at it.
Overall, it seems to me that both Connect and the Consumer could share
the same underlying "engine". The main difference is that the Consumer
assigns topic-partitions to members whereas Connect
Hi all,
I am back from vacation. I will go through and address your comments
in the coming days. Thanks for your feedback.
Cheers,
David
On Wed, Aug 3, 2022 at 10:05 PM Gregory Harris wrote:
>
> Hey All!
>
> Thanks for the KIP, it's wonderful to see cooperative rebalancing making it
> down the
Hey All!
Thanks for the KIP, it's wonderful to see cooperative rebalancing making it
down the stack!
I had a few questions:
1. The 'Rejected Alternatives' section describes how member epoch should
advance in step with the group epoch and assignment epoch values. I think
that this is a good idea
Hi Guozhang/David,
I created a confluence page to discuss how Connect would need to change
based on the new rebalance protocol. Here's the page:
https://cwiki.apache.org/confluence/display/KAFKA/%5BDRAFT%5DIntegrating+Kafka+Connect+With+New+Consumer+Rebalance+Protocol
It's also pretty longish
Hello Guozhang,
Thank you so much for the doc on Kafka Streams. Sure, I would do the
analysis and come up with such a document.
Thanks!
Sagar.
On Tue, Jul 26, 2022 at 4:47 AM Guozhang Wang wrote:
> Hello Sagar,
>
> It would be great if you could come back with some analysis on how to
>
Hello Luke,
Thanks for the detailed feedback! I tried to incorporate / answer some of
them inline below.
On Thu, Jul 21, 2022 at 6:39 AM Luke Chen wrote:
> Hi David,
>
> So excited to see the consumer protocol will be renewed!
> Thanks for David, Guozhang, and Jason for creating this great
Hello Sagar,
It would be great if you could come back with some analysis on how to
implement the Connect side integration with the new protocol; so far
besides leveraging on the new "protocol type" we did not yet think through
the Connect side implementations. For Streams here's a draft of
Hi David,
So excited to see the consumer protocol will be renewed!
Thanks for David, Guozhang, and Jason for creating this great proposal!
Some comments about the protocol:
1. Will the member reconciling jump to the latest target assignment? For
example, member epoch are A:1, B:2, but now the
Hi David,
Thank you for your response. The reason I thought connect can also fit into
this new scheme is that even today the connect uses a WorkerCoordinator
extending from AbstractCoordinator to empower rebalances of
tasks/connectors. The WorkerCoordinator sets the protocolType() to connect
and
I'll be away from July 18th to August 8th with limited access to my
emails so I will address new comments and questions when I come back.
Cheers,
David
On Fri, Jul 15, 2022 at 2:16 PM David Jacot wrote:
>
> Hi Sagar,
>
> Thanks for your comments.
>
> 1) Yes. That refers to `Assignment#error`.
Hi Sagar,
Thanks for your comments.
1) Yes. That refers to `Assignment#error`. Sure, I can mention it.
2) The idea is to transition C from his current assignment to his
target assignment when he can move to epoch 3. When that happens, the
member assignment is updated and persisted with all its
Thanks, Ismael.
> 1. Regarding java.util.regex.Pattern vs com.google.re2j.Pattern, we should
> document the differences in more detail before deciding one way or another.
> That said, if people pass java.util.regex.Pattern, they expect their
> semantics to be honored. If we are doing something
Hi David,
Thanks for the KIP. I just had minor observations:
1) In the Assignment Error section in Client Side mode Assignment process,
you mentioned => `In this case, the client side assignor can return an
error to the group coordinator`. In this case are you referring to the
Assignor returning
Thanks Hector! Our goal is to move forward with specialized API
instead of relying on one generic API. For Connect, we can apply the
exact same pattern and reuse/share the core implementation on the
server side. For the schema registry, I think that we should consider
having a tailored API to do
Three quick comments:
1. Regarding java.util.regex.Pattern vs com.google.re2j.Pattern, we should
document the differences in more detail before deciding one way or another.
That said, if people pass java.util.regex.Pattern, they expect their
semantics to be honored. If we are doing something
pache.org At: 07/06/22 04:44:59 UTC-4:00To:
> dev@kafka.apache.org
> Subject: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance
> Protocol
>
> Hi all,
>
> I would like to start a discussion thread on KIP-848: The Next
> Generation of the Consumer Rebalance P
rg
Subject: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance
Protocol
Hi all,
I would like to start a discussion thread on KIP-848: The Next
Generation of the Consumer Rebalance Protocol. With this KIP, we aim
to make the rebalance protocol (for consumers) more reliable, more
scalab
Thanks Guozhang. My answers are below:
> 1) the migration path, especially the last step when clients flip the flag
> to enable the new protocol, in which we would have a window where both new
> protocols / rpcs and old protocols / rpcs are used by members of the same
> group. How the coordinator
Thanks David! I think on the high level there are two meta points we need
to concretize a bit more:
1) the migration path, especially the last step when clients flip the flag
to enable the new protocol, in which we would have a window where both new
protocols / rpcs and old protocols / rpcs are
Hi Ismael,
Thanks for your feedback. Let me answer your questions inline.
> 1. I think it's premature to talk about target versions for deprecation and
> removal of the existing group protocol. Unlike KRaft, this affects a core
> client protocol and hence deprecation/removal will be heavily
Hi Justine,
Thanks for your comments. Please find my answers below.
- Yes, the new protocol relies on topic IDs with the exception of the
topic names based in the ConsumerGroupHeartbeatRequest. I am not sure
if using topic names is the right call here. I need to think about it
a little more.
Hi Ismael,
Thanks for the feedback. Here are some replies inlined below:
On Sat, Jul 9, 2022 at 2:53 AM Ismael Juma wrote:
> Thanks for the KIP. This has the potential to be a great improvement. A few
> initial questions/comments:
>
> 1. I think it's premature to talk about target versions for
Hi Justine,
Thanks for sharing your feedback! Here are some thoughts inlined below:
On Fri, Jul 8, 2022 at 11:33 AM Justine Olshan
wrote:
> Hi David,
> Thanks for sharing this KIP! Really exciting to hear how we are changing
> the protocol! The motivation section really made me realize how
Thanks for the KIP. This has the potential to be a great improvement. A few
initial questions/comments:
1. I think it's premature to talk about target versions for deprecation and
removal of the existing group protocol. Unlike KRaft, this affects a core
client protocol and hence
Hi David,
Thanks for sharing this KIP! Really exciting to hear how we are changing
the protocol! The motivation section really made me realize how useful this
change will be.
I've done a first pass of the KIP, and may have more questions, but thought
I'd start with a few I thought of already.
Hi all,
I would like to start a discussion thread on KIP-848: The Next
Generation of the Consumer Rebalance Protocol. With this KIP, we aim
to make the rebalance protocol (for consumers) more reliable, more
scalable, easier to implement for clients, and easier to debug for
operators.
The KIP is
55 matches
Mail list logo
