[DISCUSSION] KIP-300: Add Windowed KTable API in StreamsBuilder

Hey all, I would like to start a discussion thread on KIP 300, which introduces a new API called windowedTable() in StreamsBuilder: https://cwiki.apache.org/confluence/display/KAFKA/KIP-300%3A+Add+Windowed+KTable+API+in+StreamsBuilder The pull request I'm working on is here:

Re: [DISCUSSION] KIP-300: Add Windowed KTable API in StreamsBuilder

:29 AM To: dev@kafka.apache.org Subject: Re: [DISCUSSION] KIP-300: Add Windowed KTable API in StreamsBuilder This KIP makes sharing a WindowedKTable among Kafka Stream jobs very easy. It would be nice to get this into trunk soon. Best, Liquan On Mon, May 21, 2018 at 12:25 PM, Boyang Chen

Customizing initial consumer group rebalance delay

Hey there, I have a question regarding KIP-134. Basically this config `group.initial.rebalance.delay.ms` is set on broker side to delay the initial rebalance of consumer groups. For certain

getting permission to add KIP

Hey there, my name is Boyang who is contributing to Kafka Streams. My user ID is bchen225242, and this is my registered email. Could I get the permission to create KIP page? Thanks a lot!

Re: [ANNOUNCE] New committer: Matthias J. Sax

Great news Matthias! From: Kaufman Ng Sent: Monday, January 15, 2018 11:32 AM To: us...@kafka.apache.org Cc: dev@kafka.apache.org Subject: Re: [ANNOUNCE] New committer: Matthias J. Sax Congrats Matthias! On Sun, Jan 14, 2018 at 4:52 AM,

Re: Vote for KIP-245: Use Properties instead of StreamsConfig in KafkaStreams constructor

(adding a 4th constructor) Let me know your thoughts! Best, Boyang From: Boyang Chen <bche...@outlook.com> Sent: Saturday, January 13, 2018 9:37 AM To: Matthias J. Sax Subject: Re: Vote for KIP-245: Use Properties instead of StreamsConfig in KafkaS

Vote for KIP-245: Use Properties instead of StreamsConfig in KafkaStreams constructor

Hi there, I'm Boyang who is a newbie contributor to Kafka. I would like to start a vote for the KIP-245: https://cwiki.apache.org/confluence/display/KAFKA/KIP-245%3A+Use+Properties+instead+of+StreamsConfig+in+KafkaStreams+constructor This is linked with JIRA:

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

s for the response. I think the main point I was trying > to > > > make > > > > is the > > > > need for fencing. I am not too concerned about how to > generate > > a > > > > unique id > > > > on t

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

-optimal and require more round-trips/data-transfer? (similar to the > current algorithm) > > I'm trying to assess the potential for user-error and the impact of > user-error. > > -James > > > On Jul 27, 2018, at 11:25 AM, Boyang Chen wrote: > > > > Hey Jame

[DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hey friends, I would like to open a discussion thread on KIP-345: https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Reduce+multiple+consumer+rebalances+by+specifying+member+id This KIP is trying to resolve multiple rebalances by maintaining the consumer member id across rebalance

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

e of any requests sent to the coordinator, not saying that they will only be aware until then. In fact, when they sent Heartbeat request to the coordinator, they could also be notified that "you are fenced". Guozhang On Wed, Aug 1, 2018 at 10:41 PM, Boyang Chen wrote: > Hey Guozhang, > &g

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hey Guozhang, do you think my understanding makes sense? I would also like to hear from Jason, Mike, John and everyone's opinion on this new approach. Thanks! From: Boyang Chen Sent: Thursday, August 2, 2018 1:41 PM To: dev@kafka.apache.org Subject: Re

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

onsumer, while updating its cache with the newly generated member id. No rebalance will be triggered, and if there are other zombies with the same registry ids and already get the assignment before, although they can still fetch data, they will not be able to commit. -- Guozh

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

users, we should consider > setting the default of internal `LEAVE_GROUP_ON_CLOSE_CONFIG` to false, > since otherwise its effectiveness would be less. > > > Guozhang > > > >> On Thu, Jul 26, 2018 at 9:20 PM, Boyang Chen wrote: >> >> Hey friends, >>

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

n Sat, Aug 4, 2018 at 10:13 PM, Boyang Chen wrote: > Hey Guozhang, > > > I think this makes sense. I'm wondering whether we could reuse the > consumer client id as unique identifier instead of introducing new terms. > From my experience, they should be unique in either S

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Thanks Matthias for the input. Sorry I was busy recently and haven't got time to update this thread. To summarize what we come up so far, here is a draft updated plan: Introduce a new config called `member.name` which is supposed to be provided uniquely by the consumer client. The broker will

Re: Stream config in StreamPartitionAssignor

on, Jul 9, 2018 at 2:06 PM, Boyang Chen wrote: > Hey there, > > > over the weekend I was debugging the streams configuration not passed > within threads. I noticed that one of the code path from KafkaConsumer > (L743) was to initialize the StreamPartitionAssignor: > > th

Stream config in StreamPartitionAssignor

Hey there, over the weekend I was debugging the streams configuration not passed within threads. I noticed that one of the code path from KafkaConsumer (L743) was to initialize the StreamPartitionAssignor: this.assignors = config.getConfiguredInstances(

Re: Stream config in StreamPartitionAssignor

to reduce confusions. Guozhang On Mon, Jul 9, 2018 at 10:22 PM, Boyang Chen wrote: > Hey Guozhang, > > > thanks for the reply. Actually I'm not trying to pass config into > StreamPartitionAssignor. My debugging was about some other configs that > wasn't transferred properly (com

Re: getting permission to add KIP

Hey there, could someone help me clarify whether I'm on the mailing list for the dev email? I couldn't get reply email about my KIP vote. Thanks so much! Boyang From: Boyang Chen <bche...@outlook.com> Sent: Monday, December 25, 2017 8:45 AM T

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

prefer this) -Matthias On 4/3/18 1:59 PM, Bill Bejeck wrote: > Boyang, > > Thanks for making changes to the KIP, I'm +1 on the updated version. > > -Bill > > On Tue, Apr 3, 2018 at 1:14 AM, Boyang Chen <bche...@outlook.com> wrote: > >> Hey friends, >> >> &

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

is inherited. Same deal for a clearly >> specified set of configs for global consumers that required override. >> >> I feel like I'm also concurrently seeing the opposite side of this problem >> in Connect where we namespaced and didn't proactively implement >> inheritance;

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

Hey Ewen, could you provide some feedback on Matthias and Guozhang's reply? We hope to clear your concern and reach common understanding eventually. Thanks! Best, Boyang From: Boyang Chen <bche...@outlook.com> Sent: Friday, April 6, 2018 11:19 AM T

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

> > -Matthias > > > On 4/3/18 1:59 PM, Bill Bejeck wrote: > > Boyang, > > > > Thanks for making changes to the KIP, I'm +1 on the updated version. > > > > -Bill > > > > On Tue, Apr 3, 2018 at 1:14 AM, Boyang Chen <bche...@outlook.

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

Hey Ewen, seems that your concern is resolved. Do you mind I move forward to the voting stage for this KIP? Thank you! Boyang From: Boyang Chen <bche...@outlook.com> Sent: Sunday, April 8, 2018 2:46 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] K

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

levels of "public" configs when there are only 3 types of concrete >> configs >>>> we instantiate? >>>> >>>> More generally, I worry that we're optimizing too much to avoid >> copy/paste >>>> in less common cases to the point

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

config values for different embedded clients does help a lot in those cases, so maybe we can double check on in StreamsConfig to make sure before the clients are initialized we do print their config values (the full property map). Guozhang On Wed, Apr 11, 2018 at 7:22 PM, Boyang Chen <bche...@ou

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

Hey Ewen, in case you don't have any further concern, I shall move this KIP to the voting stage. Let me know your thoughts. Thank you! From: Boyang Chen <bche...@outlook.com> Sent: Sunday, April 15, 2018 4:25 AM To: dev@kafka.apache.org Subject: Re: [D

Re: [VOTE] KIP-276 Add StreamsConfig prefix for different consumers

ngg...@gmail.com> >> Date: 4/18/18 10:18 PM (GMT-08:00) To: dev@kafka.apache.org Subject: Re: >> [VOTE] KIP-276 Add StreamsConfig prefix for different consumers >> Thanks Boyang, +1 from me. >> >>> On Wed, Apr 18, 2018 at 4:43 PM, Boyang Chen <bche...@outl

[VOTE] KIP-276 Add StreamsConfig prefix for different consumers

Hey friends. I would like to start a vote on KIP 276: add StreamsConfig prefix for different consumers. KIP: here Pull request: here Jira:

Re: [VOTE] KIP-276 Add StreamsConfig prefix for different consumers

uot; for the restore consumer (3) "global.consumer." for the global consumer Best, Boyang ____ From: Boyang Chen <bche...@outlook.com> Sent: Tuesday, April 17, 2018 12:18 PM To: dev@kafka.apache.org Subject: [VOTE] KIP-276 Add StreamsConfig prefix for di

Re: [VOTE] KIP-276 Add StreamsConfig prefix for different consumers

gt; > +1 > > > > Thanks, > > Bill > > > > On Thu, Apr 19, 2018 at 9:50 AM, Boyang Chen <bche...@outlook.com> > wrote: > > > > > Thanks guys！ > > > > > > Sent from my iPhone > > > > > > > On Apr

[DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

Hey friends, I would like to start a discussion thread for KIP 276: https://cwiki.apache.org/confluence/display/KAFKA/KIP-276+Add+StreamsConfig+prefix+for+different+consumers And pull request is here: https://github.com/apache/kafka/pull/4805

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

>>> Looking forward to your thoughts. >>> >>> >>> -Matthias >>> >>> >>> On 4/1/18 5:55 PM, Guozhang Wang wrote: >>>> Thanks for the KIP Boyang, the KIP looks good to me. >>>> >>>> For config value

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

est, Boyang ____ From: Boyang Chen <bche...@outlook.com> Sent: Tuesday, April 3, 2018 11:39 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers Thanks Matthias, Ted and Guozhang for the inputs. I shall address t

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

the consumer takes longer than session timeout to > >> return, > >>>> we shall still trigger rebalance but it could still try to catch > >>>> `change-group-timeout`.": what does this mean? I thought your proposal > >> is > >>>&

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

ot; which means that for given M partitions and N consumers in one consumer group, Kafka will try to balance the load between consumers and ideally have ... ________ From: Boyang Chen Sent: Monday, October 29, 2018 12:34 PM To: dev@kafka.apache.org Subject: Re: [DISCUSS]

Re: KSQL with changelog topic

. Thank you! From: Matthias J. Sax Sent: Friday, November 2, 2018 5:06 AM To: dev@kafka.apache.org Subject: Re: KSQL with changelog topic I am not sure if I understand the question. Can you elaborate? -Matthias On 11/1/18 10:38 AM, Boyang Chen wrote: >

Re: KSQL with changelog topic

. -Matthias On 11/1/18 9:28 PM, Boyang Chen wrote: > For example, I have a stream windowed aggregation changlog topic > KStream--XXX--changelog-26, does KSQL support a simple query like > > ``` > > select * from > > changelog_topic > > limit 10 > > ``

KSQL with changelog topic

Hey there, a dumb question that I haven't found the answer anywhere: does KSQL support query towards Stream applications' changelog topic? Especially window store changelog, since it has a more complex key schema. Thanks!

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

nal topic "static_member_map". Would the semantics (GroupCoordinator broker, topic configs) be the same as __consumer_offsets topic? Thanks, Mayuresh On Wed, Nov 7, 2018 at 12:17 AM Boyang Chen wrote: > I took a quick pass of the proposal. First I would say it's a very > bril

[VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

Hey all, thanks so much for all the inputs on KIP-345 so far. The original proposal has enhanced a lot with your help. To make sure the implementation go smoothly without back and forth, I would like to start a vote on the final design agreement now:

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

Adding discussion thread here: https://lists.apache.org/thread.html/9868b07692960d7ab2924d2c66ad6ce7e42e492c35af9cbb7c9981ae@%3Cdev.kafka.apache.org%3E From: Boyang Chen Sent: Friday, November 9, 2018 6:35 AM To: dev@kafka.apache.org Subject: [VOTE] KIP-345

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

ce/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies We should make sure that the proposal and this KIP align to each other. Thoughts? -Matthias On 11/5/18 7:31 PM, Boyang Chen wrote: > Hey Mike, > > > thanks for the feedback, the two question are very thou

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

at complementation to KIP 345. In a high level, we are not having any collision on the path and both proposals are making sense here. Just need better sync to avoid duplicate effort :) Best, Boyang From: Boyang Chen Sent: Wednesday, November 7, 2018 1:57 PM T

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

dea I had was adding the ability for a CLI or Admin API to force a rebalance of the group. This would allow for an admin to trigger the rebalance manually without waiting the entire registration timeout on shrink down. What do you think? Mike On 10/30/18, 1:55 AM, "Boyang Chen" wrote

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

ay, it would be helpful to include the full schema definition for any protocol changes in the proposal. Thanks, Jason On Mon, Nov 12, 2018 at 8:56 AM, Boyang Chen wrote: > Thanks Mayuresh for the feedback! Do you have a quick example for passing > in consumer config dynamically? I mainly use K

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

cause a broker is down. The expectation is that it the broker will eventually return. Similarly for stateful applications, we would have the option to do the same thing. The registration is fixed until someone changes it. Does that make sense? Thanks, Jason On Wed, Nov 14, 2018 at 2:46 PM, Boyang Chen

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

behavior. Just my 2 cents. Thanks, Mayuresh On Fri, Nov 9, 2018 at 8:27 PM Boyang Chen wrote: > Hey Mayuresh, > > > thanks for the thoughtful questions! Let me try to answer your questions > one by one. > > > For having a consumer config at runtime, I think it's not neces

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

nge much more > quickly. There would be no need to wait 5 minutes for all members to join > and it would be robust in the presence of failures. Ironically, static > membership in this case makes the group more dynamic ;). > > That said, I can see how the registration timeout

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

from pov of triggering rebalances. This is assuming that the time interval to move from static to dynamic membership is small. Thanks, Mayuresh On Fri, Dec 28, 2018 at 11:37 PM Boyang Chen wrote: > Hey Mayuresh, > > thanks for giving another thorough thought on this matter! The session >

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

member be removed? I know the consumer will not do this, but we'll still have to handle the case on the broker. Best, Jason On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen wrote: > Thanks Stanislav！ > > Get Outlook for iOS<https://aka.ms/o0ukef> > >

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

if there is a good reason to keep them separate. In any case, my vote is +1. Thanks for the hard work on this KIP! Best, Jason On Fri, Jan 4, 2019 at 1:09 PM Boyang Chen wrote: > Thanks Guozhang for the proposal! The update is done. > > > From: Guozhang

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

g, > > If there aren't any more thoughts on the KIP I'll start a vote thread in > the new year > > On Sat, Dec 29, 2018 at 12:58 AM Boyang Chen wrote: > > > Yep Stanislav, that's what I'm proposing, and your explanation makes sense. > > > > Boyang > >

[VOTE] KIP-300: Add Windowed KTable API in StreamsBuilder

Hey friends, I would like to start the vote thread for KIP-300 so that we could agree on the high level API. Feel free to continue making comment on the discussion thread. Best, Boyang

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

sue in Connect. Thank you both for your initial comments. Konstantine On Mon, Jan 14, 2019 at 9:24 AM Boyang Chen wrote: > Hey Konstantine, > > great work for making this happen! Incremental rebalancing is super > important for avoiding unnecessary resource shuffle and improving the &g

Re: [DISCUSS] KIP-300: Add Windowed KTable API in StreamsBuilder

2/19 8:35 PM, Boyang Chen wrote: > Hey Matthias, > > thanks for taking a look! It would be great to see this pushed in 2.2. > Depending on the tight timeline, I hope to at least get the KIP approved so > that we don't see back and forth again as the KTable API has been constantly &

Re: [VOTE] KIP-420: Add Single Value Fetch in Session Stores

Hey Guozhang, this is nice catch! One question I have is that this seems more like a bug fix than a new feature proposal, maybe we could just update KIP-261 interface and resolve the JIRA to track the change? Boyang From: Guozhang Wang Sent: Saturday, January

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

econd join group request in the common case. If we considered and rejected this option, it would be good to include why in the "Rejected Alternatives" section. Ismael On Mon, Nov 26, 2018, 5:48 PM Boyang Chen Hey friends, > > > I would like to start a discussion thread for KI

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

Hey Konstantine, great work for making this happen! Incremental rebalancing is super important for avoiding unnecessary resource shuffle and improving the overall Connect framework stability. After the first pass, two questions across my mind are: 1. For my understanding, the general

Re: [ANNOUNCE] New Committer: Vahid Hashemian

This is exciting moment! Congrats Vahid! Boyang From: Rajini Sivaram Sent: Wednesday, January 16, 2019 6:50 AM To: Users Cc: dev Subject: Re: [ANNOUNCE] New Committer: Vahid Hashemian Congratulations, Vahid! Well deserved!! Regards, Rajini On Tue, Jan 15,

Re: [DISCUSS] KIP-300: Add Windowed KTable API in StreamsBuilder

y for >> internal implementation if we don't carry >> the window size around within existing data struct. I cannot follow here. But because this is internal stuff anyway, I would prefer to discuss this on the PR instead of the mailing list. -Matthias On 1/18/19 10:58 AM, Boyang Chen wro

[DISCUSS] KIP-221: Enhance KStream with Connecting Topic Creation and Repartition Hint

Hey all, I would like to start a new discussion thread for refined KIP-221: https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint The major goal for this KIP is simplified to empower KStream #to and #groupBy APIs

Kafka Trunk Build Failure with Gradle 5.0

Hey friends, I'm wondering anyone has seen this error before: Could not find method annotationProcessor() for arguments [org.openjdk.jmh:jmh-generator-annprocess:1.21] on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler. Basically I'm building the

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

in a rush, we may need to extend or modify > it soon > after we realize its limits in operations. > > Otherwise, I'm +1 on the proposal. > > Guozhang > > > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen > wrote: > > > Hey co

Re: [VOTE] KIP-394: Require member.id for initial join group request

for the KIP. -Bill On Wed, Dec 5, 2018 at 1:43 PM Matthias J. Sax wrote: > Thanks for the KIP. > > +1 (binding) > > -Matthias > > > On 12/5/18 7:53 AM, Mayuresh Gharat wrote: > > +1 (non-binding) > > > > Thanks, > > > > Mayuresh > &g

Re: KIP-408: Add Asynchronous Processing to Kafka Streams

Thanks Richard for proposing this feature! We also have encountered some similar feature request that we want to define a generic async processing API. However I guess the motivation here is that we should skip big records during normal

SessionWindows type base class

Hey there, I was working on some KTable API related stuff and noticed something a little bit confusing. For class SessionWindows that defines the spec for a session window, it doesn't have a base class: public final class SessionWindows On the other hand, the `TimeWindows` class is subclass

Re: SessionWindows type base class

ndowedBy()` each accepting a different type of window and both use a different operator implementation internally. Does this answer your question? -Matthias On 12/23/18 5:23 AM, Boyang Chen wrote: > Hey there, > > I was working on some KTable API related stuff and noticed something a little >

Re: [VOTE] KIP-394: Require member.id for initial join group request

. This is very much needed. -Harsha On Mon, Dec 10, 2018, at 11:00 AM, Guozhang Wang wrote: > +1. Thanks Boyang! > > > Guozhang > > On Mon, Dec 10, 2018 at 10:29 AM Jason Gustafson wrote: > > > +1 Thanks for the KIP, Boyang! > > > > -Jason > > > >

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

prevent some of them from rejoining after the rebalance completes. It sounds reasonable if I've understood correctly. Thanks, Jason On Fri, Dec 7, 2018 at 6:47 AM Boyang Chen wrote: > Yep, LGTM on my side. Thanks Stanislav! > > From: Stanislav Kozlovsk

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

lances This is great work, Boyang. Thank you very much. +1 (non-binding) On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen wrote: > Hey there, could I get more votes on this thread? > > Thanks for the vote from Mayuresh and Mike :) > > Best, > Boyang > >

Re: KIP-408: Add Asynchronous Processing to Kafka Streams

some time later on. But I honestly don't know what they will do. It would be good if you could give me some insight into this. Cheers, Richard On Fri, Dec 21, 2018 at 4:24 PM Boyang Chen wrote: > Thanks Richard for proposing this feature! We also have encountered some > similar feature r

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

and `instance.id` field of LeaveGroupRequest to be an array instead of a single entry. e) We can also remove the admin ConsumerRebalanceRequest as well for simplicity (why not? paranoid of having as less request protocols as possible :), as it is not needed anymore with the above proposal. WDYT?

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

rejoin should be > quick anyway, so I don't think it's worth optimizing for unjoined new > members. Just my two cents. This is more of an implementation detail, so > need not necessarily be resolved here. > > -Jason > > On Fri, Nov 30, 2018 at 12:56 AM Boyang Chen wrote: > >&

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

roup request keeps failing due to > connection timeout, or the consumer keeps restarting, > > From my understanding, this KIP move the issue from the first to the > second joinGroup request (or broker joinGroup response). > > But maybe I am missing something. Can you help me out? > >

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

o: dev Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances 2. That's a good point. I think I'm convinced. Guozhang On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen wrote: > Thanks Guozhang for the reply! > > 1. RemoveMemberFromGroupOptions seems not d

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

`joinGroup` requests. Does that sound like it would work? On Wed, Dec 5, 2018 at 1:13 AM Boyang Chen wrote: > Hey Stanislav, > > I read the latest KIP and saw that we already changed the default value to > -1. Do > we still need to take care of the consumer group shrink

[VOTE] KIP-394: Require member.id for initial join group request

Hey friends, I would like to start a vote for KIP-394. The goal of this KIP is to improve broker stability by fencing invalid join group requests. Best, Boyang

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

add the tool in a rush, we may need to extend or modify it soon after we realize its limits in operations. Otherwise, I'm +1 on the proposal. Guozhang On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen wrote: > Hey community friends, > > after another month of polishing, KIP-345< >

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

membership protocol to reduce consumer rebalances Hi Folks, Would it be good to move this to the DISCUSS thread and keep this thread only for voting purposes, else it will be hard to coordinate responses between 2 threads. Thanks, Mayuresh On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen wrote

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

that they would improve KIP-389's behavior as well as the normal rebalance scenarios On Wed, Dec 5, 2018 at 12:09 PM Boyang Chen wrote: > Hey Stanislav, > > thanks for the question! `Trivial rebalance` means "we don't start > reassignment right now, but you need to know it's coming soon

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

since you two involve most in this KIP, could you let me know if there is still any unclarity we want to resolve before moving to vote? Best, Boyang From: Boyang Chen Sent: Saturday, December 1, 2018 10:53 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] K

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

Thanks for your time! Boyang From: Boyang Chen Sent: Friday, November 9, 2018 6:35 AM To: dev@kafka.apache.org Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances Hey all, thanks so much for all the inputs on KIP-345

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

option that avoids this issue. Thoughts? -Matthias On 11/28/18 8:15 PM, Boyang Chen wrote: > Thanks Matthias for the question, and Stanislav for the explanation! > > For the scenario described, we will never let a member join the GroupMetadata > map > if it uses UNKNOW

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

prefix, and list[instanceid] for clarity purpose. As you know, two options are equivalent since full name is subset of prefix. Let me know your thoughts! Boyang From: Boyang Chen Sent: Thursday, November 29, 2018 3:39 PM To: dev@kafka.apache.org Subject: Re

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

ctively rebuilding the group, since > the > > > > cache of active consumers is not stored outside the Coordinator's > > memory. > > > > (please do say if that is incorrect) > > > > Then, I believe that working as if this is a new group is a > reasonable &g

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

we could come up with a better term for the new field. StaticTag, StaticLabel, or even StaticName are some suggestions that could potentially help with confusion between MemberId and MemberName and what corresponds to what. But I wouldn't like to disrupt the discussion with naming conventions too muc

[Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

Hey folks, I would like to start a discussion on KIP-389: https://cwiki.apache.org/confluence/display/KAFKA/KIP-389%3A+Enforce+group.max.size+to+cap+member+metadata+growth This is a pretty simple change to cap the consumer group size for broker stability. Give me your valuable feedback when

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

be global or *per-topic*. For the time >> being, I believe that having it per-topic with a global default might be >> the best situation. Having it global only seems a bit restricting to me and >> it never hurts to support more fine-grained configurability (given it's the >> s

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

sumers do not have this logic. > > Thanks, > Jason > > On Mon, Nov 26, 2018 at 5:47 PM Boyang Chen wrote: > > > Hey friends, > > > > > > I would like to start a discussion thread for KIP-394 which is trying to > > mitigate broker cache bursting issue

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

lso below I read: "By allowing > consumers to optionally specifying a member id" which probably implies > "member name" again. In a sense this section highlights a potential > confusion between member name and member id. I wonder if we could come up > with a better term fo

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

APIs to set hard-coded client ids to the group and replace the dead host, or as you proposed to define spare host as what I understood as hot backup. I will put both Jason and your suggestions into a separate section called "Future works". Note that this spare host idea may be also solvable

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Thanks Guozhang for the great summary here, and I have been following up the action items here. 1. I already updated the KIP to remove the expansion timeout and registration timeout. Great to see them being addressed in client side! 2. I double checked the design and I believe that it is

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

luster or per broker? Either way the default value seems a bit high to me, but that could just be from my own usage patterns. I’d have probably started with 500 or 1k but could be easily convinced that’s wrong. Thanks, Matt On Mon, Nov 19, 2018 at 8:51 PM Boyang Chen wrote: > Hey folks, >

[DISCUSS] KIP-394: Require member.id for initial join group request

Hey friends, I would like to start a discussion thread for KIP-394 which is trying to mitigate broker cache bursting issue due to anonymous join group requests: https://cwiki.apache.org/confluence/display/KAFKA/KIP-394%3A+Require+member.id+for+initial+join+group+request Thanks! Boyang

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

o send this request to. > - I assume this should be same as the way we find the coordinator, > today right? If yes, should we specify it in the KIP ? >7. Are there any specific failure scenarios when you say "other >potential failure cases."? It would be goo

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

tlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-394%253A%2BRequire%2Bmember.id%2Bfor%2Binitial%2Bjoin%2Bgroup%2Brequestdata=02%7C01%7C%7Ccd1e9e2eee0f440987bf08d654aa1dd9%7C84df9e7fe9f640afb435%7C1%7C0%7C636789486977454595sdata=lEv9SuBZgATGhOSe5zUr%

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

"invoke". And it seems simpler to allows this API to trigger rebalance regardless of whether consumer is configured with memberName. 8) It is not very clear how the newly added AdminClient API trigger rebalance. For example, does it send request? Can this be explained in the KIP? Thanks, Dong

[DISCUSS] KIP-300: Add Windowed KTable API in StreamsBuilder

Hey folks, I would like to start a discussion thread on adding new time/session windowed KTable APIs for KStream: https://cwiki.apache.org/confluence/display/KAFKA/KIP-300%3A+Add+Windowed+KTable+API+in+StreamsBuilder We have been working on this thread around 7 months ago, and it is

  1   2   3   4   5   6   7   8   >