Re: [ANNOUNCE] New Committer: Vahid Hashemian

This is exciting moment! Congrats Vahid! Boyang From: Rajini Sivaram Sent: Wednesday, January 16, 2019 6:50 AM To: Users Cc: dev Subject: Re: [ANNOUNCE] New Committer: Vahid Hashemian Congratulations, Vahid! Well deserved!! Regards, Rajini On Tue, Jan 15,

[jira] [Created] (KAFKA-7824) Require member.id for initial join group request

Boyang Chen created KAFKA-7824: -- Summary: Require member.id for initial join group request Key: KAFKA-7824 URL: https://issues.apache.org/jira/browse/KAFKA-7824 Project: Kafka Issue Type

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

Hey Konstantine, great work for making this happen! Incremental rebalancing is super important for avoiding unnecessary resource shuffle and improving the overall Connect framework stability. After the first pass, two questions across my mind are: 1. For my understanding, the general

Re: [VOTE] KIP-414: Expose Embedded ClientIds in Kafka Streams

Hey Guozhang, thanks a lot for proposing the KIP so fast! Two high level comments are: 1. In the code we define the function to extract admin client id as `getShardAdminClientId`. Do we also want to add `shared` keyword in the public API so that people know the admin client is not

Re: [DISCUSS] KIP-300: Add Windowed KTable API in StreamsBuilder

to get this into 2.2 release? KIP deadline is 1/24, so quite soon. Overall, the KIP is very useful. I can review again in more details if you aim for 2.2 -- did you address all previous comment about the KIP already? -Matthias On 1/8/19 2:50 PM, Boyang Chen wrote: > Hey folks, > >

[jira] [Created] (KAFKA-7816) Windowed topic should have window size as part of the metadata

Boyang Chen created KAFKA-7816: -- Summary: Windowed topic should have window size as part of the metadata Key: KAFKA-7816 URL: https://issues.apache.org/jira/browse/KAFKA-7816 Project: Kafka

Re: [VOTE] KIP-389: Introduce a configurable consumer group size limit

+1 (non-binding) Thanks for the great work! Get Outlook for iOS From: Gwen Shapira Sent: Friday, January 11, 2019 9:13 AM To: dev Subject: Re: [VOTE] KIP-389: Introduce a configurable consumer group size limit +1 Thank you for driving

[DISCUSS] KIP-300: Add Windowed KTable API in StreamsBuilder

Hey folks, I would like to start a discussion thread on adding new time/session windowed KTable APIs for KStream: https://cwiki.apache.org/confluence/display/KAFKA/KIP-300%3A+Add+Windowed+KTable+API+in+StreamsBuilder We have been working on this thread around 7 months ago, and it is

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

will be managing 2.2 release and hope to get some work done before the deadline. Feel free to keep raising comments or questions on the discussion thread, I will reply them at earliest convenience. Best, Boyang From: Boyang Chen Sent: Saturday, January 5, 2019 8:58 AM

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

On Thu, Jan 3, 2019 at 1:59 AM Boyang Chen wrote: > Thanks Gwen for the suggestion! +1 on the guidance of defining > group.max.size. I guess a sample formula would be: > 2 * (# of brokers * average metadata cache size * 80%) / (# of consumer > groups * size of a single member metadat

[jira] [Reopened] (KAFKA-4835) Allow users control over repartitioning

[ https://issues.apache.org/jira/browse/KAFKA-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Chen reopened KAFKA-4835: > Allow users control over repartition

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

if there is a good reason to keep them separate. In any case, my vote is +1. Thanks for the hard work on this KIP! Best, Jason On Fri, Jan 4, 2019 at 1:09 PM Boyang Chen wrote: > Thanks Guozhang for the proposal! The update is done. > > > From: Guozhang

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

I think we can continue the voting process after that. Guozhang On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen wrote: > Thanks Jason for the comment! I answered it on the discuss thread. > > Folks, could we continue the vote for this KIP? This is a very critical > improvement fo

[jira] [Resolved] (KAFKA-4835) Allow users control over repartitioning

[ https://issues.apache.org/jira/browse/KAFKA-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Chen resolved KAFKA-4835. Resolution: Fixed > Allow users control over repartition

Re: [DISCUSS] KIP-408: Add Asynchronous Processing to Kafka Streams

ad of just a callback. >>>> I don't know whether or not the tradeoffs are acceptable to the client. >>>> Ordering could be guaranteed, but it would be hard to do. For example, if >>>> there was a crash, we might lose track of which offsets numbers and ranges >>>

Re: Suggestion to make 0 grace period the default for suppress

Thanks for the proposal Jingguo. Guozhang when you mention to "change the default value of retention to 0 consistently", will we introduce any backward incompatible issue since the retention cutoff might change on the changelog topics that are unexpected. Boyang

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

g, > > If there aren't any more thoughts on the KIP I'll start a vote thread in > the new year > > On Sat, Dec 29, 2018 at 12:58 AM Boyang Chen wrote: > > > Yep Stanislav, that's what I'm proposing, and your explanation makes sense. > > > > Boyang > >

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

member be removed? I know the consumer will not do this, but we'll still have to handle the case on the broker. Best, Jason On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen wrote: > Thanks Stanislav！ > > Get Outlook for iOS<https://aka.ms/o0ukef> > >

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

from pov of triggering rebalances. This is assuming that the time interval to move from static to dynamic membership is small. Thanks, Mayuresh On Fri, Dec 28, 2018 at 11:37 PM Boyang Chen wrote: > Hey Mayuresh, > > thanks for giving another thorough thought on this matter! The session >

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hey Mayuresh, thanks for giving another thorough thought on this matter! The session timeout is controlled on the member level (or more specifically member.id level), which has nothing to do with the static member mapping. Putting aside the low possibility of having a duplicate member id

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

but I can't seem to come up with a better alternative > So either the timeout or all members calling joinGroup, yes > > > On Tue, Dec 11, 2018 at 8:14 PM Boyang Chen wrote: > >> Hey Jason, >> >> I think this is the correct understanding. One more question is whethe

Re: KIP-408: Add Asynchronous Processing to Kafka Streams

some time later on. But I honestly don't know what they will do. It would be good if you could give me some insight into this. Cheers, Richard On Fri, Dec 21, 2018 at 4:24 PM Boyang Chen wrote: > Thanks Richard for proposing this feature! We also have encountered some > similar feature r

Re: SessionWindows type base class

ndowedBy()` each accepting a different type of window and both use a different operator implementation internally. Does this answer your question? -Matthias On 12/23/18 5:23 AM, Boyang Chen wrote: > Hey there, > > I was working on some KTable API related stuff and noticed something a little >

SessionWindows type base class

Hey there, I was working on some KTable API related stuff and noticed something a little bit confusing. For class SessionWindows that defines the spec for a session window, it doesn't have a base class: public final class SessionWindows On the other hand, the `TimeWindows` class is subclass

Re: KIP-408: Add Asynchronous Processing to Kafka Streams

Thanks Richard for proposing this feature! We also have encountered some similar feature request that we want to define a generic async processing API. However I guess the motivation here is that we should skip big records during normal

[jira] [Created] (KAFKA-7728) Add JoinReason to the join group request for better rebalance handling

Boyang Chen created KAFKA-7728: -- Summary: Add JoinReason to the join group request for better rebalance handling Key: KAFKA-7728 URL: https://issues.apache.org/jira/browse/KAFKA-7728 Project: Kafka

Kafka Trunk Build Failure with Gradle 5.0

Hey friends, I'm wondering anyone has seen this error before: Could not find method annotationProcessor() for arguments [org.openjdk.jmh:jmh-generator-annprocess:1.21] on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler. Basically I'm building the

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

prevent some of them from rejoining after the rebalance completes. It sounds reasonable if I've understood correctly. Thanks, Jason On Fri, Dec 7, 2018 at 6:47 AM Boyang Chen wrote: > Yep, LGTM on my side. Thanks Stanislav! > > From: Stanislav Kozlovsk

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

lances This is great work, Boyang. Thank you very much. +1 (non-binding) On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen wrote: > Hey there, could I get more votes on this thread? > > Thanks for the vote from Mayuresh and Mike :) > > Best, > Boyang > >

Re: [VOTE] KIP-394: Require member.id for initial join group request

. This is very much needed. -Harsha On Mon, Dec 10, 2018, at 11:00 AM, Guozhang Wang wrote: > +1. Thanks Boyang! > > > Guozhang > > On Mon, Dec 10, 2018 at 10:29 AM Jason Gustafson wrote: > > > +1 Thanks for the KIP, Boyang! > > > > -Jason > > > >

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

in a rush, we may need to extend or modify > it soon > after we realize its limits in operations. > > Otherwise, I'm +1 on the proposal. > > Guozhang > > > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen > wrote: > > > Hey co

Re: [VOTE] KIP-394: Require member.id for initial join group request

for the KIP. -Bill On Wed, Dec 5, 2018 at 1:43 PM Matthias J. Sax wrote: > Thanks for the KIP. > > +1 (binding) > > -Matthias > > > On 12/5/18 7:53 AM, Mayuresh Gharat wrote: > > +1 (non-binding) > > > > Thanks, > > > > Mayuresh > &g

Re: Vote for KIP-393 (Fix time windowed serde to deserialize changelog topic)

Hey Shawn, thanks for the great writeup! The motivation is super clear and solution makes a lot of sense. +1 (non-binding) Boyang From: Liquan Pei Sent: Friday, December 7, 2018 5:22 AM To: dev@kafka.apache.org; shavvnnngu...@gmail.com Subject: Re: Vote for

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

that they would improve KIP-389's behavior as well as the normal rebalance scenarios On Wed, Dec 5, 2018 at 12:09 PM Boyang Chen wrote: > Hey Stanislav, > > thanks for the question! `Trivial rebalance` means "we don't start > reassignment right now, but you need to know it's coming soon

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

`joinGroup` requests. Does that sound like it would work? On Wed, Dec 5, 2018 at 1:13 AM Boyang Chen wrote: > Hey Stanislav, > > I read the latest KIP and saw that we already changed the default value to > -1. Do > we still need to take care of the consumer group shrink

[VOTE] KIP-394: Require member.id for initial join group request

Hey friends, I would like to start a vote for KIP-394. The goal of this KIP is to improve broker stability by fencing invalid join group requests. Best, Boyang

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

ctively rebuilding the group, since > the > > > > cache of active consumers is not stored outside the Coordinator's > > memory. > > > > (please do say if that is incorrect) > > > > Then, I believe that working as if this is a new group is a > reasonable &g

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

o: dev Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances 2. That's a good point. I think I'm convinced. Guozhang On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen wrote: > Thanks Guozhang for the reply! > > 1. RemoveMemberFromGroupOptions seems not d

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

membership protocol to reduce consumer rebalances Hi Folks, Would it be good to move this to the DISCUSS thread and keep this thread only for voting purposes, else it will be hard to coordinate responses between 2 threads. Thanks, Mayuresh On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen wrote

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

add the tool in a rush, we may need to extend or modify it soon after we realize its limits in operations. Otherwise, I'm +1 on the proposal. Guozhang On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen wrote: > Hey community friends, > > after another month of polishing, KIP-345< >

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

Thanks for your time! Boyang From: Boyang Chen Sent: Friday, November 9, 2018 6:35 AM To: dev@kafka.apache.org Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances Hey all, thanks so much for all the inputs on KIP-345

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

since you two involve most in this KIP, could you let me know if there is still any unclarity we want to resolve before moving to vote? Best, Boyang From: Boyang Chen Sent: Saturday, December 1, 2018 10:53 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] K

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

rejoin should be > quick anyway, so I don't think it's worth optimizing for unjoined new > members. Just my two cents. This is more of an implementation detail, so > need not necessarily be resolved here. > > -Jason > > On Fri, Nov 30, 2018 at 12:56 AM Boyang Chen wrote: > >&

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

option that avoids this issue. Thoughts? -Matthias On 11/28/18 8:15 PM, Boyang Chen wrote: > Thanks Matthias for the question, and Stanislav for the explanation! > > For the scenario described, we will never let a member join the GroupMetadata > map > if it uses UNKNOW

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

prefix, and list[instanceid] for clarity purpose. As you know, two options are equivalent since full name is subset of prefix. Let me know your thoughts! Boyang From: Boyang Chen Sent: Thursday, November 29, 2018 3:39 PM To: dev@kafka.apache.org Subject: Re

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

and `instance.id` field of LeaveGroupRequest to be an array instead of a single entry. e) We can also remove the admin ConsumerRebalanceRequest as well for simplicity (why not? paranoid of having as less request protocols as possible :), as it is not needed anymore with the above proposal. WDYT?

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

roup request keeps failing due to > connection timeout, or the consumer keeps restarting, > > From my understanding, this KIP move the issue from the first to the > second joinGroup request (or broker joinGroup response). > > But maybe I am missing something. Can you help me out? > >

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

tlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-394%253A%2BRequire%2Bmember.id%2Bfor%2Binitial%2Bjoin%2Bgroup%2Brequestdata=02%7C01%7C%7Ccd1e9e2eee0f440987bf08d654aa1dd9%7C84df9e7fe9f640afb435%7C1%7C0%7C636789486977454595sdata=lEv9SuBZgATGhOSe5zUr%

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

sumers do not have this logic. > > Thanks, > Jason > > On Mon, Nov 26, 2018 at 5:47 PM Boyang Chen wrote: > > > Hey friends, > > > > > > I would like to start a discussion thread for KIP-394 which is trying to > > mitigate broker cache bursting issue

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

o send this request to. > - I assume this should be same as the way we find the coordinator, > today right? If yes, should we specify it in the KIP ? >7. Are there any specific failure scenarios when you say "other >potential failure cases."? It would be goo

[DISCUSS] KIP-394: Require member.id for initial join group request

Hey friends, I would like to start a discussion thread for KIP-394 which is trying to mitigate broker cache bursting issue due to anonymous join group requests: https://cwiki.apache.org/confluence/display/KAFKA/KIP-394%3A+Require+member.id+for+initial+join+group+request Thanks! Boyang

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

"invoke". And it seems simpler to allows this API to trigger rebalance regardless of whether consumer is configured with memberName. 8) It is not very clear how the newly added AdminClient API trigger rebalance. For example, does it send request? Can this be explained in the KIP? Thanks, Dong

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

be global or *per-topic*. For the time >> being, I believe that having it per-topic with a global default might be >> the best situation. Having it global only seems a bit restricting to me and >> it never hurts to support more fine-grained configurability (given it's the >> s

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

APIs to set hard-coded client ids to the group and replace the dead host, or as you proposed to define spare host as what I understood as hot backup. I will put both Jason and your suggestions into a separate section called "Future works". Note that this spare host idea may be also solvable

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

lso below I read: "By allowing > consumers to optionally specifying a member id" which probably implies > "member name" again. In a sense this section highlights a potential > confusion between member name and member id. I wonder if we could come up > with a better term fo

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

we could come up with a better term for the new field. StaticTag, StaticLabel, or even StaticName are some suggestions that could potentially help with confusion between MemberId and MemberName and what corresponds to what. But I wouldn't like to disrupt the discussion with naming conventions too muc

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Thanks Guozhang for the great summary here, and I have been following up the action items here. 1. I already updated the KIP to remove the expansion timeout and registration timeout. Great to see them being addressed in client side! 2. I double checked the design and I believe that it is

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

luster or per broker? Either way the default value seems a bit high to me, but that could just be from my own usage patterns. I’d have probably started with 500 or 1k but could be easily convinced that’s wrong. Thanks, Matt On Mon, Nov 19, 2018 at 8:51 PM Boyang Chen wrote: > Hey folks, >

[Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

Hey folks, I would like to start a discussion on KIP-389: https://cwiki.apache.org/confluence/display/KAFKA/KIP-389%3A+Enforce+group.max.size+to+cap+member+metadata+growth This is a pretty simple change to cap the consumer group size for broker stability. Give me your valuable feedback when

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

nge much more > quickly. There would be no need to wait 5 minutes for all members to join > and it would be robust in the presence of failures. Ironically, static > membership in this case makes the group more dynamic ;). > > That said, I can see how the registration timeout

[jira] [Created] (KAFKA-7641) Add `group.max.size` to cap group metadata on broker

Boyang Chen created KAFKA-7641: -- Summary: Add `group.max.size` to cap group metadata on broker Key: KAFKA-7641 URL: https://issues.apache.org/jira/browse/KAFKA-7641 Project: Kafka Issue Type

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

cause a broker is down. The expectation is that it the broker will eventually return. Similarly for stateful applications, we would have the option to do the same thing. The registration is fixed until someone changes it. Does that make sense? Thanks, Jason On Wed, Nov 14, 2018 at 2:46 PM, Boyang Chen

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

ay, it would be helpful to include the full schema definition for any protocol changes in the proposal. Thanks, Jason On Mon, Nov 12, 2018 at 8:56 AM, Boyang Chen wrote: > Thanks Mayuresh for the feedback! Do you have a quick example for passing > in consumer config dynamically? I mainly use K

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

behavior. Just my 2 cents. Thanks, Mayuresh On Fri, Nov 9, 2018 at 8:27 PM Boyang Chen wrote: > Hey Mayuresh, > > > thanks for the thoughtful questions! Let me try to answer your questions > one by one. > > > For having a consumer config at runtime, I think it's not neces

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

nal topic "static_member_map". Would the semantics (GroupCoordinator broker, topic configs) be the same as __consumer_offsets topic? Thanks, Mayuresh On Wed, Nov 7, 2018 at 12:17 AM Boyang Chen wrote: > I took a quick pass of the proposal. First I would say it's a very > bril

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

Adding discussion thread here: https://lists.apache.org/thread.html/9868b07692960d7ab2924d2c66ad6ce7e42e492c35af9cbb7c9981ae@%3Cdev.kafka.apache.org%3E From: Boyang Chen Sent: Friday, November 9, 2018 6:35 AM To: dev@kafka.apache.org Subject: [VOTE] KIP-345

[VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

Hey all, thanks so much for all the inputs on KIP-345 so far. The original proposal has enhanced a lot with your help. To make sure the implementation go smoothly without back and forth, I would like to start a vote on the final design agreement now:

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

at complementation to KIP 345. In a high level, we are not having any collision on the path and both proposals are making sense here. Just need better sync to avoid duplicate effort :) Best, Boyang From: Boyang Chen Sent: Wednesday, November 7, 2018 1:57 PM T

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

ce/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies We should make sure that the proposal and this KIP align to each other. Thoughts? -Matthias On 11/5/18 7:31 PM, Boyang Chen wrote: > Hey Mike, > > > thanks for the feedback, the two question are very thou

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

dea I had was adding the ability for a CLI or Admin API to force a rebalance of the group. This would allow for an admin to trigger the rebalance manually without waiting the entire registration timeout on shrink down. What do you think? Mike On 10/30/18, 1:55 AM, "Boyang Chen" wrote

Re: KSQL with changelog topic

. -Matthias On 11/1/18 9:28 PM, Boyang Chen wrote: > For example, I have a stream windowed aggregation changlog topic > KStream--XXX--changelog-26, does KSQL support a simple query like > > ``` > > select * from > > changelog_topic > > limit 10 > > ``

Re: KSQL with changelog topic

. Thank you! From: Matthias J. Sax Sent: Friday, November 2, 2018 5:06 AM To: dev@kafka.apache.org Subject: Re: KSQL with changelog topic I am not sure if I understand the question. Can you elaborate? -Matthias On 11/1/18 10:38 AM, Boyang Chen wrote: >

KSQL with changelog topic

Hey there, a dumb question that I haven't found the answer anywhere: does KSQL support query towards Stream applications' changelog topic? Especially window store changelog, since it has a more complex key schema. Thanks!

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

ot; which means that for given M partitions and N consumers in one consumer group, Kafka will try to balance the load between consumers and ideally have ... ________ From: Boyang Chen Sent: Monday, October 29, 2018 12:34 PM To: dev@kafka.apache.org Subject: Re: [DISCUSS]

[jira] [Created] (KAFKA-7566) Add sidecar job to leader (or a random single follower) only

Boyang Chen created KAFKA-7566: -- Summary: Add sidecar job to leader (or a random single follower) only Key: KAFKA-7566 URL: https://issues.apache.org/jira/browse/KAFKA-7566 Project: Kafka

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

the consumer takes longer than session timeout to > >> return, > >>>> we shall still trigger rebalance but it could still try to catch > >>>> `change-group-timeout`.": what does this mean? I thought your proposal > >> is > >>>&

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Thanks Matthias for the input. Sorry I was busy recently and haven't got time to update this thread. To summarize what we come up so far, here is a draft updated plan: Introduce a new config called `member.name` which is supposed to be provided uniquely by the consumer client. The broker will

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

n Sat, Aug 4, 2018 at 10:13 PM, Boyang Chen wrote: > Hey Guozhang, > > > I think this makes sense. I'm wondering whether we could reuse the > consumer client id as unique identifier instead of introducing new terms. > From my experience, they should be unique in either S

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

e of any requests sent to the coordinator, not saying that they will only be aware until then. In fact, when they sent Heartbeat request to the coordinator, they could also be notified that "you are fenced". Guozhang On Wed, Aug 1, 2018 at 10:41 PM, Boyang Chen wrote: > Hey Guozhang, > &g

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hey Guozhang, do you think my understanding makes sense? I would also like to hear from Jason, Mike, John and everyone's opinion on this new approach. Thanks! From: Boyang Chen Sent: Thursday, August 2, 2018 1:41 PM To: dev@kafka.apache.org Subject: Re

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

onsumer, while updating its cache with the newly generated member id. No rebalance will be triggered, and if there are other zombies with the same registry ids and already get the assignment before, although they can still fetch data, they will not be able to commit. -- Guozh

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

s for the response. I think the main point I was trying > to > > > make > > > > is the > > > > need for fencing. I am not too concerned about how to > generate > > a > > > > unique id > > > > on t

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

-optimal and require more round-trips/data-transfer? (similar to the > current algorithm) > > I'm trying to assess the potential for user-error and the impact of > user-error. > > -James > > > On Jul 27, 2018, at 11:25 AM, Boyang Chen wrote: > > > > Hey Jame

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

users, we should consider > setting the default of internal `LEAVE_GROUP_ON_CLOSE_CONFIG` to false, > since otherwise its effectiveness would be less. > > > Guozhang > > > >> On Thu, Jul 26, 2018 at 9:20 PM, Boyang Chen wrote: >> >> Hey friends, >>

[DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hey friends, I would like to open a discussion thread on KIP-345: https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Reduce+multiple+consumer+rebalances+by+specifying+member+id This KIP is trying to resolve multiple rebalances by maintaining the consumer member id across rebalance

Re: Stream config in StreamPartitionAssignor

to reduce confusions. Guozhang On Mon, Jul 9, 2018 at 10:22 PM, Boyang Chen wrote: > Hey Guozhang, > > > thanks for the reply. Actually I'm not trying to pass config into > StreamPartitionAssignor. My debugging was about some other configs that > wasn't transferred properly (com

Re: Stream config in StreamPartitionAssignor

on, Jul 9, 2018 at 2:06 PM, Boyang Chen wrote: > Hey there, > > > over the weekend I was debugging the streams configuration not passed > within threads. I noticed that one of the code path from KafkaConsumer > (L743) was to initialize the StreamPartitionAssignor: > > th

Stream config in StreamPartitionAssignor

Hey there, over the weekend I was debugging the streams configuration not passed within threads. I noticed that one of the code path from KafkaConsumer (L743) was to initialize the StreamPartitionAssignor: this.assignors = config.getConfiguredInstances(

Customizing initial consumer group rebalance delay

Hey there, I have a question regarding KIP-134. Basically this config `group.initial.rebalance.delay.ms` is set on broker side to delay the initial rebalance of consumer groups. For certain

[jira] [Resolved] (KAFKA-7071) specify number of partitions when using repartition logic

[ https://issues.apache.org/jira/browse/KAFKA-7071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Chen resolved KAFKA-7071. Resolution: Duplicate Dup with 6037 > specify number of partitions when using repartition lo

[jira] [Created] (KAFKA-7071) specify number of partitions when using repartition logic

Boyang Chen created KAFKA-7071: -- Summary: specify number of partitions when using repartition logic Key: KAFKA-7071 URL: https://issues.apache.org/jira/browse/KAFKA-7071 Project: Kafka Issue

[jira] [Created] (KAFKA-7018) persist memberId for consumer restart

Boyang Chen created KAFKA-7018: -- Summary: persist memberId for consumer restart Key: KAFKA-7018 URL: https://issues.apache.org/jira/browse/KAFKA-7018 Project: Kafka Issue Type: Improvement

[jira] [Created] (KAFKA-6995) Make config "internal.leave.group.on.close" public

Boyang Chen created KAFKA-6995: -- Summary: Make config "internal.leave.group.on.close" public Key: KAFKA-6995 URL: https://issues.apache.org/jira/browse/KAFKA-6995 Project: Kafka

[jira] [Created] (KAFKA-6986) Export Admin Client metrics through Stream Threads

Boyang Chen created KAFKA-6986: -- Summary: Export Admin Client metrics through Stream Threads Key: KAFKA-6986 URL: https://issues.apache.org/jira/browse/KAFKA-6986 Project: Kafka Issue Type

Re: [DISCUSSION] KIP-300: Add Windowed KTable API in StreamsBuilder

:29 AM To: dev@kafka.apache.org Subject: Re: [DISCUSSION] KIP-300: Add Windowed KTable API in StreamsBuilder This KIP makes sharing a WindowedKTable among Kafka Stream jobs very easy. It would be nice to get this into trunk soon. Best, Liquan On Mon, May 21, 2018 at 12:25 PM, Boyang Chen

[DISCUSSION] KIP-300: Add Windowed KTable API in StreamsBuilder

Hey all, I would like to start a discussion thread on KIP 300, which introduces a new API called windowedTable() in StreamsBuilder: https://cwiki.apache.org/confluence/display/KAFKA/KIP-300%3A+Add+Windowed+KTable+API+in+StreamsBuilder The pull request I'm working on is here:

[jira] [Created] (KAFKA-6896) add producer metrics exporting in KafkaStreams.java

Boyang Chen created KAFKA-6896: -- Summary: add producer metrics exporting in KafkaStreams.java Key: KAFKA-6896 URL: https://issues.apache.org/jira/browse/KAFKA-6896 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-6657) Add StreamsConfig prefix for different consumers

[ https://issues.apache.org/jira/browse/KAFKA-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Chen resolved KAFKA-6657. Resolution: Delivered The code is merged: https://github.com/apache/kafka/pull/4805 >

[jira] [Created] (KAFKA-6840) support windowing in ktable API

Boyang Chen created KAFKA-6840: -- Summary: support windowing in ktable API Key: KAFKA-6840 URL: https://issues.apache.org/jira/browse/KAFKA-6840 Project: Kafka Issue Type: Improvement

Re: [VOTE] KIP-276 Add StreamsConfig prefix for different consumers

gt; > +1 > > > > Thanks, > > Bill > > > > On Thu, Apr 19, 2018 at 9:50 AM, Boyang Chen <bche...@outlook.com> > wrote: > > > > > Thanks guys！ > > > > > > Sent from my iPhone > > > > > > > On Apr

<    2   3   4   5   6   7   8   >