Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth
Please take me off this Discuss list Best, Patrick Williams Sales Manager, UK & Ireland, Nordics & Israel StorageOS +44 (0)7549 676279 patrick.willi...@storageos.com 20 Midtown 20 Proctor Street Holborn London WC1V 6NX Twitter: @patch37 LinkedIn: linkedin.com/in/patrickwilliams4 <http://linkedin.com/in/patrickwilliams4> https://slack.storageos.com/ On 03/12/2018, 12:35, "Stanislav Kozlovski" wrote: Hi Jason, > 2. Do you think we should make this a dynamic config? I'm not sure. Looking at the config from the perspective of a prescriptive config, we may get away with not updating it dynamically. But in my opinion, it always makes sense to have a config be dynamically configurable. As long as we limit it to being a cluster-wide config, we should be fine. > 1. I think it would be helpful to clarify the details on how the coordinator will shrink the group. It will need to choose which members to remove. Are we going to give current members an opportunity to commit offsets before kicking them from the group? This turns out to be somewhat tricky. I think that we may not be able to guarantee that consumers don't process a message twice. My initial approach was to do as much as we could to let consumers commit offsets. I was thinking that we mark a group to be shrunk, we could keep a map of consumer_id->boolean indicating whether they have committed offsets. I then thought we could delay the rebalance until every consumer commits (or some time passes). In the meantime, we would block all incoming fetch calls (by either returning empty records or a retriable error) and we would continue to accept offset commits (even twice for a single consumer) I see two problems with this approach: * We have async offset commits, which implies that we can receive fetch requests before the offset commit req has been handled. i.e consmer sends fetchReq A, offsetCommit B, fetchReq C - we may receive A,C,B in the broker. Meaning we could have saved the offsets for B but rebalance before the offsetCommit for the offsets processed in C come in. * KIP-392 Allow consumers to fetch from closest replica <https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica> would make it significantly harder to block poll() calls on consumers whose groups are being shrunk. Even if we implemented a solution, the same race condition noted above seems to apply and probably others Given those constraints, I think that we can simply mark the group as `PreparingRebalance` with a rebalanceTimeout of the server setting ` group.max.session.timeout.ms`. That's a bit long by default (5 minutes) but I can't seem to come up with a better alternative I'm interested in hearing your thoughts. Thanks, Stanislav On Fri, Nov 30, 2018 at 8:38 AM Jason Gustafson wrote: > Hey Stanislav, > > What do you think about the use case I mentioned in my previous reply about > > a more resilient self-service Kafka? I believe the benefit there is > bigger. > > > I see this config as analogous to the open file limit. Probably this limit > was intended to be prescriptive at some point about what was deemed a > reasonable number of open files for an application. But mostly people treat > it as an annoyance which they have to work around. If it happens to be hit, > usually you just increase it because it is not tied to an actual resource > constraint. However, occasionally hitting the limit does indicate an > application bug such as a leak, so I wouldn't say it is useless. Similarly, > the issue in KAFKA-7610 was a consumer leak and having this limit would > have allowed the problem to be detected before it impacted the cluster. To > me, that's the main benefit. It's possible that it could be used > prescriptively to prevent poor usage of groups, but like the open file > limit, I suspect administrators will just set it large enough that users > are unlikely to complain. > > Anyway, just a couple additional questions: > > 1. I think it would be helpful to clarify the details on how the > coordinator will shrink the group. It will need to choose which members to > remove. Are we going to give current members an opportunity to commit > offsets before kicking them from the group? > > 2. Do you think we should make this a dynamic config? > > Thanks, > Jason > > > > > On Wed, Nov 28, 2018 at 2:42 AM Stanislav Kozlovski < > stanis...@confluent.io> > wrote: > > > Hi Jason
Re: [jira] [Created] (KAFKA-7699) Improve wall-clock time punctuations
Please take me off this JIRA list Best, Patrick Williams Sales Manager, UK & Ireland, Nordics & Israel StorageOS +44 (0)7549 676279 patrick.willi...@storageos.com 20 Midtown 20 Proctor Street Holborn London WC1V 6NX Twitter: @patch37 LinkedIn: linkedin.com/in/patrickwilliams4 <http://linkedin.com/in/patrickwilliams4> https://slack.storageos.com/ On 03/12/2018, 18:14, "Matthias J. Sax (JIRA)" wrote: Matthias J. Sax created KAFKA-7699: -- Summary: Improve wall-clock time punctuations Key: KAFKA-7699 URL: https://issues.apache.org/jira/browse/KAFKA-7699 Project: Kafka Issue Type: New Feature Components: streams Reporter: Matthias J. Sax Currently, wall-clock time punctuation allow to schedule periodic call backs based on wall-clock time progress. The punctuation time starts, when the punctuation is scheduled, thus, it's non-deterministic what is desired for many use cases (I want a call-back in 5 minutes from "now"). It would be a nice improvement, to allow users to "anchor" wall-clock punctation, too, similar to a cron job: Thus, a punctuation would be triggered at "fixed" times like the beginning of the next hour, independent when the punctuation was registered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances
Pls take me off this VOTE list Best, Patrick Williams Sales Manager, UK & Ireland, Nordics & Israel StorageOS +44 (0)7549 676279 patrick.willi...@storageos.com 20 Midtown 20 Proctor Street Holborn London WC1V 6NX Twitter: @patch37 LinkedIn: linkedin.com/in/patrickwilliams4 <http://linkedin.com/in/patrickwilliams4> https://slack.storageos.com/ On 03/12/2018, 17:34, "Guozhang Wang" wrote: Hello Boyang, I've browsed through the new wiki and there are still a couple of minor things to notice: 1. RemoveMemberFromGroupOptions seems not defined anywhere. 2. LeaveGroupRequest added a list of group instance id, but still keep the member id as a singleton; is that intentional? I think to make the protocol consistent both member id and instance ids could be plural. 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we can defer adding this while just add the corresponding calls of the LeaveGroupRequest inside Streams until we have used it in production and hence have a better understanding on how flexible or extensible if we want to add any cmd tools. The rationale is that if we do not necessarily need it now, we can always add it later with a more think-through API design, but if we add the tool in a rush, we may need to extend or modify it soon after we realize its limits in operations. Otherwise, I'm +1 on the proposal. Guozhang On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen wrote: > Hey community friends, > > after another month of polishing, KIP-345< > https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances> > design is ready for vote. Feel free to add your comment on the discussion > thread or here. > > Thanks for your time! > > Boyang > > From: Boyang Chen > Sent: Friday, November 9, 2018 6:35 AM > To: dev@kafka.apache.org > Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce > consumer rebalances > > Hey all, > > > thanks so much for all the inputs on KIP-345 so far. The original proposal > has enhanced a lot with your help. To make sure the implementation go > smoothly without back and forth, I would like to start a vote on the final > design agreement now: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-< > https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances > > > > 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances< > https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances > > > > KIP-345: Introduce static membership protocol to reduce ...< > https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances > > > cwiki.apache.org > For stateful applications, one of the biggest performance bottleneck is > the state shuffling. In Kafka consumer, there is a concept called > "rebalance" which means that for given M partitions and N consumers in one > consumer group, Kafka will try to balance the load between consumers and > ideally have ... > > > Let me know if you have any questions. > > > Best, > > Boyang > > -- -- Guozhang
Re: Confluence Wiki access
HI Matthias,
Can you help me unsubscribe from all these Kafka and Apache channels please?
Tried to follow previous directions but still getting loads of spam
Thanks
Best,
Patrick Williams
Head of Sales, UK & Ireland
+44 (0)7549 676279
patrick.willi...@storageos.com
20 Midtown
20 Proctor Street
Holborn
London WC1V 6NX
Twitter: @patch37
LinkedIn: linkedin.com/in/patrickwilliams4
<http://linkedin.com/in/patrickwilliams4>
https://slack.storageos.com/
On 20/09/2018, 17:46, "Matthias J. Sax" wrote:
Could not find this user ID. Can you double check? Also, it the
email-address really your ID?
I also tried `srinivas96alluri` -- not found either.
Note, that JIRA and Wiki is two different account (in case this is your
JIRA id).
-Matthias
On 9/20/18 3:45 AM, Srinivas Reddy wrote:
> Greetings,
>
> I need to edit KIP-373, can anyone help me with an access to this wiki id
>
> srinivas96all...@gmail.com
>
> Thank you in advance.
>
>
> -
> Srinivas
>
> - Typed on tiny keys. pls ignore typos.{mobile app}
>
Re: [VOTE] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation
Hi Matthias, I'm still getting a lot of messages from Kafka and being cc'd on lots of things I don't need to get despite unsubscribing from the lists. Is there anything more I can do? Thanks Best, Patrick Williams Enterprise New Business Manager +44 (0)7549 676279 patrick.willi...@storageos.com 20 Midtown 20 Proctor Street Holborn London WC1V 6NX Twitter: @patch37 LinkedIn: linkedin.com/in/patrickwilliams4 <http://linkedin.com/in/patrickwilliams4> https://slack.storageos.com/ On 16/09/2018, 19:40, "Matthias J. Sax" wrote: +1 (binding) -Matthias On 9/14/18 4:57 PM, Ismael Juma wrote: > Thanks for the KIP, +1 (binding). > > Ismael > > On Fri, Sep 14, 2018 at 4:56 PM Dhruvil Shah wrote: > >> Hi all, >> >> I would like to start a vote on KIP-361. >> >> Link to the KIP: >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation >> >> Thanks, >> Dhruvil >> >
Re: [ANNOUNCE] New Kafka PMC member: Dong Lin
How do I unsubscribe from this list? Best, Patrick Williams Enterprise New Business Manager +44 (0)7549 676279 patrick.willi...@storageos.com 20 Midtown 20 Proctor Street Holborn London WC1V 6NX Twitter: @patch37 LinkedIn: linkedin.com/in/patrickwilliams4 <http://linkedin.com/in/patrickwilliams4> https://slack.storageos.com/ On 21/08/2018, 09:59, "Viktor Somogyi-Vass" wrote: Apache Kafka PMC
