Sounds good. On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps <jay.kr...@gmail.com> wrote:
> None on my part. > > -Jay > > On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > One amendment I would like to bring up for consideration wrt the KIP > > process (before we formally include it in our by-laws) is to not > > restrict the votes to be a lazy majority of the PMC, and to instead > > make it a lazy majority of committers. > > > > Our current requirement for code changes per our by-laws are +1 from a > > committer (who is not the contributor) followed by lazy approval. I > > think a lazy majority vote for more significant code changes (i.e., a > > KIP) should be sufficient. > > > > Any objection to this? > > > > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote: > > > Great! Sounds like everyone is on the same page > > > > > > - I created a template page to make things easier. If you do > > Tools->Copy > > > on this page you can just fill in the italic portions with your > > details. > > > - I retrofitted KIP-1 to match this formatting > > > - I added the metadata section people asked for (a link to the > > > discussion, the JIRA, and the current status). Let's make sure we > > remember > > > to update the current status as things are figured out. > > > - Let's keep the discussion on the mailing list rather than on the > > wiki > > > pages. It makes sense to do one or the other so all the comments are > > in one > > > place and I think prior experience is that the wiki comments are the > > worse > > > way. > > > > > > I think it would be great do KIPs for some of the in-flight items folks > > > mentioned. > > > > > > -Jay > > > > > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira <gshap...@cloudera.com> > > wrote: > > > > > > > +1 > > > > > > > > Will be happy to provide a KIP for the multiple-listeners patch. > > > > > > > > Gwen > > > > > > > > On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein <joe.st...@stealth.ly> > > wrote: > > > > > +1 to everything we have been saying and where this (has settled > > to)/(is > > > > > settling to). > > > > > > > > > > I am sure other folks have some more feedback and think we should > > try to > > > > > keep this discussion going if need be. I am also a firm believer of > > form > > > > > following function so kicking the tires some to flesh out the > > details of > > > > > this and have some organic growth with the process will be healthy > > and > > > > > beneficial to the community. > > > > > > > > > > For my part, what I will do is open a few KIP based on some of the > > work I > > > > > have been involved with for 0.8.3. Off the top of my head this > would > > > > > include 1) changes to re-assignment of partitions 2) kafka cli 3) > > global > > > > > configs 4) security white list black list by ip 5) SSL 6) We > probably > > > > will > > > > > have lots of Security related KIPs and should treat them all > > individually > > > > > when the time is appropriate 7) Kafka on Mesos. > > > > > > > > > > If someone else wants to jump in to start getting some of the > > security > > > > KIP > > > > > that we are going to have in 0.8.3 I think that would be great > (e.g. > > > > > Multiple Listeners for Kafka Brokers). There are also a few other > > > > tickets I > > > > > can think of that are important to have in the code in 0.8.3 that > > should > > > > > have KIP also that I haven't really been involved in. I will take a > > few > > > > > minutes and go through JIRA (one I can think of like auto assign id > > that > > > > is > > > > > already committed I think) and ask for a KIP if appropriate or if I > > feel > > > > > that I can write it up (both from a time and understanding > > perspective) > > > > do > > > > > so. > > > > > > > > > > long story short, I encourage folks to start moving ahead with the > > KIP > > > > for > > > > > 0.8.3 as how we operate. any objections? > > > > > > > > > > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang <wangg...@gmail.com > > > > > > wrote: > > > > > > > > > >> +1 on the idea, and we could mutually link the KIP wiki page with > > the > > > > the > > > > >> created JIRA ticket (i.e. include the JIRA number on the page and > > the > > > > KIP > > > > >> url on the ticket description). > > > > >> > > > > >> Regarding the KIP process, probably we do not need two phase > > > > communication > > > > >> of a [DISCUSS] followed by [VOTE], as Jay said the voting should > be > > > > clear > > > > >> while people discuss about that. > > > > >> > > > > >> About who should trigger the process, I think the only involved > > people > > > > >> would be 1) when the patch is submitted / or even the ticket is > > created, > > > > >> the assignee could choose to start the KIP process if she thought > > it is > > > > >> necessary; 2) the reviewer of the patch can also suggest starting > > KIP > > > > >> discussions. > > > > >> > > > > >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira < > > gshap...@cloudera.com> > > > > >> wrote: > > > > >> > > > > >> > +1 to Ewen's suggestions: Deprecation, status and version. > > > > >> > > > > > >> > Perhaps add the JIRA where the KIP was implemented to the > > metadata. > > > > >> > This will help tie things together. > > > > >> > > > > > >> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava > > > > >> > <e...@confluent.io> wrote: > > > > >> > > I think adding a section about deprecation would be helpful. A > > good > > > > >> > > fraction of the time I would expect the goal of a KIP is to > fix > > or > > > > >> > replace > > > > >> > > older functionality that needs continued support for > > compatibility, > > > > but > > > > >> > > should eventually be phased out. This helps Kafka devs > > understand > > > > how > > > > >> > long > > > > >> > > they'll end up supporting multiple versions of features and > > helps > > > > users > > > > >> > > understand when they're going to have to make updates to their > > > > >> > applications. > > > > >> > > > > > > >> > > Less important but useful -- having a bit of standard metadata > > like > > > > >> PEPs > > > > >> > > do. Two I think are important are status (if someone lands on > > the > > > > KIP > > > > >> > page, > > > > >> > > can they tell whether this KIP was ever completed?) and/or the > > > > version > > > > >> > the > > > > >> > > KIP was first released in. > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy < > > jjkosh...@gmail.com> > > > > >> wrote: > > > > >> > > > > > > >> > >> I'm definitely +1 on the KIP concept. As Joe mentioned, we > are > > > > already > > > > >> > >> doing this in one form or the other. However, IMO it is > fairly > > ad > > > > hoc > > > > >> > >> - i.e., a combination of DISCUSS threads, jira comments, RB > and > > > > code > > > > >> > >> comments, wikis and html documentation. In the past I have > had > > to > > > > dig > > > > >> > >> into a bunch of these to try and figure out why something was > > > > >> > >> implemented a certain way. I think KIPs can help a lot here > > first > > > > by > > > > >> > >> providing guidelines on what to think about (compatibility, > new > > > > APIs, > > > > >> > >> etc.) when working through a major feature; and second by > > becoming > > > > a > > > > >> > >> crisp source of truth documentation for new releases. E.g., > > for > > > > >> > >> feature X: see relevant KIPs: a, b, c, etc. > > > > >> > >> > > > > >> > >> On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote: > > > > >> > >> > Hey Joe, > > > > >> > >> > > > > > >> > >> > Yeah I guess the question is what is the definition of > > major? I > > > > >> agree > > > > >> > we > > > > >> > >> > definitely don't want to generate a bunch of paperwork. We > > have > > > > >> enough > > > > >> > >> > problems just getting all the contributions reviewed and > > checked > > > > in > > > > >> > in a > > > > >> > >> > timely fashion... > > > > >> > >> > > > > > >> > >> > So obviously bug fixes would not apply here. > > > > >> > >> > > > > > >> > >> > I think it is also pretty clear that big features should > get > > > > >> reviewed > > > > >> > and > > > > >> > >> > discussed. To pick on myself, for example, the log > compaction > > > > work > > > > >> was > > > > >> > >> done > > > > >> > >> > without enough public discussion about how it worked and > why > > > > >> (imho). I > > > > >> > >> > hope/claim that enough rigour in thinking about use-cases > and > > > > >> > operations > > > > >> > >> > and so on was done that it turned out well, but the > > discussion > > > > was > > > > >> > just > > > > >> > >> > between a few people with no real public output. This kind > of > > > > >> feature > > > > >> > is > > > > >> > >> > clearly a big change and something we should discuss. > > > > >> > >> > > > > > >> > >> > If we limit ourselves to just the public contracts the KIP > > > > >> introduces > > > > >> > the > > > > >> > >> > discussion would just be on the new configs and monitoring > > > > without > > > > >> > >> really a > > > > >> > >> > discussion of the design and how it works which is > obviously > > > > closely > > > > >> > >> > related. > > > > >> > >> > > > > > >> > >> > I don't think this should be more work because in practice > > we are > > > > >> > making > > > > >> > >> > wiki pages for any big thing anyway. So this would just be > a > > > > >> > consistent > > > > >> > >> way > > > > >> > >> > of numbering and structuring these pages. This would also > > give a > > > > >> clear > > > > >> > >> call > > > > >> > >> > to action: "hey kafka people, come read my wiki and think > > this > > > > >> > through". > > > > >> > >> > > > > > >> > >> > I actually thinking the voting aspect is less important. I > > think > > > > it > > > > >> is > > > > >> > >> > generally clear when there is agreement on something and > > not. So > > > > >> from > > > > >> > my > > > > >> > >> > point of view we could actually just eliminate that part if > > that > > > > is > > > > >> > too > > > > >> > >> > formal, it just seemed like a good way to formally adopt > > > > something. > > > > >> > >> > > > > > >> > >> > To address some of your comments from the wiki: > > > > >> > >> > > > > > >> > >> > 1. This doesn't inhibit someone coming along and putting > up a > > > > patch. > > > > >> > It > > > > >> > >> is > > > > >> > >> > just that when they do if it is a big thing introducing new > > > > >> > functionality > > > > >> > >> > we would ask for a little discussion on the basic > > > > feature/contracts > > > > >> > prior > > > > >> > >> > to code review. > > > > >> > >> > > > > > >> > >> > 2. We definitely definitely don't want people generating a > > lot of > > > > >> > these > > > > >> > >> > things every time they have an idea that they aren't going > to > > > > >> > implement. > > > > >> > >> So > > > > >> > >> > this is only applicable to things you absolutely will check > > in > > > > code > > > > >> > for. > > > > >> > >> We > > > > >> > >> > also don't want to be making proposals before things are > > thought > > > > >> > through, > > > > >> > >> > which often requires writing the code. So I think the right > > time > > > > >> for a > > > > >> > >> KIP > > > > >> > >> > is when you are far enough along that you know the issues > and > > > > >> > tradeoffs > > > > >> > >> but > > > > >> > >> > not so far along that you are going to be totally opposed > to > > any > > > > >> > change. > > > > >> > >> > Sometimes that is prior to writing any code and sometimes > not > > > > until > > > > >> > you > > > > >> > >> are > > > > >> > >> > practically done. > > > > >> > >> > > > > > >> > >> > The key problem I see this fixing is that there is enough > new > > > > >> > development > > > > >> > >> > happening that it is pretty hard for everyone to review > every > > > > line > > > > >> of > > > > >> > >> every > > > > >> > >> > iteration of every patch. But all of us should be fully > > aware of > > > > new > > > > >> > >> > features, the ramifications, the new public interfaces, > etc. > > If > > > > we > > > > >> > aren't > > > > >> > >> > aware of that we can't really build a holistic system that > is > > > > >> > beautiful > > > > >> > >> and > > > > >> > >> > consistent across. So the idea is that if you fully review > > the > > > > KIPs > > > > >> > you > > > > >> > >> can > > > > >> > >> > be sure that even if you don't know every new line of code, > > you > > > > know > > > > >> > the > > > > >> > >> > major changes coming in. > > > > >> > >> > > > > > >> > >> > -Jay > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein < > > > > joe.st...@stealth.ly> > > > > >> > >> wrote: > > > > >> > >> > > > > > >> > >> > > Thanks Jay for kicking this off! I think the confluence > > page > > > > you > > > > >> > wrote > > > > >> > >> up > > > > >> > >> > > is a great start. > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > The KIP makes sense to me (at a minimum) if there is > going > > to > > > > be > > > > >> any > > > > >> > >> > > "breaking change". This way Kafka can continue to grow > and > > > > blossom > > > > >> > and > > > > >> > >> we > > > > >> > >> > > have a process in place if we are going to release a > thorn > > ... > > > > and > > > > >> > >> when we > > > > >> > >> > > do it is *CLEAR* about what and why that is. We can > easily > > > > >> document > > > > >> > >> which > > > > >> > >> > > KIPs where involved with this release (which I think > > should get > > > > >> > >> committed > > > > >> > >> > > afterwards somewhere so no chance of edit after release). > > This > > > > >> > >> approach I > > > > >> > >> > > had been thinking about also allows changes to occur as > > they do > > > > >> now > > > > >> > as > > > > >> > >> long > > > > >> > >> > > as they are backwards compatible. Hopefully we never > need > > a > > > > KIP > > > > >> but > > > > >> > >> when > > > > >> > >> > > we do the PMC can vote on it and folks can read the > release > > > > notes > > > > >> > with > > > > >> > >> > > *CLEAR* understanding what is going to break their > existing > > > > >> > setup... at > > > > >> > >> > > least that is how I have been thinking about it. > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > Let me know what you think about this base minimum > > approach... > > > > I > > > > >> > hadn't > > > > >> > >> > > really thought of the KIP for *ANY* "major change" and I > > have > > > > to > > > > >> > think > > > > >> > >> more > > > > >> > >> > > about that. I have some other comments for minor items in > > the > > > > >> > >> confluence > > > > >> > >> > > page I will make once I think more about how I feel > having > > a > > > > KIP > > > > >> for > > > > >> > >> more > > > > >> > >> > > than what I was thinking about already. > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > I do think we should have "major changes" go through > > > > confluence, > > > > >> > >> mailing > > > > >> > >> > > list discuss and JIRA but kind of feel we have been doing > > that > > > > >> > already > > > > >> > >> ... > > > > >> > >> > > if there are cases where that isn't the case we should > > > > highlight > > > > >> and > > > > >> > >> learn > > > > >> > >> > > from them and formalize that more if need be. > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > /******************************************* > > > > >> > >> > > Joe Stein > > > > >> > >> > > Founder, Principal Consultant > > > > >> > >> > > Big Data Open Source Security LLC > > > > >> > >> > > http://www.stealth.ly > > > > >> > >> > > Twitter: @allthingshadoop < > > > > >> http://www.twitter.com/allthingshadoop> > > > > >> > >> > > ********************************************/ > > > > >> > >> > > > > > > >> > >> > > On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps < > > > > jay.kr...@gmail.com> > > > > >> > >> wrote: > > > > >> > >> > > > > > > >> > >> > > > The idea of KIPs came up in a previous discussion but > > there > > > > was > > > > >> no > > > > >> > >> real > > > > >> > >> > > > crisp definition of what they were. Here is an attempt > at > > > > >> > defining a > > > > >> > >> > > > process: > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > >> > > > > > >> > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > >> > >> > > > > > > > >> > >> > > > The trick here is to have something light-weight enough > > that > > > > it > > > > >> > >> isn't a > > > > >> > >> > > > hassle for small changes, but enough so that changes > get > > the > > > > >> > >> eyeballs of > > > > >> > >> > > > the committers and heavy users. > > > > >> > >> > > > > > > > >> > >> > > > Thoughts? > > > > >> > >> > > > > > > > >> > >> > > > -Jay > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > >> > >> > > > > >> > > > > > > >> > > > > > > >> > > -- > > > > >> > > Thanks, > > > > >> > > Ewen > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> -- > > > > >> -- Guozhang > > > > >> > > > > > > > > > -- Thanks, Neha
