I think this is closer to a procedural issue than a code modification issue, hence why majority. If everyone thinks consensus is better, I don't care. Again, I don't feel strongly about the way we achieve clarity, just that we achieve clarity.
On Mon, Oct 10, 2016 at 2:02 PM, Ryan Blue <rb...@netflix.com> wrote: > Sorry, I missed that the proposal includes majority approval. Why majority > instead of consensus? I think we want to build consensus around these > proposals and it makes sense to discuss until no one would veto. > > rb > > On Mon, Oct 10, 2016 at 11:54 AM, Ryan Blue <rb...@netflix.com> wrote: >> >> +1 to votes to approve proposals. I agree that proposals should have an >> official mechanism to be accepted, and a vote is an established means of >> doing that well. I like that it includes a period to review the proposal and >> I think proposals should have been discussed enough ahead of a vote to >> survive the possibility of a veto. >> >> I also like the names that are short and (mostly) unique, like SEP. >> >> Where I disagree is with the requirement that a committer must formally >> propose an enhancement. I don't see the value of restricting this: if >> someone has the will to write up a proposal then they should be encouraged >> to do so and start a discussion about it. Even if there is a political >> reality as Cody says, what is the value of codifying that in our process? I >> think restricting who can submit proposals would only undermine them by >> pushing contributors out. Maybe I'm missing something here? >> >> rb >> >> >> >> On Mon, Oct 10, 2016 at 7:41 AM, Cody Koeninger <c...@koeninger.org> >> wrote: >>> >>> Yes, users suggesting SIPs is a good thing and is explicitly called >>> out in the linked document under the Who? section. Formally proposing >>> them, not so much, because of the political realities. >>> >>> Yes, implementation strategy definitely affects goals. There are all >>> kinds of examples of this, I'll pick one that's my fault so as to >>> avoid sounding like I'm blaming: >>> >>> When I implemented the Kafka DStream, one of my (not explicitly agreed >>> upon by the community) goals was to make sure people could use the >>> Dstream with however they were already using Kafka at work. The lack >>> of explicit agreement on that goal led to all kinds of fighting with >>> committers, that could have been avoided. The lack of explicit >>> up-front strategy discussion led to the DStream not really working >>> with compacted topics. I knew about compacted topics, but don't have >>> a use for them, so had a blind spot there. If there was explicit >>> up-front discussion that my strategy was "assume that batches can be >>> defined on the driver solely by beginning and ending offsets", there's >>> a greater chance that a user would have seen that and said, "hey, what >>> about non-contiguous offsets in a compacted topic". >>> >>> This kind of thing is only going to happen smoothly if we have a >>> lightweight user-visible process with clear outcomes. >>> >>> On Mon, Oct 10, 2016 at 1:34 AM, assaf.mendelson >>> <assaf.mendel...@rsa.com> wrote: >>> > I agree with most of what Cody said. >>> > >>> > Two things: >>> > >>> > First we can always have other people suggest SIPs but mark them as >>> > “unreviewed” and have committers basically move them forward. The >>> > problem is >>> > that writing a good document takes time. This way we can leverage non >>> > committers to do some of this work (it is just another way to >>> > contribute). >>> > >>> > >>> > >>> > As for strategy, in many cases implementation strategy can affect the >>> > goals. >>> > I will give a small example: In the current structured streaming >>> > strategy, >>> > we group by the time to achieve a sliding window. This is definitely an >>> > implementation decision and not a goal. However, I can think of several >>> > aggregation functions which have the time inside their calculation >>> > buffer. >>> > For example, let’s say we want to return a set of all distinct values. >>> > One >>> > way to implement this would be to make the set into a map and have the >>> > value >>> > contain the last time seen. Multiplying it across the groupby would >>> > cost a >>> > lot in performance. So adding such a strategy would have a great effect >>> > on >>> > the type of aggregations and their performance which does affect the >>> > goal. >>> > Without adding the strategy, it is easy for whoever goes to the design >>> > document to not think about these cases. Furthermore, it might be >>> > decided >>> > that these cases are rare enough so that the strategy is still good >>> > enough >>> > but how would we know it without user feedback? >>> > >>> > I believe this example is exactly what Cody was talking about. Since >>> > many >>> > times implementation strategies have a large effect on the goal, we >>> > should >>> > have it discussed when discussing the goals. In addition, while it is >>> > often >>> > easy to throw out completely infeasible goals, it is often much harder >>> > to >>> > figure out that the goals are unfeasible without fine tuning. >>> > >>> > >>> > >>> > >>> > >>> > Assaf. >>> > >>> > >>> > >>> > From: Cody Koeninger-2 [via Apache Spark Developers List] >>> > [mailto:ml-node+[hidden email]] >>> > Sent: Monday, October 10, 2016 2:25 AM >>> > To: Mendelson, Assaf >>> > Subject: Re: Spark Improvement Proposals >>> > >>> > >>> > >>> > Only committers should formally submit SIPs because in an apache >>> > project only commiters have explicit political power. If a user can't >>> > find a commiter willing to sponsor an SIP idea, they have no way to >>> > get the idea passed in any case. If I can't find a committer to >>> > sponsor this meta-SIP idea, I'm out of luck. >>> > >>> > I do not believe unrealistic goals can be found solely by inspection. >>> > We've managed to ignore unrealistic goals even after implementation! >>> > Focusing on APIs can allow people to think they've solved something, >>> > when there's really no way of implementing that API while meeting the >>> > goals. Rapid iteration is clearly the best way to address this, but >>> > we've already talked about why that hasn't really worked. If adding a >>> > non-binding API section to the template is important to you, I'm not >>> > against it, but I don't think it's sufficient. >>> > >>> > On your PRD vs design doc spectrum, I'm saying this is closer to a >>> > PRD. Clear agreement on goals is the most important thing and that's >>> > why it's the thing I want binding agreement on. But I cannot agree to >>> > goals unless I have enough minimal technical info to judge whether the >>> > goals are likely to actually be accomplished. >>> > >>> > >>> > >>> > On Sun, Oct 9, 2016 at 5:35 PM, Matei Zaharia <[hidden email]> wrote: >>> > >>> > >>> >> Well, I think there are a few things here that don't make sense. >>> >> First, >>> >> why >>> >> should only committers submit SIPs? Development in the project should >>> >> be >>> >> open to all contributors, whether they're committers or not. Second, I >>> >> think >>> >> unrealistic goals can be found just by inspecting the goals, and I'm >>> >> not >>> >> super worried that we'll accept a lot of SIPs that are then infeasible >>> >> -- >>> >> we >>> >> can then submit new ones. But this depends on whether you want this >>> >> process >>> >> to be a "design doc lite", where people also agree on implementation >>> >> strategy, or just a way to agree on goals. This is what I asked >>> >> earlier >>> >> about PRDs vs design docs (and I'm open to either one but I'd just >>> >> like >>> >> clarity). Finally, both as a user and designer of software, I always >>> >> want >>> >> to >>> >> give feedback on APIs, so I'd really like a culture of having those >>> >> early. >>> >> People don't argue about prettiness when they discuss APIs, they argue >>> >> about >>> >> the core concepts to expose in order to meet various goals, and then >>> >> they're >>> >> stuck maintaining those for a long time. >>> >> >>> >> Matei >>> >> >>> >> On Oct 9, 2016, at 3:10 PM, Cody Koeninger <[hidden email]> wrote: >>> >> >>> >> Users instead of people, sure. Commiters and contributors are (or at >>> >> least >>> >> should be) a subset of users. >>> >> >>> >> Non goals, sure. I don't care what the name is, but we need to clearly >>> >> say >>> >> e.g. 'no we are not maintaining compatibility with XYZ right now'. >>> >> >>> >> API, what I care most about is whether it allows me to accomplish the >>> >> goals. >>> >> Arguing about how ugly or pretty it is can be saved for design/ >>> >> implementation imho. >>> >> >>> >> Strategy, this is necessary because otherwise goals can be out of line >>> >> with >>> >> reality. Don't propose goals you don't have at least some idea of how >>> >> to >>> >> implement. >>> >> >>> >> Rejected strategies, given that commiters are the only ones I'm saying >>> >> should formally submit SPARKLIs or SIPs, if they put junk in a >>> >> required >>> >> section then slap them down for it and tell them to fix it. >>> >> >>> >> >>> >> On Oct 9, 2016 4:36 PM, "Matei Zaharia" <[hidden email]> wrote: >>> >>> >>> >>> Yup, this is the stuff that I found unclear. Thanks for clarifying >>> >>> here, >>> >>> but we should also clarify it in the writeup. In particular: >>> >>> >>> >>> - Goals needs to be about user-facing behavior ("people" is broad) >>> >>> >>> >>> - I'd rename Rejected Goals to Non-Goals. Otherwise someone will dig >>> >>> up >>> >>> one of these and say "Spark's developers have officially rejected X, >>> >>> which >>> >>> our awesome system has". >>> >>> >>> >>> - For user-facing stuff, I think you need a section on API. Virtually >>> >>> all >>> >>> other *IPs I've seen have that. >>> >>> >>> >>> - I'm still not sure why the strategy section is needed if the >>> >>> purpose is >>> >>> to define user-facing behavior -- unless this is the strategy for >>> >>> setting >>> >>> the goals or for defining the API. That sounds squarely like a design >>> >>> doc >>> >>> issue. In some sense, who cares whether the proposal is technically >>> >>> feasible >>> >>> right now? If it's infeasible, that will be discovered later during >>> >>> design >>> >>> and implementation. Same thing with rejected strategies -- listing >>> >>> some >>> >>> of >>> >>> those is definitely useful sometimes, but if you make this a >>> >>> *required* >>> >>> section, people are just going to fill it in with bogus stuff (I've >>> >>> seen >>> >>> this happen before). >>> >>> >>> >>> Matei >>> >>> >>> > >>> >>> > On Oct 9, 2016, at 2:14 PM, Cody Koeninger <[hidden email]> wrote: >>> >>> > >>> >>> > So to focus the discussion on the specific strategy I'm suggesting, >>> >>> > documented at >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-improvement-proposals.md >>> >>> > >>> >>> > "Goals: What must this allow people to do, that they can't >>> >>> > currently?" >>> >>> > >>> >>> > Is it unclear that this is focusing specifically on people-visible >>> >>> > behavior? >>> >>> > >>> >>> > Rejected goals - are important because otherwise people keep >>> >>> > trying >>> >>> > to argue about scope. Of course you can change things later with a >>> >>> > different SIP and different vote, the point is to focus. >>> >>> > >>> >>> > Use cases - are something that people are going to bring up in >>> >>> > discussion. If they aren't clearly documented as a goal ("This >>> >>> > must >>> >>> > allow me to connect using SSL"), they should be added. >>> >>> > >>> >>> > Internal architecture - if the people who need specific behavior >>> >>> > are >>> >>> > implementers of other parts of the system, that's fine. >>> >>> > >>> >>> > Rejected strategies - If you have none of these, you have no >>> >>> > evidence >>> >>> > that the proponent didn't just go with the first thing they had in >>> >>> > mind (or have already implemented), which is a big problem >>> >>> > currently. >>> >>> > Approval isn't binding as to specifics of implementation, so these >>> >>> > aren't handcuffs. The goals are the contract, the strategy is >>> >>> > evidence that contract can actually be met. >>> >>> > >>> >>> > Design docs - I'm not touching design docs. The markdown file I >>> >>> > linked specifically says of the strategy section "This is not a >>> >>> > full >>> >>> > design document." Is this unclear? Design docs can be worked on >>> >>> > obviously, but that's not what I'm concerned with here. >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > On Sun, Oct 9, 2016 at 2:34 PM, Matei Zaharia <[hidden email]> >>> >>> > wrote: >>> >>> >> Hi Cody, >>> >>> >> >>> >>> >> I think this would be a lot more concrete if we had a more >>> >>> >> detailed >>> >>> >> template >>> >>> >> for SIPs. Right now, it's not super clear what's in scope -- e.g. >>> >>> >> are >>> >>> >> they >>> >>> >> a way to solicit feedback on the user-facing behavior or on the >>> >>> >> internals? >>> >>> >> "Goals" can cover both things. I've been thinking of SIPs more as >>> >>> >> Product >>> >>> >> Requirements Docs (PRDs), which focus on *what* a code change >>> >>> >> should >>> >>> >> do >>> >>> >> as >>> >>> >> opposed to how. >>> >>> >> >>> >>> >> In particular, here are some things that you may or may not >>> >>> >> consider >>> >>> >> in >>> >>> >> scope for SIPs: >>> >>> >> >>> >>> >> - Goals and non-goals: This is definitely in scope, and IMO should >>> >>> >> focus on >>> >>> >> user-visible behavior (e.g. "system supports SQL window functions" >>> >>> >> or >>> >>> >> "system continues working if one node fails"). BTW I wouldn't say >>> >>> >> "rejected >>> >>> >> goals" because some of them might become goals later, so we're not >>> >>> >> definitively rejecting them. >>> >>> >> >>> >>> >> - Public API: Probably should be included in most SIPs unless it's >>> >>> >> too >>> >>> >> large >>> >>> >> to fully specify then (e.g. "let's add an ML library"). >>> >>> >> >>> >>> >> - Use cases: I usually find this very useful in PRDs to better >>> >>> >> communicate >>> >>> >> the goals. >>> >>> >> >>> >>> >> - Internal architecture: This is usually *not* a thing users can >>> >>> >> easily >>> >>> >> comment on and it sounds more like a design doc item. Of course >>> >>> >> it's >>> >>> >> important to show that the SIP is feasible to implement. One >>> >>> >> exception, >>> >>> >> however, is that I think we'll have some SIPs primarily on >>> >>> >> internals >>> >>> >> (e.g. >>> >>> >> if somebody wants to refactor Spark's query optimizer or >>> >>> >> something). >>> >>> >> >>> >>> >> - Rejected strategies: I personally wouldn't put this, because >>> >>> >> what's >>> >>> >> the >>> >>> >> point of voting to reject a strategy before you've really begun >>> >>> >> designing >>> >>> >> and implementing something? What if you discover that the strategy >>> >>> >> is >>> >>> >> actually better when you start doing stuff? >>> >>> >> >>> >>> >> At a super high level, it depends on whether you want the SIPs to >>> >>> >> be >>> >>> >> PRDs >>> >>> >> for getting some quick feedback on the goals of a feature before >>> >>> >> it is >>> >>> >> designed, or something more like full-fledged design docs (just a >>> >>> >> more >>> >>> >> visible design doc for bigger changes). I looked at Kafka's KIPs, >>> >>> >> and >>> >>> >> they >>> >>> >> actually seem to be more like design docs. This can work too but >>> >>> >> it >>> >>> >> does >>> >>> >> require more work from the proposer and it can lead to the same >>> >>> >> problems you >>> >>> >> mentioned with people already having a design and implementation >>> >>> >> in >>> >>> >> mind. >>> >>> >> >>> >>> >> Basically, the question is, are you trying to iterate faster on >>> >>> >> design >>> >>> >> by >>> >>> >> adding a step for user feedback earlier? Or are you just trying to >>> >>> >> make >>> >>> >> design docs for key features more visible (and their approval more >>> >>> >> formal)? >>> >>> >> >>> >>> >> BTW note that in either case, I'd like to have a template for >>> >>> >> design >>> >>> >> docs >>> >>> >> too, which should also include goals. I think that would've >>> >>> >> avoided >>> >>> >> some of >>> >>> >> the issues you brought up. >>> >>> >> >>> >>> >> Matei >>> >>> >> >>> >>> >> On Oct 9, 2016, at 10:40 AM, Cody Koeninger <[hidden email]> >>> >>> >> wrote: >>> >>> >> >>> >>> >> Here's my specific proposal (meta-proposal?) >>> >>> >> >>> >>> >> Spark Improvement Proposals (SIP) >>> >>> >> >>> >>> >> >>> >>> >> Background: >>> >>> >> >>> >>> >> The current problem is that design and implementation of large >>> >>> >> features >>> >>> >> are >>> >>> >> often done in private, before soliciting user feedback. >>> >>> >> >>> >>> >> When feedback is solicited, it is often as to detailed design >>> >>> >> specifics, not >>> >>> >> focused on goals. >>> >>> >> >>> >>> >> When implementation does take place after design, there is often >>> >>> >> disagreement as to what goals are or are not in scope. >>> >>> >> >>> >>> >> This results in commits that don't fully meet user needs. >>> >>> >> >>> >>> >> >>> >>> >> Goals: >>> >>> >> >>> >>> >> - Ensure user, contributor, and committer goals are clearly >>> >>> >> identified >>> >>> >> and >>> >>> >> agreed upon, before implementation takes place. >>> >>> >> >>> >>> >> - Ensure that a technically feasible strategy is chosen that is >>> >>> >> likely >>> >>> >> to >>> >>> >> meet the goals. >>> >>> >> >>> >>> >> >>> >>> >> Rejected Goals: >>> >>> >> >>> >>> >> - SIPs are not for detailed design. Design by committee doesn't >>> >>> >> work. >>> >>> >> >>> >>> >> - SIPs are not for every change. We dont need that much process. >>> >>> >> >>> >>> >> >>> >>> >> Strategy: >>> >>> >> >>> >>> >> My suggestion is outlined as a Spark Improvement Proposal process >>> >>> >> documented >>> >>> >> at >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-improvement-proposals.md >>> >>> >> >>> >>> >> Specifics of Jira manipulation are an implementation detail we can >>> >>> >> figure >>> >>> >> out. >>> >>> >> >>> >>> >> I'm suggesting voting; the need here is for a _clear_ outcome. >>> >>> >> >>> >>> >> >>> >>> >> Rejected Strategies: >>> >>> >> >>> >>> >> Having someone who understands the problem implement it first >>> >>> >> works, >>> >>> >> but >>> >>> >> only if significant iteration after user feedback is allowed. >>> >>> >> >>> >>> >> Historically this has been problematic due to pressure to limit >>> >>> >> public >>> >>> >> api >>> >>> >> changes. >>> >>> >> >>> >>> >> >>> >>> >> On Fri, Oct 7, 2016 at 5:16 PM, Reynold Xin <[hidden email]> >>> >>> >> wrote: >>> >>> >>> >>> >>> >>> Alright looks like there are quite a bit of support. We should >>> >>> >>> wait >>> >>> >>> to >>> >>> >>> hear from more people too. >>> >>> >>> >>> >>> >>> To push this forward, Cody and I will be working together in the >>> >>> >>> next >>> >>> >>> couple of weeks to come up with a concrete, detailed proposal on >>> >>> >>> what >>> >>> >>> this >>> >>> >>> entails, and then we can discuss this the specific proposal as >>> >>> >>> well. >>> >>> >>> >>> >>> >>> >>> >>> >>> On Fri, Oct 7, 2016 at 2:29 PM, Cody Koeninger <[hidden email]> >>> >>> >>> wrote: >>> >>> >>>> >>> >>> >>>> Yeah, in case it wasn't clear, I was talking about SIPs for >>> >>> >>>> major >>> >>> >>>> user-facing or cross-cutting changes, not minor feature adds. >>> >>> >>>> >>> >>> >>>> On Fri, Oct 7, 2016 at 3:58 PM, Stavros Kontopoulos >>> >>> >>>> <[hidden email]> wrote: >>> >>> >>>>> >>> >>> >>>>> +1 to the SIP label as long as it does not slow down things and >>> >>> >>>>> it >>> >>> >>>>> targets optimizing efforts, coordination etc. For example >>> >>> >>>>> really >>> >>> >>>>> small >>> >>> >>>>> features should not need to go through this process (assuming >>> >>> >>>>> they >>> >>> >>>>> dont >>> >>> >>>>> touch public interfaces) or re-factorings and hope it will be >>> >>> >>>>> kept >>> >>> >>>>> this >>> >>> >>>>> way. So as a guideline doc should be provided, like in the KIP >>> >>> >>>>> case. >>> >>> >>>>> >>> >>> >>>>> IMHO so far aside from tagging things and linking them >>> >>> >>>>> elsewhere >>> >>> >>>>> simply >>> >>> >>>>> having design docs and prototypes implementations in PRs is not >>> >>> >>>>> something >>> >>> >>>>> that has not worked so far. What is really a pain in many >>> >>> >>>>> projects >>> >>> >>>>> out there >>> >>> >>>>> is discontinuity in progress of PRs, missing features, slow >>> >>> >>>>> reviews >>> >>> >>>>> which is >>> >>> >>>>> understandable to some extent... it is not only about Spark but >>> >>> >>>>> things can >>> >>> >>>>> be improved for sure for this project in particular as already >>> >>> >>>>> stated. >>> >>> >>>>> >>> >>> >>>>> On Fri, Oct 7, 2016 at 11:14 PM, Cody Koeninger <[hidden >>> >>> >>>>> email]> >>> >>> >>>>> wrote: >>> >>> >>>>>> >>> >>> >>>>>> +1 to adding an SIP label and linking it from the website. I >>> >>> >>>>>> think >>> >>> >>>>>> it >>> >>> >>>>>> needs >>> >>> >>>>>> >>> >>> >>>>>> - template that focuses it towards soliciting user goals / non >>> >>> >>>>>> goals >>> >>> >>>>>> - clear resolution as to which strategy was chosen to pursue. >>> >>> >>>>>> I'd >>> >>> >>>>>> recommend a vote. >>> >>> >>>>>> >>> >>> >>>>>> Matei asked me to clarify what I meant by changing interfaces, >>> >>> >>>>>> I >>> >>> >>>>>> think >>> >>> >>>>>> it's directly relevant to the SIP idea so I'll clarify here, >>> >>> >>>>>> and >>> >>> >>>>>> split >>> >>> >>>>>> a thread for the other discussion per Nicholas' request. >>> >>> >>>>>> >>> >>> >>>>>> I meant changing public user interfaces. I think the first >>> >>> >>>>>> design >>> >>> >>>>>> is >>> >>> >>>>>> unlikely to be right, because it's done at a time when you >>> >>> >>>>>> have >>> >>> >>>>>> the >>> >>> >>>>>> least information. As a user, I find it considerably more >>> >>> >>>>>> frustrating >>> >>> >>>>>> to be unable to use a tool to get my job done, than I do >>> >>> >>>>>> having to >>> >>> >>>>>> make minor changes to my code in order to take advantage of >>> >>> >>>>>> features. >>> >>> >>>>>> I've seen committers be seriously reluctant to allow changes >>> >>> >>>>>> to >>> >>> >>>>>> @experimental code that are needed in order for it to really >>> >>> >>>>>> work >>> >>> >>>>>> right. You need to be able to iterate, and if people on both >>> >>> >>>>>> sides >>> >>> >>>>>> of >>> >>> >>>>>> the fence aren't going to respect that some newer apis are >>> >>> >>>>>> subject >>> >>> >>>>>> to >>> >>> >>>>>> change, then why even mark them as such? >>> >>> >>>>>> >>> >>> >>>>>> Ideally a finished SIP should give me a checklist of things >>> >>> >>>>>> that >>> >>> >>>>>> an >>> >>> >>>>>> implementation must do, and things that it doesn't need to do. >>> >>> >>>>>> Contributors/committers should be seriously discouraged from >>> >>> >>>>>> putting >>> >>> >>>>>> out a version 0.1 that doesn't have at least a prototype >>> >>> >>>>>> implementation of all those things, especially if they're then >>> >>> >>>>>> going >>> >>> >>>>>> to argue against interface changes necessary to get the the >>> >>> >>>>>> rest >>> >>> >>>>>> of >>> >>> >>>>>> the things done in the 0.2 version. >>> >>> >>>>>> >>> >>> >>>>>> >>> >>> >>>>>> On Fri, Oct 7, 2016 at 2:18 PM, Reynold Xin <[hidden email]> >>> >>> >>>>>> wrote: >>> >>> >>>>>>> I like the lightweight proposal to add a SIP label. >>> >>> >>>>>>> >>> >>> >>>>>>> During Spark 2.0 development, Tom (Graves) and I suggested >>> >>> >>>>>>> using >>> >>> >>>>>>> wiki >>> >>> >>>>>>> to >>> >>> >>>>>>> track the list of major changes, but that never really >>> >>> >>>>>>> materialized >>> >>> >>>>>>> due to >>> >>> >>>>>>> the overhead. Adding a SIP label on major JIRAs and then link >>> >>> >>>>>>> to >>> >>> >>>>>>> them >>> >>> >>>>>>> prominently on the Spark website makes a lot of sense. >>> >>> >>>>>>> >>> >>> >>>>>>> >>> >>> >>>>>>> On Fri, Oct 7, 2016 at 10:50 AM, Matei Zaharia >>> >>> >>>>>>> <[hidden email]> >>> >>> >>>>>>> wrote: >>> >>> >>>>>>>> >>> >>> >>>>>>>> For the improvement proposals, I think one major point was >>> >>> >>>>>>>> to >>> >>> >>>>>>>> make >>> >>> >>>>>>>> them >>> >>> >>>>>>>> really visible to users who are not contributors, so we >>> >>> >>>>>>>> should >>> >>> >>>>>>>> do >>> >>> >>>>>>>> more than >>> >>> >>>>>>>> sending stuff to dev@. One very lightweight idea is to have >>> >>> >>>>>>>> a >>> >>> >>>>>>>> new >>> >>> >>>>>>>> type of >>> >>> >>>>>>>> JIRA called a SIP and have a link to a filter that shows all >>> >>> >>>>>>>> such >>> >>> >>>>>>>> JIRAs from >>> >>> >>>>>>>> http://spark.apache.org. I also like the idea of SIP and >>> >>> >>>>>>>> design >>> >>> >>>>>>>> doc >>> >>> >>>>>>>> templates (in fact many projects have them). >>> >>> >>>>>>>> >>> >>> >>>>>>>> Matei >>> >>> >>>>>>>> >>> >>> >>>>>>>> On Oct 7, 2016, at 10:38 AM, Reynold Xin <[hidden email]> >>> >>> >>>>>>>> wrote: >>> >>> >>>>>>>> >>> >>> >>>>>>>> I called Cody last night and talked about some of the topics >>> >>> >>>>>>>> in >>> >>> >>>>>>>> his >>> >>> >>>>>>>> email. >>> >>> >>>>>>>> It became clear to me Cody genuinely cares about the >>> >>> >>>>>>>> project. >>> >>> >>>>>>>> >>> >>> >>>>>>>> Some of the frustrations come from the success of the >>> >>> >>>>>>>> project >>> >>> >>>>>>>> itself >>> >>> >>>>>>>> becoming very "hot", and it is difficult to get clarity from >>> >>> >>>>>>>> people >>> >>> >>>>>>>> who >>> >>> >>>>>>>> don't dedicate all their time to Spark. In fact, it is in >>> >>> >>>>>>>> some >>> >>> >>>>>>>> ways >>> >>> >>>>>>>> similar >>> >>> >>>>>>>> to scaling an engineering team in a successful startup: old >>> >>> >>>>>>>> processes that >>> >>> >>>>>>>> worked well might not work so well when it gets to a certain >>> >>> >>>>>>>> size, >>> >>> >>>>>>>> cultures >>> >>> >>>>>>>> can get diluted, building culture vs building process, etc. >>> >>> >>>>>>>> >>> >>> >>>>>>>> I also really like to have a more visible process for larger >>> >>> >>>>>>>> changes, >>> >>> >>>>>>>> especially major user facing API changes. Historically we >>> >>> >>>>>>>> upload >>> >>> >>>>>>>> design docs >>> >>> >>>>>>>> for major changes, but it is not always consistent and >>> >>> >>>>>>>> difficult >>> >>> >>>>>>>> to >>> >>> >>>>>>>> quality >>> >>> >>>>>>>> of the docs, due to the volunteering nature of the >>> >>> >>>>>>>> organization. >>> >>> >>>>>>>> >>> >>> >>>>>>>> Some of the more concrete ideas we discussed focus on >>> >>> >>>>>>>> building a >>> >>> >>>>>>>> culture >>> >>> >>>>>>>> to improve clarity: >>> >>> >>>>>>>> >>> >>> >>>>>>>> - Process: Large changes should have design docs posted on >>> >>> >>>>>>>> JIRA. >>> >>> >>>>>>>> One >>> >>> >>>>>>>> thing >>> >>> >>>>>>>> Cody and I didn't discuss but an idea that just came to me >>> >>> >>>>>>>> is we >>> >>> >>>>>>>> should >>> >>> >>>>>>>> create a design doc template for the project and ask >>> >>> >>>>>>>> everybody >>> >>> >>>>>>>> to >>> >>> >>>>>>>> follow. >>> >>> >>>>>>>> The design doc template should also explicitly list goals >>> >>> >>>>>>>> and >>> >>> >>>>>>>> non-goals, to >>> >>> >>>>>>>> make design doc more consistent. >>> >>> >>>>>>>> >>> >>> >>>>>>>> - Process: Email dev@ to solicit feedback. We have some this >>> >>> >>>>>>>> with >>> >>> >>>>>>>> some >>> >>> >>>>>>>> changes, but again very inconsistent. Just posting something >>> >>> >>>>>>>> on >>> >>> >>>>>>>> JIRA >>> >>> >>>>>>>> isn't >>> >>> >>>>>>>> sufficient, because there are simply too many JIRAs and the >>> >>> >>>>>>>> signal >>> >>> >>>>>>>> get lost >>> >>> >>>>>>>> in the noise. While this is generally impossible to enforce >>> >>> >>>>>>>> because >>> >>> >>>>>>>> we can't >>> >>> >>>>>>>> force all volunteers to conform to a process (or they might >>> >>> >>>>>>>> not >>> >>> >>>>>>>> even >>> >>> >>>>>>>> be >>> >>> >>>>>>>> aware of this), those who are more familiar with the >>> >>> >>>>>>>> project >>> >>> >>>>>>>> can >>> >>> >>>>>>>> help by >>> >>> >>>>>>>> emailing the dev@ when they see something that hasn't been. >>> >>> >>>>>>>> >>> >>> >>>>>>>> - Culture: The design doc author(s) should be open to >>> >>> >>>>>>>> feedback. >>> >>> >>>>>>>> A >>> >>> >>>>>>>> design >>> >>> >>>>>>>> doc should serve as the base for discussion and is by no >>> >>> >>>>>>>> means >>> >>> >>>>>>>> the >>> >>> >>>>>>>> final >>> >>> >>>>>>>> design. Of course, this does not mean the author has to >>> >>> >>>>>>>> accept >>> >>> >>>>>>>> every >>> >>> >>>>>>>> feedback. They should also be comfortable accepting / >>> >>> >>>>>>>> rejecting >>> >>> >>>>>>>> ideas on >>> >>> >>>>>>>> technical grounds. >>> >>> >>>>>>>> >>> >>> >>>>>>>> - Process / Culture: For major ongoing projects, it can be >>> >>> >>>>>>>> useful >>> >>> >>>>>>>> to >>> >>> >>>>>>>> have >>> >>> >>>>>>>> some monthly Google hangouts that are open to the world. I >>> >>> >>>>>>>> am >>> >>> >>>>>>>> actually not >>> >>> >>>>>>>> sure how well this will work, because of the volunteering >>> >>> >>>>>>>> nature >>> >>> >>>>>>>> and >>> >>> >>>>>>>> we need >>> >>> >>>>>>>> to adjust for timezones for people across the globe, but it >>> >>> >>>>>>>> seems >>> >>> >>>>>>>> worth >>> >>> >>>>>>>> trying. >>> >>> >>>>>>>> >>> >>> >>>>>>>> - Culture: Contributors (including committers) should be >>> >>> >>>>>>>> more >>> >>> >>>>>>>> direct >>> >>> >>>>>>>> in >>> >>> >>>>>>>> setting expectations, including whether they are working on >>> >>> >>>>>>>> a >>> >>> >>>>>>>> specific >>> >>> >>>>>>>> issue, whether they will be working on a specific issue, and >>> >>> >>>>>>>> whether >>> >>> >>>>>>>> an >>> >>> >>>>>>>> issue or pr or jira should be rejected. Most people I know >>> >>> >>>>>>>> in >>> >>> >>>>>>>> this >>> >>> >>>>>>>> community >>> >>> >>>>>>>> are nice and don't enjoy telling other people no, but it is >>> >>> >>>>>>>> often >>> >>> >>>>>>>> more >>> >>> >>>>>>>> annoying to a contributor to not know anything than getting >>> >>> >>>>>>>> a >>> >>> >>>>>>>> no. >>> >>> >>>>>>>> >>> >>> >>>>>>>> >>> >>> >>>>>>>> On Fri, Oct 7, 2016 at 10:03 AM, Matei Zaharia >>> >>> >>>>>>>> <[hidden email]> >>> >>> >>>>>>>> wrote: >>> >>> >>>>>>>>> >>> >>> >>>>>>>>> >>> >>> >>>>>>>>> Love the idea of a more visible "Spark Improvement >>> >>> >>>>>>>>> Proposal" >>> >>> >>>>>>>>> process that >>> >>> >>>>>>>>> solicits user input on new APIs. For what it's worth, I >>> >>> >>>>>>>>> don't >>> >>> >>>>>>>>> think >>> >>> >>>>>>>>> committers are trying to minimize their own work -- every >>> >>> >>>>>>>>> committer >>> >>> >>>>>>>>> cares >>> >>> >>>>>>>>> about making the software useful for users. However, it is >>> >>> >>>>>>>>> always >>> >>> >>>>>>>>> hard to >>> >>> >>>>>>>>> get user input and so it helps to have this kind of >>> >>> >>>>>>>>> process. >>> >>> >>>>>>>>> I've >>> >>> >>>>>>>>> certainly >>> >>> >>>>>>>>> looked at the *IPs a lot in other software I use just to >>> >>> >>>>>>>>> see >>> >>> >>>>>>>>> the >>> >>> >>>>>>>>> biggest >>> >>> >>>>>>>>> things on the roadmap. >>> >>> >>>>>>>>> >>> >>> >>>>>>>>> When you're talking about "changing interfaces", are you >>> >>> >>>>>>>>> talking >>> >>> >>>>>>>>> about >>> >>> >>>>>>>>> public or internal APIs? I do think many people hate >>> >>> >>>>>>>>> changing >>> >>> >>>>>>>>> public APIs >>> >>> >>>>>>>>> and I actually think that's for the best of the project. >>> >>> >>>>>>>>> That's >>> >>> >>>>>>>>> a >>> >>> >>>>>>>>> technical >>> >>> >>>>>>>>> debate, but basically, the worst thing when you're using a >>> >>> >>>>>>>>> piece >>> >>> >>>>>>>>> of >>> >>> >>>>>>>>> software >>> >>> >>>>>>>>> is that the developers constantly ask you to rewrite your >>> >>> >>>>>>>>> app >>> >>> >>>>>>>>> to >>> >>> >>>>>>>>> update to a >>> >>> >>>>>>>>> new version (and thus benefit from bug fixes, etc). Cue >>> >>> >>>>>>>>> anyone >>> >>> >>>>>>>>> who's used >>> >>> >>>>>>>>> Protobuf, or Guava. The "let's get everyone to change their >>> >>> >>>>>>>>> code >>> >>> >>>>>>>>> this >>> >>> >>>>>>>>> release" model works well within a single large company, >>> >>> >>>>>>>>> but >>> >>> >>>>>>>>> doesn't work >>> >>> >>>>>>>>> well for a community, which is why nearly all *very* widely >>> >>> >>>>>>>>> used >>> >>> >>>>>>>>> programming >>> >>> >>>>>>>>> interfaces (I'm talking things like Java standard library, >>> >>> >>>>>>>>> Windows >>> >>> >>>>>>>>> API, etc) >>> >>> >>>>>>>>> almost *never* break backwards compatibility. All this is >>> >>> >>>>>>>>> done >>> >>> >>>>>>>>> within reason >>> >>> >>>>>>>>> though, e.g. we do change things in major releases (2.x, >>> >>> >>>>>>>>> 3.x, >>> >>> >>>>>>>>> etc). >>> >>> >>>>>>>> >>> >>> >>>>>>>> >>> >>> >>>>>>>> >>> >>> >>>>>>>> >>> >>> >>>>>>> >>> >>> >>>>>> >>> >>> >>>>>> >>> >>> >>>>>> >>> >>> >>>>>> >>> >>> >>>>>> --------------------------------------------------------------------- >>> >>> >>>>>> To unsubscribe e-mail: [hidden email] >>> >>> >>>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> -- >>> >>> >>>>> Stavros Kontopoulos >>> >>> >>>>> Senior Software Engineer >>> >>> >>>>> Lightbend, Inc. >>> >>> >>>>> p: +30 6977967274 >>> >>> >>>>> e: [hidden email] >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>> >>> >>> >>> >>> >>> >> >>> >>> >> >>> >>> >>> >> >>> > >>> > >>> > --------------------------------------------------------------------- >>> > To unsubscribe e-mail: [hidden email] >>> > >>> > >>> > ________________________________ >>> > >>> > If you reply to this email, your message will be added to the >>> > discussion >>> > below: >>> > >>> > >>> > http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Improvement-Proposals-tp19268p19359.html >>> > >>> > To start a new topic under Apache Spark Developers List, email [hidden >>> > email] >>> > To unsubscribe from Apache Spark Developers List, click here. >>> > NAML >>> > >>> > >>> > ________________________________ >>> > View this message in context: RE: Spark Improvement Proposals >>> > Sent from the Apache Spark Developers List mailing list archive at >>> > Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >> >> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix > > > > > -- > Ryan Blue > Software Engineer > Netflix --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
