+1 to votes to approve proposals. I agree that proposals should have an official mechanism to be accepted, and a vote is an established means of doing that well. I like that it includes a period to review the proposal and I think proposals should have been discussed enough ahead of a vote to survive the possibility of a veto.
I also like the names that are short and (mostly) unique, like SEP. Where I disagree is with the requirement that a committer must formally propose an enhancement. I don't see the value of restricting this: if someone has the will to write up a proposal then they should be encouraged to do so and start a discussion about it. Even if there is a political reality as Cody says, what is the value of codifying that in our process? I think restricting who can submit proposals would only undermine them by pushing contributors out. Maybe I'm missing something here? rb On Mon, Oct 10, 2016 at 7:41 AM, Cody Koeninger <c...@koeninger.org> wrote: > Yes, users suggesting SIPs is a good thing and is explicitly called > out in the linked document under the Who? section. Formally proposing > them, not so much, because of the political realities. > > Yes, implementation strategy definitely affects goals. There are all > kinds of examples of this, I'll pick one that's my fault so as to > avoid sounding like I'm blaming: > > When I implemented the Kafka DStream, one of my (not explicitly agreed > upon by the community) goals was to make sure people could use the > Dstream with however they were already using Kafka at work. The lack > of explicit agreement on that goal led to all kinds of fighting with > committers, that could have been avoided. The lack of explicit > up-front strategy discussion led to the DStream not really working > with compacted topics. I knew about compacted topics, but don't have > a use for them, so had a blind spot there. If there was explicit > up-front discussion that my strategy was "assume that batches can be > defined on the driver solely by beginning and ending offsets", there's > a greater chance that a user would have seen that and said, "hey, what > about non-contiguous offsets in a compacted topic". > > This kind of thing is only going to happen smoothly if we have a > lightweight user-visible process with clear outcomes. > > On Mon, Oct 10, 2016 at 1:34 AM, assaf.mendelson > <assaf.mendel...@rsa.com> wrote: > > I agree with most of what Cody said. > > > > Two things: > > > > First we can always have other people suggest SIPs but mark them as > > “unreviewed” and have committers basically move them forward. The > problem is > > that writing a good document takes time. This way we can leverage non > > committers to do some of this work (it is just another way to > contribute). > > > > > > > > As for strategy, in many cases implementation strategy can affect the > goals. > > I will give a small example: In the current structured streaming > strategy, > > we group by the time to achieve a sliding window. This is definitely an > > implementation decision and not a goal. However, I can think of several > > aggregation functions which have the time inside their calculation > buffer. > > For example, let’s say we want to return a set of all distinct values. > One > > way to implement this would be to make the set into a map and have the > value > > contain the last time seen. Multiplying it across the groupby would cost > a > > lot in performance. So adding such a strategy would have a great effect > on > > the type of aggregations and their performance which does affect the > goal. > > Without adding the strategy, it is easy for whoever goes to the design > > document to not think about these cases. Furthermore, it might be decided > > that these cases are rare enough so that the strategy is still good > enough > > but how would we know it without user feedback? > > > > I believe this example is exactly what Cody was talking about. Since many > > times implementation strategies have a large effect on the goal, we > should > > have it discussed when discussing the goals. In addition, while it is > often > > easy to throw out completely infeasible goals, it is often much harder to > > figure out that the goals are unfeasible without fine tuning. > > > > > > > > > > > > Assaf. > > > > > > > > From: Cody Koeninger-2 [via Apache Spark Developers List] > > [mailto:ml-node+[hidden email]] > > Sent: Monday, October 10, 2016 2:25 AM > > To: Mendelson, Assaf > > Subject: Re: Spark Improvement Proposals > > > > > > > > Only committers should formally submit SIPs because in an apache > > project only commiters have explicit political power. If a user can't > > find a commiter willing to sponsor an SIP idea, they have no way to > > get the idea passed in any case. If I can't find a committer to > > sponsor this meta-SIP idea, I'm out of luck. > > > > I do not believe unrealistic goals can be found solely by inspection. > > We've managed to ignore unrealistic goals even after implementation! > > Focusing on APIs can allow people to think they've solved something, > > when there's really no way of implementing that API while meeting the > > goals. Rapid iteration is clearly the best way to address this, but > > we've already talked about why that hasn't really worked. If adding a > > non-binding API section to the template is important to you, I'm not > > against it, but I don't think it's sufficient. > > > > On your PRD vs design doc spectrum, I'm saying this is closer to a > > PRD. Clear agreement on goals is the most important thing and that's > > why it's the thing I want binding agreement on. But I cannot agree to > > goals unless I have enough minimal technical info to judge whether the > > goals are likely to actually be accomplished. > > > > > > > > On Sun, Oct 9, 2016 at 5:35 PM, Matei Zaharia <[hidden email]> wrote: > > > > > >> Well, I think there are a few things here that don't make sense. First, > >> why > >> should only committers submit SIPs? Development in the project should be > >> open to all contributors, whether they're committers or not. Second, I > >> think > >> unrealistic goals can be found just by inspecting the goals, and I'm not > >> super worried that we'll accept a lot of SIPs that are then infeasible > -- > >> we > >> can then submit new ones. But this depends on whether you want this > >> process > >> to be a "design doc lite", where people also agree on implementation > >> strategy, or just a way to agree on goals. This is what I asked earlier > >> about PRDs vs design docs (and I'm open to either one but I'd just like > >> clarity). Finally, both as a user and designer of software, I always > want > >> to > >> give feedback on APIs, so I'd really like a culture of having those > early. > >> People don't argue about prettiness when they discuss APIs, they argue > >> about > >> the core concepts to expose in order to meet various goals, and then > >> they're > >> stuck maintaining those for a long time. > >> > >> Matei > >> > >> On Oct 9, 2016, at 3:10 PM, Cody Koeninger <[hidden email]> wrote: > >> > >> Users instead of people, sure. Commiters and contributors are (or at > >> least > >> should be) a subset of users. > >> > >> Non goals, sure. I don't care what the name is, but we need to clearly > say > >> e.g. 'no we are not maintaining compatibility with XYZ right now'. > >> > >> API, what I care most about is whether it allows me to accomplish the > >> goals. > >> Arguing about how ugly or pretty it is can be saved for design/ > >> implementation imho. > >> > >> Strategy, this is necessary because otherwise goals can be out of line > >> with > >> reality. Don't propose goals you don't have at least some idea of how > to > >> implement. > >> > >> Rejected strategies, given that commiters are the only ones I'm saying > >> should formally submit SPARKLIs or SIPs, if they put junk in a required > >> section then slap them down for it and tell them to fix it. > >> > >> > >> On Oct 9, 2016 4:36 PM, "Matei Zaharia" <[hidden email]> wrote: > >>> > >>> Yup, this is the stuff that I found unclear. Thanks for clarifying > here, > >>> but we should also clarify it in the writeup. In particular: > >>> > >>> - Goals needs to be about user-facing behavior ("people" is broad) > >>> > >>> - I'd rename Rejected Goals to Non-Goals. Otherwise someone will dig up > >>> one of these and say "Spark's developers have officially rejected X, > >>> which > >>> our awesome system has". > >>> > >>> - For user-facing stuff, I think you need a section on API. Virtually > all > >>> other *IPs I've seen have that. > >>> > >>> - I'm still not sure why the strategy section is needed if the purpose > is > >>> to define user-facing behavior -- unless this is the strategy for > setting > >>> the goals or for defining the API. That sounds squarely like a design > doc > >>> issue. In some sense, who cares whether the proposal is technically > >>> feasible > >>> right now? If it's infeasible, that will be discovered later during > >>> design > >>> and implementation. Same thing with rejected strategies -- listing some > >>> of > >>> those is definitely useful sometimes, but if you make this a *required* > >>> section, people are just going to fill it in with bogus stuff (I've > seen > >>> this happen before). > >>> > >>> Matei > >>> > > > >>> > On Oct 9, 2016, at 2:14 PM, Cody Koeninger <[hidden email]> wrote: > >>> > > >>> > So to focus the discussion on the specific strategy I'm suggesting, > >>> > documented at > >>> > > >>> > > >>> > > >>> > https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark- > improvement-proposals.md > >>> > > >>> > "Goals: What must this allow people to do, that they can't > currently?" > >>> > > >>> > Is it unclear that this is focusing specifically on people-visible > >>> > behavior? > >>> > > >>> > Rejected goals - are important because otherwise people keep trying > >>> > to argue about scope. Of course you can change things later with a > >>> > different SIP and different vote, the point is to focus. > >>> > > >>> > Use cases - are something that people are going to bring up in > >>> > discussion. If they aren't clearly documented as a goal ("This must > >>> > allow me to connect using SSL"), they should be added. > >>> > > >>> > Internal architecture - if the people who need specific behavior are > >>> > implementers of other parts of the system, that's fine. > >>> > > >>> > Rejected strategies - If you have none of these, you have no evidence > >>> > that the proponent didn't just go with the first thing they had in > >>> > mind (or have already implemented), which is a big problem currently. > >>> > Approval isn't binding as to specifics of implementation, so these > >>> > aren't handcuffs. The goals are the contract, the strategy is > >>> > evidence that contract can actually be met. > >>> > > >>> > Design docs - I'm not touching design docs. The markdown file I > >>> > linked specifically says of the strategy section "This is not a full > >>> > design document." Is this unclear? Design docs can be worked on > >>> > obviously, but that's not what I'm concerned with here. > >>> > > >>> > > >>> > > >>> > > >>> > On Sun, Oct 9, 2016 at 2:34 PM, Matei Zaharia <[hidden email]> > >>> > wrote: > >>> >> Hi Cody, > >>> >> > >>> >> I think this would be a lot more concrete if we had a more detailed > >>> >> template > >>> >> for SIPs. Right now, it's not super clear what's in scope -- e.g. > are > >>> >> they > >>> >> a way to solicit feedback on the user-facing behavior or on the > >>> >> internals? > >>> >> "Goals" can cover both things. I've been thinking of SIPs more as > >>> >> Product > >>> >> Requirements Docs (PRDs), which focus on *what* a code change should > >>> >> do > >>> >> as > >>> >> opposed to how. > >>> >> > >>> >> In particular, here are some things that you may or may not consider > >>> >> in > >>> >> scope for SIPs: > >>> >> > >>> >> - Goals and non-goals: This is definitely in scope, and IMO should > >>> >> focus on > >>> >> user-visible behavior (e.g. "system supports SQL window functions" > or > >>> >> "system continues working if one node fails"). BTW I wouldn't say > >>> >> "rejected > >>> >> goals" because some of them might become goals later, so we're not > >>> >> definitively rejecting them. > >>> >> > >>> >> - Public API: Probably should be included in most SIPs unless it's > too > >>> >> large > >>> >> to fully specify then (e.g. "let's add an ML library"). > >>> >> > >>> >> - Use cases: I usually find this very useful in PRDs to better > >>> >> communicate > >>> >> the goals. > >>> >> > >>> >> - Internal architecture: This is usually *not* a thing users can > >>> >> easily > >>> >> comment on and it sounds more like a design doc item. Of course it's > >>> >> important to show that the SIP is feasible to implement. One > >>> >> exception, > >>> >> however, is that I think we'll have some SIPs primarily on internals > >>> >> (e.g. > >>> >> if somebody wants to refactor Spark's query optimizer or something). > >>> >> > >>> >> - Rejected strategies: I personally wouldn't put this, because > what's > >>> >> the > >>> >> point of voting to reject a strategy before you've really begun > >>> >> designing > >>> >> and implementing something? What if you discover that the strategy > is > >>> >> actually better when you start doing stuff? > >>> >> > >>> >> At a super high level, it depends on whether you want the SIPs to be > >>> >> PRDs > >>> >> for getting some quick feedback on the goals of a feature before it > is > >>> >> designed, or something more like full-fledged design docs (just a > more > >>> >> visible design doc for bigger changes). I looked at Kafka's KIPs, > and > >>> >> they > >>> >> actually seem to be more like design docs. This can work too but it > >>> >> does > >>> >> require more work from the proposer and it can lead to the same > >>> >> problems you > >>> >> mentioned with people already having a design and implementation in > >>> >> mind. > >>> >> > >>> >> Basically, the question is, are you trying to iterate faster on > design > >>> >> by > >>> >> adding a step for user feedback earlier? Or are you just trying to > >>> >> make > >>> >> design docs for key features more visible (and their approval more > >>> >> formal)? > >>> >> > >>> >> BTW note that in either case, I'd like to have a template for design > >>> >> docs > >>> >> too, which should also include goals. I think that would've avoided > >>> >> some of > >>> >> the issues you brought up. > >>> >> > >>> >> Matei > >>> >> > >>> >> On Oct 9, 2016, at 10:40 AM, Cody Koeninger <[hidden email]> wrote: > >>> >> > >>> >> Here's my specific proposal (meta-proposal?) > >>> >> > >>> >> Spark Improvement Proposals (SIP) > >>> >> > >>> >> > >>> >> Background: > >>> >> > >>> >> The current problem is that design and implementation of large > >>> >> features > >>> >> are > >>> >> often done in private, before soliciting user feedback. > >>> >> > >>> >> When feedback is solicited, it is often as to detailed design > >>> >> specifics, not > >>> >> focused on goals. > >>> >> > >>> >> When implementation does take place after design, there is often > >>> >> disagreement as to what goals are or are not in scope. > >>> >> > >>> >> This results in commits that don't fully meet user needs. > >>> >> > >>> >> > >>> >> Goals: > >>> >> > >>> >> - Ensure user, contributor, and committer goals are clearly > identified > >>> >> and > >>> >> agreed upon, before implementation takes place. > >>> >> > >>> >> - Ensure that a technically feasible strategy is chosen that is > likely > >>> >> to > >>> >> meet the goals. > >>> >> > >>> >> > >>> >> Rejected Goals: > >>> >> > >>> >> - SIPs are not for detailed design. Design by committee doesn't > work. > >>> >> > >>> >> - SIPs are not for every change. We dont need that much process. > >>> >> > >>> >> > >>> >> Strategy: > >>> >> > >>> >> My suggestion is outlined as a Spark Improvement Proposal process > >>> >> documented > >>> >> at > >>> >> > >>> >> > >>> >> > >>> >> https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark- > improvement-proposals.md > >>> >> > >>> >> Specifics of Jira manipulation are an implementation detail we can > >>> >> figure > >>> >> out. > >>> >> > >>> >> I'm suggesting voting; the need here is for a _clear_ outcome. > >>> >> > >>> >> > >>> >> Rejected Strategies: > >>> >> > >>> >> Having someone who understands the problem implement it first works, > >>> >> but > >>> >> only if significant iteration after user feedback is allowed. > >>> >> > >>> >> Historically this has been problematic due to pressure to limit > public > >>> >> api > >>> >> changes. > >>> >> > >>> >> > >>> >> On Fri, Oct 7, 2016 at 5:16 PM, Reynold Xin <[hidden email]> > >>> >> wrote: > >>> >>> > >>> >>> Alright looks like there are quite a bit of support. We should wait > >>> >>> to > >>> >>> hear from more people too. > >>> >>> > >>> >>> To push this forward, Cody and I will be working together in the > next > >>> >>> couple of weeks to come up with a concrete, detailed proposal on > what > >>> >>> this > >>> >>> entails, and then we can discuss this the specific proposal as > well. > >>> >>> > >>> >>> > >>> >>> On Fri, Oct 7, 2016 at 2:29 PM, Cody Koeninger <[hidden email]> > >>> >>> wrote: > >>> >>>> > >>> >>>> Yeah, in case it wasn't clear, I was talking about SIPs for major > >>> >>>> user-facing or cross-cutting changes, not minor feature adds. > >>> >>>> > >>> >>>> On Fri, Oct 7, 2016 at 3:58 PM, Stavros Kontopoulos > >>> >>>> <[hidden email]> wrote: > >>> >>>>> > >>> >>>>> +1 to the SIP label as long as it does not slow down things and > it > >>> >>>>> targets optimizing efforts, coordination etc. For example really > >>> >>>>> small > >>> >>>>> features should not need to go through this process (assuming > they > >>> >>>>> dont > >>> >>>>> touch public interfaces) or re-factorings and hope it will be > kept > >>> >>>>> this > >>> >>>>> way. So as a guideline doc should be provided, like in the KIP > >>> >>>>> case. > >>> >>>>> > >>> >>>>> IMHO so far aside from tagging things and linking them elsewhere > >>> >>>>> simply > >>> >>>>> having design docs and prototypes implementations in PRs is not > >>> >>>>> something > >>> >>>>> that has not worked so far. What is really a pain in many > projects > >>> >>>>> out there > >>> >>>>> is discontinuity in progress of PRs, missing features, slow > reviews > >>> >>>>> which is > >>> >>>>> understandable to some extent... it is not only about Spark but > >>> >>>>> things can > >>> >>>>> be improved for sure for this project in particular as already > >>> >>>>> stated. > >>> >>>>> > >>> >>>>> On Fri, Oct 7, 2016 at 11:14 PM, Cody Koeninger <[hidden email]> > >>> >>>>> wrote: > >>> >>>>>> > >>> >>>>>> +1 to adding an SIP label and linking it from the website. I > >>> >>>>>> think > >>> >>>>>> it > >>> >>>>>> needs > >>> >>>>>> > >>> >>>>>> - template that focuses it towards soliciting user goals / non > >>> >>>>>> goals > >>> >>>>>> - clear resolution as to which strategy was chosen to pursue. > I'd > >>> >>>>>> recommend a vote. > >>> >>>>>> > >>> >>>>>> Matei asked me to clarify what I meant by changing interfaces, I > >>> >>>>>> think > >>> >>>>>> it's directly relevant to the SIP idea so I'll clarify here, and > >>> >>>>>> split > >>> >>>>>> a thread for the other discussion per Nicholas' request. > >>> >>>>>> > >>> >>>>>> I meant changing public user interfaces. I think the first > design > >>> >>>>>> is > >>> >>>>>> unlikely to be right, because it's done at a time when you have > >>> >>>>>> the > >>> >>>>>> least information. As a user, I find it considerably more > >>> >>>>>> frustrating > >>> >>>>>> to be unable to use a tool to get my job done, than I do having > to > >>> >>>>>> make minor changes to my code in order to take advantage of > >>> >>>>>> features. > >>> >>>>>> I've seen committers be seriously reluctant to allow changes to > >>> >>>>>> @experimental code that are needed in order for it to really > work > >>> >>>>>> right. You need to be able to iterate, and if people on both > >>> >>>>>> sides > >>> >>>>>> of > >>> >>>>>> the fence aren't going to respect that some newer apis are > subject > >>> >>>>>> to > >>> >>>>>> change, then why even mark them as such? > >>> >>>>>> > >>> >>>>>> Ideally a finished SIP should give me a checklist of things that > >>> >>>>>> an > >>> >>>>>> implementation must do, and things that it doesn't need to do. > >>> >>>>>> Contributors/committers should be seriously discouraged from > >>> >>>>>> putting > >>> >>>>>> out a version 0.1 that doesn't have at least a prototype > >>> >>>>>> implementation of all those things, especially if they're then > >>> >>>>>> going > >>> >>>>>> to argue against interface changes necessary to get the the rest > >>> >>>>>> of > >>> >>>>>> the things done in the 0.2 version. > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> On Fri, Oct 7, 2016 at 2:18 PM, Reynold Xin <[hidden email]> > >>> >>>>>> wrote: > >>> >>>>>>> I like the lightweight proposal to add a SIP label. > >>> >>>>>>> > >>> >>>>>>> During Spark 2.0 development, Tom (Graves) and I suggested > using > >>> >>>>>>> wiki > >>> >>>>>>> to > >>> >>>>>>> track the list of major changes, but that never really > >>> >>>>>>> materialized > >>> >>>>>>> due to > >>> >>>>>>> the overhead. Adding a SIP label on major JIRAs and then link > to > >>> >>>>>>> them > >>> >>>>>>> prominently on the Spark website makes a lot of sense. > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> On Fri, Oct 7, 2016 at 10:50 AM, Matei Zaharia > >>> >>>>>>> <[hidden email]> > >>> >>>>>>> wrote: > >>> >>>>>>>> > >>> >>>>>>>> For the improvement proposals, I think one major point was to > >>> >>>>>>>> make > >>> >>>>>>>> them > >>> >>>>>>>> really visible to users who are not contributors, so we should > >>> >>>>>>>> do > >>> >>>>>>>> more than > >>> >>>>>>>> sending stuff to dev@. One very lightweight idea is to have a > >>> >>>>>>>> new > >>> >>>>>>>> type of > >>> >>>>>>>> JIRA called a SIP and have a link to a filter that shows all > >>> >>>>>>>> such > >>> >>>>>>>> JIRAs from > >>> >>>>>>>> http://spark.apache.org. I also like the idea of SIP and > design > >>> >>>>>>>> doc > >>> >>>>>>>> templates (in fact many projects have them). > >>> >>>>>>>> > >>> >>>>>>>> Matei > >>> >>>>>>>> > >>> >>>>>>>> On Oct 7, 2016, at 10:38 AM, Reynold Xin <[hidden email]> > >>> >>>>>>>> wrote: > >>> >>>>>>>> > >>> >>>>>>>> I called Cody last night and talked about some of the topics > in > >>> >>>>>>>> his > >>> >>>>>>>> email. > >>> >>>>>>>> It became clear to me Cody genuinely cares about the project. > >>> >>>>>>>> > >>> >>>>>>>> Some of the frustrations come from the success of the project > >>> >>>>>>>> itself > >>> >>>>>>>> becoming very "hot", and it is difficult to get clarity from > >>> >>>>>>>> people > >>> >>>>>>>> who > >>> >>>>>>>> don't dedicate all their time to Spark. In fact, it is in some > >>> >>>>>>>> ways > >>> >>>>>>>> similar > >>> >>>>>>>> to scaling an engineering team in a successful startup: old > >>> >>>>>>>> processes that > >>> >>>>>>>> worked well might not work so well when it gets to a certain > >>> >>>>>>>> size, > >>> >>>>>>>> cultures > >>> >>>>>>>> can get diluted, building culture vs building process, etc. > >>> >>>>>>>> > >>> >>>>>>>> I also really like to have a more visible process for larger > >>> >>>>>>>> changes, > >>> >>>>>>>> especially major user facing API changes. Historically we > upload > >>> >>>>>>>> design docs > >>> >>>>>>>> for major changes, but it is not always consistent and > difficult > >>> >>>>>>>> to > >>> >>>>>>>> quality > >>> >>>>>>>> of the docs, due to the volunteering nature of the > organization. > >>> >>>>>>>> > >>> >>>>>>>> Some of the more concrete ideas we discussed focus on > building a > >>> >>>>>>>> culture > >>> >>>>>>>> to improve clarity: > >>> >>>>>>>> > >>> >>>>>>>> - Process: Large changes should have design docs posted on > JIRA. > >>> >>>>>>>> One > >>> >>>>>>>> thing > >>> >>>>>>>> Cody and I didn't discuss but an idea that just came to me is > we > >>> >>>>>>>> should > >>> >>>>>>>> create a design doc template for the project and ask everybody > >>> >>>>>>>> to > >>> >>>>>>>> follow. > >>> >>>>>>>> The design doc template should also explicitly list goals and > >>> >>>>>>>> non-goals, to > >>> >>>>>>>> make design doc more consistent. > >>> >>>>>>>> > >>> >>>>>>>> - Process: Email dev@ to solicit feedback. We have some this > >>> >>>>>>>> with > >>> >>>>>>>> some > >>> >>>>>>>> changes, but again very inconsistent. Just posting something > on > >>> >>>>>>>> JIRA > >>> >>>>>>>> isn't > >>> >>>>>>>> sufficient, because there are simply too many JIRAs and the > >>> >>>>>>>> signal > >>> >>>>>>>> get lost > >>> >>>>>>>> in the noise. While this is generally impossible to enforce > >>> >>>>>>>> because > >>> >>>>>>>> we can't > >>> >>>>>>>> force all volunteers to conform to a process (or they might > not > >>> >>>>>>>> even > >>> >>>>>>>> be > >>> >>>>>>>> aware of this), those who are more familiar with the project > >>> >>>>>>>> can > >>> >>>>>>>> help by > >>> >>>>>>>> emailing the dev@ when they see something that hasn't been. > >>> >>>>>>>> > >>> >>>>>>>> - Culture: The design doc author(s) should be open to > feedback. > >>> >>>>>>>> A > >>> >>>>>>>> design > >>> >>>>>>>> doc should serve as the base for discussion and is by no means > >>> >>>>>>>> the > >>> >>>>>>>> final > >>> >>>>>>>> design. Of course, this does not mean the author has to accept > >>> >>>>>>>> every > >>> >>>>>>>> feedback. They should also be comfortable accepting / > rejecting > >>> >>>>>>>> ideas on > >>> >>>>>>>> technical grounds. > >>> >>>>>>>> > >>> >>>>>>>> - Process / Culture: For major ongoing projects, it can be > >>> >>>>>>>> useful > >>> >>>>>>>> to > >>> >>>>>>>> have > >>> >>>>>>>> some monthly Google hangouts that are open to the world. I am > >>> >>>>>>>> actually not > >>> >>>>>>>> sure how well this will work, because of the volunteering > nature > >>> >>>>>>>> and > >>> >>>>>>>> we need > >>> >>>>>>>> to adjust for timezones for people across the globe, but it > >>> >>>>>>>> seems > >>> >>>>>>>> worth > >>> >>>>>>>> trying. > >>> >>>>>>>> > >>> >>>>>>>> - Culture: Contributors (including committers) should be more > >>> >>>>>>>> direct > >>> >>>>>>>> in > >>> >>>>>>>> setting expectations, including whether they are working on a > >>> >>>>>>>> specific > >>> >>>>>>>> issue, whether they will be working on a specific issue, and > >>> >>>>>>>> whether > >>> >>>>>>>> an > >>> >>>>>>>> issue or pr or jira should be rejected. Most people I know in > >>> >>>>>>>> this > >>> >>>>>>>> community > >>> >>>>>>>> are nice and don't enjoy telling other people no, but it is > >>> >>>>>>>> often > >>> >>>>>>>> more > >>> >>>>>>>> annoying to a contributor to not know anything than getting a > >>> >>>>>>>> no. > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> On Fri, Oct 7, 2016 at 10:03 AM, Matei Zaharia > >>> >>>>>>>> <[hidden email]> > >>> >>>>>>>> wrote: > >>> >>>>>>>>> > >>> >>>>>>>>> > >>> >>>>>>>>> Love the idea of a more visible "Spark Improvement Proposal" > >>> >>>>>>>>> process that > >>> >>>>>>>>> solicits user input on new APIs. For what it's worth, I don't > >>> >>>>>>>>> think > >>> >>>>>>>>> committers are trying to minimize their own work -- every > >>> >>>>>>>>> committer > >>> >>>>>>>>> cares > >>> >>>>>>>>> about making the software useful for users. However, it is > >>> >>>>>>>>> always > >>> >>>>>>>>> hard to > >>> >>>>>>>>> get user input and so it helps to have this kind of process. > >>> >>>>>>>>> I've > >>> >>>>>>>>> certainly > >>> >>>>>>>>> looked at the *IPs a lot in other software I use just to see > >>> >>>>>>>>> the > >>> >>>>>>>>> biggest > >>> >>>>>>>>> things on the roadmap. > >>> >>>>>>>>> > >>> >>>>>>>>> When you're talking about "changing interfaces", are you > >>> >>>>>>>>> talking > >>> >>>>>>>>> about > >>> >>>>>>>>> public or internal APIs? I do think many people hate changing > >>> >>>>>>>>> public APIs > >>> >>>>>>>>> and I actually think that's for the best of the project. > That's > >>> >>>>>>>>> a > >>> >>>>>>>>> technical > >>> >>>>>>>>> debate, but basically, the worst thing when you're using a > >>> >>>>>>>>> piece > >>> >>>>>>>>> of > >>> >>>>>>>>> software > >>> >>>>>>>>> is that the developers constantly ask you to rewrite your app > >>> >>>>>>>>> to > >>> >>>>>>>>> update to a > >>> >>>>>>>>> new version (and thus benefit from bug fixes, etc). Cue > anyone > >>> >>>>>>>>> who's used > >>> >>>>>>>>> Protobuf, or Guava. The "let's get everyone to change their > >>> >>>>>>>>> code > >>> >>>>>>>>> this > >>> >>>>>>>>> release" model works well within a single large company, but > >>> >>>>>>>>> doesn't work > >>> >>>>>>>>> well for a community, which is why nearly all *very* widely > >>> >>>>>>>>> used > >>> >>>>>>>>> programming > >>> >>>>>>>>> interfaces (I'm talking things like Java standard library, > >>> >>>>>>>>> Windows > >>> >>>>>>>>> API, etc) > >>> >>>>>>>>> almost *never* break backwards compatibility. All this is > done > >>> >>>>>>>>> within reason > >>> >>>>>>>>> though, e.g. we do change things in major releases (2.x, 3.x, > >>> >>>>>>>>> etc). > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>> > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> ------------------------------------------------------------ > --------- > >>> >>>>>> To unsubscribe e-mail: [hidden email] > >>> >>>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> -- > >>> >>>>> Stavros Kontopoulos > >>> >>>>> Senior Software Engineer > >>> >>>>> Lightbend, Inc. > >>> >>>>> p: +30 6977967274 > >>> >>>>> e: [hidden email] > >>> >>>>> > >>> >>>>> > >>> >>>> > >>> >>> > >>> >> > >>> >> > >>> > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: [hidden email] > > > > > > ________________________________ > > > > If you reply to this email, your message will be added to the discussion > > below: > > > > http://apache-spark-developers-list.1001551.n3. > nabble.com/Spark-Improvement-Proposals-tp19268p19359.html > > > > To start a new topic under Apache Spark Developers List, email [hidden > > email] > > To unsubscribe from Apache Spark Developers List, click here. > > NAML > > > > > > ________________________________ > > View this message in context: RE: Spark Improvement Proposals > > Sent from the Apache Spark Developers List mailing list archive at > > Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- Ryan Blue Software Engineer Netflix
