+1 for SIP lebles,waiting for Reynolds detailed proposal . Regards, Vaquar khan
On 8 Oct 2016 16:22, "Matei Zaharia" <matei.zaha...@gmail.com> wrote: > Sounds good. Just to comment on the compatibility part: > > > I meant changing public user interfaces. I think the first design is > > unlikely to be right, because it's done at a time when you have the > > least information. As a user, I find it considerably more frustrating > > to be unable to use a tool to get my job done, than I do having to > > make minor changes to my code in order to take advantage of features. > > I've seen committers be seriously reluctant to allow changes to > > @experimental code that are needed in order for it to really work > > right. You need to be able to iterate, and if people on both sides of > > the fence aren't going to respect that some newer apis are subject to > > change, then why even mark them as such? > > > > Ideally a finished SIP should give me a checklist of things that an > > implementation must do, and things that it doesn't need to do. > > Contributors/committers should be seriously discouraged from putting > > out a version 0.1 that doesn't have at least a prototype > > implementation of all those things, especially if they're then going > > to argue against interface changes necessary to get the the rest of > > the things done in the 0.2 version. > > Experimental APIs and alpha components are indeed supposed to be > changeable (https://cwiki.apache.org/confluence/display/SPARK/ > Spark+Versioning+Policy). Maybe people are being too conservative in some > cases, but I do want to note that regardless of what precise policy we try > to write down, this type of issue will ultimately be a judgment call. Is it > worth making a small cosmetic change in an API that's marked experimental, > but has been used widely for a year? Perhaps not. Is it worth making it in > something one month old, or even in an older API as we move to 2.0? Maybe > yes. I think we should just discuss each one (start an email thread if > resolving it on JIRA is too complex) and perhaps be more religious about > making things non-experimental when we think they're done. > > Matei > > > > > > > > On Fri, Oct 7, 2016 at 2:18 PM, Reynold Xin <r...@databricks.com> wrote: > >> I like the lightweight proposal to add a SIP label. > >> > >> During Spark 2.0 development, Tom (Graves) and I suggested using wiki to > >> track the list of major changes, but that never really materialized due > to > >> the overhead. Adding a SIP label on major JIRAs and then link to them > >> prominently on the Spark website makes a lot of sense. > >> > >> > >> On Fri, Oct 7, 2016 at 10:50 AM, Matei Zaharia <matei.zaha...@gmail.com > > > >> wrote: > >>> > >>> For the improvement proposals, I think one major point was to make them > >>> really visible to users who are not contributors, so we should do more > than > >>> sending stuff to dev@. One very lightweight idea is to have a new > type of > >>> JIRA called a SIP and have a link to a filter that shows all such > JIRAs from > >>> http://spark.apache.org. I also like the idea of SIP and design doc > >>> templates (in fact many projects have them). > >>> > >>> Matei > >>> > >>> On Oct 7, 2016, at 10:38 AM, Reynold Xin <r...@databricks.com> wrote: > >>> > >>> I called Cody last night and talked about some of the topics in his > email. > >>> It became clear to me Cody genuinely cares about the project. > >>> > >>> Some of the frustrations come from the success of the project itself > >>> becoming very "hot", and it is difficult to get clarity from people who > >>> don't dedicate all their time to Spark. In fact, it is in some ways > similar > >>> to scaling an engineering team in a successful startup: old processes > that > >>> worked well might not work so well when it gets to a certain size, > cultures > >>> can get diluted, building culture vs building process, etc. > >>> > >>> I also really like to have a more visible process for larger changes, > >>> especially major user facing API changes. Historically we upload > design docs > >>> for major changes, but it is not always consistent and difficult to > quality > >>> of the docs, due to the volunteering nature of the organization. > >>> > >>> Some of the more concrete ideas we discussed focus on building a > culture > >>> to improve clarity: > >>> > >>> - Process: Large changes should have design docs posted on JIRA. One > thing > >>> Cody and I didn't discuss but an idea that just came to me is we should > >>> create a design doc template for the project and ask everybody to > follow. > >>> The design doc template should also explicitly list goals and > non-goals, to > >>> make design doc more consistent. > >>> > >>> - Process: Email dev@ to solicit feedback. We have some this with some > >>> changes, but again very inconsistent. Just posting something on JIRA > isn't > >>> sufficient, because there are simply too many JIRAs and the signal get > lost > >>> in the noise. While this is generally impossible to enforce because we > can't > >>> force all volunteers to conform to a process (or they might not even be > >>> aware of this), those who are more familiar with the project can help > by > >>> emailing the dev@ when they see something that hasn't been. > >>> > >>> - Culture: The design doc author(s) should be open to feedback. A > design > >>> doc should serve as the base for discussion and is by no means the > final > >>> design. Of course, this does not mean the author has to accept every > >>> feedback. They should also be comfortable accepting / rejecting ideas > on > >>> technical grounds. > >>> > >>> - Process / Culture: For major ongoing projects, it can be useful to > have > >>> some monthly Google hangouts that are open to the world. I am actually > not > >>> sure how well this will work, because of the volunteering nature and > we need > >>> to adjust for timezones for people across the globe, but it seems worth > >>> trying. > >>> > >>> - Culture: Contributors (including committers) should be more direct in > >>> setting expectations, including whether they are working on a specific > >>> issue, whether they will be working on a specific issue, and whether an > >>> issue or pr or jira should be rejected. Most people I know in this > community > >>> are nice and don't enjoy telling other people no, but it is often more > >>> annoying to a contributor to not know anything than getting a no. > >>> > >>> > >>> On Fri, Oct 7, 2016 at 10:03 AM, Matei Zaharia < > matei.zaha...@gmail.com> > >>> wrote: > >>>> > >>>> > >>>> Love the idea of a more visible "Spark Improvement Proposal" process > that > >>>> solicits user input on new APIs. For what it's worth, I don't think > >>>> committers are trying to minimize their own work -- every committer > cares > >>>> about making the software useful for users. However, it is always > hard to > >>>> get user input and so it helps to have this kind of process. I've > certainly > >>>> looked at the *IPs a lot in other software I use just to see the > biggest > >>>> things on the roadmap. > >>>> > >>>> When you're talking about "changing interfaces", are you talking about > >>>> public or internal APIs? I do think many people hate changing public > APIs > >>>> and I actually think that's for the best of the project. That's a > technical > >>>> debate, but basically, the worst thing when you're using a piece of > software > >>>> is that the developers constantly ask you to rewrite your app to > update to a > >>>> new version (and thus benefit from bug fixes, etc). Cue anyone who's > used > >>>> Protobuf, or Guava. The "let's get everyone to change their code this > >>>> release" model works well within a single large company, but doesn't > work > >>>> well for a community, which is why nearly all *very* widely used > programming > >>>> interfaces (I'm talking things like Java standard library, Windows > API, etc) > >>>> almost *never* break backwards compatibility. All this is done within > reason > >>>> though, e.g. we do change things in major releases (2.x, 3.x, etc). > >>> > >>> > >>> > >>> > >> > > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >