Re: Spark Improvement Proposals

2016-10-08 Thread Matei Zaharia
Sounds good. Just to comment on the compatibility part: > I meant changing public user interfaces. I think the first design is > unlikely to be right, because it's done at a time when you have the > least information. As a user, I find it considerably more frustrating > to be unable to use a too

Re: Improving governance / committers (split from Spark Improvement Proposals thread)

2016-10-08 Thread Matei Zaharia
This makes a lot of sense; just to comment on a few things: > - More committers > Just looking at the ratio of committers to open tickets, or committers > to contributors, I don't think you have enough human power. > I realize this is a touchy issue. I don't have dog in this fight, > because I'm

Re: Improving volunteer management / JIRAs (split from Spark Improvement Proposals thread)

2016-10-08 Thread Matei Zaharia
icholas Chammas [via Apache Spark Developers List] [mailto:ml-node+ > [hidden email] > <http://user/SendEmail.jtp?type=node&node=19322&i=0>] > Sent: Saturday, October 08, 2016 12:42 AM > To: Mendelson, Assaf > Subject: Re: Improving volunteer management / JIRAs (spli

Re: Improving volunteer management / JIRAs (split from Spark Improvement Proposals thread)

2016-10-08 Thread Cody Koeninger
ml-node+ [hidden email] > <http:///user/SendEmail.jtp?type=node&node=19322&i=0>] > *Sent:* Saturday, October 08, 2016 12:42 AM > *To:* Mendelson, Assaf > *Subject:* Re: Improving volunteer management / JIRAs (split from Spark > Improvement Proposals thread) > &

RE: Improving volunteer management / JIRAs (split from Spark Improvement Proposals thread)

2016-10-08 Thread assaf.mendelson
: Nicholas Chammas [via Apache Spark Developers List] [mailto:ml-node+s1001551n19310...@n3.nabble.com] Sent: Saturday, October 08, 2016 12:42 AM To: Mendelson, Assaf Subject: Re: Improving volunteer management / JIRAs (split from Spark Improvement Proposals thread) I agree with Cody and others that

Re: Spark Improvement Proposals

2016-10-07 Thread Reynold Xin
Alright looks like there are quite a bit of support. We should wait to hear from more people too. To push this forward, Cody and I will be working together in the next couple of weeks to come up with a concrete, detailed proposal on what this entails, and then we can discuss this the specific prop

Re: Improving volunteer management / JIRAs (split from Spark Improvement Proposals thread)

2016-10-07 Thread Nicholas Chammas
Ah yes, on a given JIRA issue the number of watchers is often a better indicator of community interest than votes. But yeah, it could be any metric or formula we want, as long as it yielded a "reasonable" bar to cross for unsolicited contributions to get committer review--or at the very least a co

Re: Improving volunteer management / JIRAs (split from Spark Improvement Proposals thread)

2016-10-07 Thread Cody Koeninger
I really like the idea of using jira votes (and/or watchers?) as a filter! On Fri, Oct 7, 2016 at 4:41 PM, Nicholas Chammas wrote: > I agree with Cody and others that we need some automation — or at least an > adjusted process — to help us manage organic contributions better. > > The objections a

Re: Improving volunteer management / JIRAs (split from Spark Improvement Proposals thread)

2016-10-07 Thread Nicholas Chammas
I agree with Cody and others that we need some automation — or at least an adjusted process — to help us manage organic contributions better. The objections about automated closing being potentially abrasive are understood, but I wouldn’t accept that as a defeat for automation. Instead, it seems l

Re: Spark Improvement Proposals

2016-10-07 Thread Cody Koeninger
Yeah, in case it wasn't clear, I was talking about SIPs for major user-facing or cross-cutting changes, not minor feature adds. On Fri, Oct 7, 2016 at 3:58 PM, Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > +1 to the SIP label as long as it does not slow down things and it targ

Improving governance / committers (split from Spark Improvement Proposals thread)

2016-10-07 Thread Cody Koeninger
So concrete problems / potential solutions: - Technical discussion needs to be public, or you don't hear use cases and alternative viewpoints. Yet email communication is low-bandwidth and hard to read people's emotions, so committers who are colocated talk and decide things. A possible alternativ

Improving volunteer management / JIRAs (split from Spark Improvement Proposals thread)

2016-10-07 Thread Cody Koeninger
Matei asked: > I agree about empowering people interested here to contribute, but I'm > wondering, do you think there are technical things that people don't want to > work on, or is it a matter of what there's been time to do? It's a matter of mismanagement and miscommunication. The structur

Re: Spark Improvement Proposals

2016-10-07 Thread Cody Koeninger
+1 to adding an SIP label and linking it from the website. I think it needs - template that focuses it towards soliciting user goals / non goals - clear resolution as to which strategy was chosen to pursue. I'd recommend a vote. Matei asked me to clarify what I meant by changing interfaces, I t

Re: Spark Improvement Proposals

2016-10-07 Thread Reynold Xin
I like the lightweight proposal to add a SIP label. During Spark 2.0 development, Tom (Graves) and I suggested using wiki to track the list of major changes, but that never really materialized due to the overhead. Adding a SIP label on major JIRAs and then link to them prominently on the Spark web

Re: Spark Improvement Proposals

2016-10-07 Thread Hyukjin Kwon
I am glad that it was not only what I was thinking. I also do agree with Holden, Sean and Cody. All I wanted to say were all said. 2016-10-08 1:16 GMT+09:00 Holden Karau : > First off, thanks Cody for taking the time to put together these proposals > - I think it has kicked off some wonderful d

Re: Spark Improvement Proposals

2016-10-07 Thread Matei Zaharia
For the improvement proposals, I think one major point was to make them really visible to users who are not contributors, so we should do more than sending stuff to dev@. One very lightweight idea is to have a new type of JIRA called a SIP and have a link to a filter that shows all such JIRAs fr

Re: Spark Improvement Proposals

2016-10-07 Thread Reynold Xin
I called Cody last night and talked about some of the topics in his email. It became clear to me Cody genuinely cares about the project. Some of the frustrations come from the success of the project itself becoming very "hot", and it is difficult to get clarity from people who don't dedicate all t

Re: Spark Improvement Proposals

2016-10-07 Thread Nicholas Chammas
There are several important discussions happening simultaneously. Should we perhaps split them up into separate threads? Otherwise it’s really difficult to follow. It seems like the discussion about having a more formal “Spark Improvement Proposal” process should take priority here. Other discuss

Re: Spark Improvement Proposals

2016-10-07 Thread Matei Zaharia
I think people misunderstood my comment about trolls a bit -- I'm not saying to just dismiss what people say, but to focus on what improves the project instead of being upset that people criticize stuff. This stuff happens all the time to any project in a "hot" area, as Sean said. I don't think

Re: Spark Improvement Proposals

2016-10-07 Thread Holden Karau
First off, thanks Cody for taking the time to put together these proposals - I think it has kicked off some wonderful discussion. I think dismissing people's complaints with Spark as largely trolls does us a disservice, it’s important for us to recognize our own shortcomings - otherwise we are bli

Re: Spark Improvement Proposals

2016-10-07 Thread Cody Koeninger
Sean, that was very eloquently put, and I 100% agree. If I ever meet you in person, I'll buy you multiple rounds of beverages of your choice ;) This is probably reiterating some of what you said in a less clear manner, but I'll throw more of my 2 cents in. - Design. Yes, design by committee doesn

Re: Spark Improvement Proposals

2016-10-07 Thread Sean Owen
Suggestion actions way at the bottom. On Fri, Oct 7, 2016 at 5:14 AM Matei Zaharia wrote: since March. But it's true that other things such as the Kafka source for it didn't have as much design on JIRA. Nonetheless, this component is still early on and there's still a lot of time to change it, w

Re: Spark Improvement Proposals

2016-10-06 Thread Xiao Li
Let us continue to improve Apache Spark! I volunteer to go through all the SQL-related open JIRAs. Xiao Li 2016-10-06 21:14 GMT-07:00 Matei Zaharia : > Hey Cody, > > Thanks for bringing these things up. You're talking about quite a few > different things here, but let me get to them each in tur

Re: Spark Improvement Proposals

2016-10-06 Thread Matei Zaharia
Hey Cody, Thanks for bringing these things up. You're talking about quite a few different things here, but let me get to them each in turn. 1) About technical / design discussion -- I fully agree that everything big should go through a lot of review, and I like the idea of a more formal way to

Re: Spark Improvement Proposals

2016-10-06 Thread DW @ Gmail
I was there, too. I agree with Cody's assessments and recommendations Dean Sent from my rotary phone. > On Oct 6, 2016, at 9:51 PM, Cody Koeninger wrote: > > I love Spark. 3 or 4 years ago it was the first distributed computing > environment that felt usable, and the community was welcoming

Spark Improvement Proposals

2016-10-06 Thread Cody Koeninger
I love Spark. 3 or 4 years ago it was the first distributed computing environment that felt usable, and the community was welcoming. But I just got back from the Reactive Summit, and this is what I observed: - Industry leaders on stage making fun of Spark's streaming model - Open source project

<    1   2