Re: Spark Improvement Proposals

2017-03-13 Thread Sean Owen
Responding to your request for a vote, I meant that this isn't required per se and the consensus here was not to vote on it. Hence the jokes about meta-voting protocol. In that sense nothing new happened process-wise, nothing against ASF norms, if that's your concern. I think it's just an agreed c

Re: Spark Improvement Proposals

2017-03-13 Thread Tom Graves
Another thing I think you should send out is when exactly does this take affect.  Is it any major new feature without a pull request?   Is it anything major starting with the 2.3 release?   Tom On Monday, March 13, 2017 1:08 PM, Tom Graves wrote: I'm not sure how you can say its not a

Re: Spark Improvement Proposals

2017-03-13 Thread Tom Graves
I'm not sure how you can say its not a new process.  If that is the case why do we need a page documenting it?   As a developer if I want to put up a major improvement I have to now follow the SPIP whereas before I didn't, that certain seems like a new process.  As a PMC member I now have the ab

Re: Spark Improvement Proposals

2017-03-13 Thread Sean Owen
It's not a new process, in that it doesn't entail anything not already in http://apache.org/foundation/voting.html . We're just deciding to call a VOTE for this type of code modification. To your point -- yes, it's been around a long time with no further comment, and I called several times for mor

Re: Spark Improvement Proposals

2017-03-13 Thread Tom Graves
It seems like if you are adding responsibilities you should do a vote.  SPIP'S require votes from PMC members so you are now putting more responsibility on them. It feels like we should have an official vote to make sure they (PMC members) agree with that and to make sure everyone pays attention

Re: Spark Improvement Proposals

2017-03-13 Thread Sean Owen
This ended up proceeding as a normal doc change, instead of precipitating a meta-vote. However, the text that's on the web site now can certainly be further amended if anyone wants to propose a change from here. On Mon, Mar 13, 2017 at 1:50 PM Tom Graves wrote: > I think a vote here would be goo

Re: Spark Improvement Proposals

2017-03-13 Thread Tom Graves
I think a vote here would be good. I think most of the discussion was done by 4 or 5 people and its a long thread.  If nothing else it summarizes everything and gets people attention to the change. Tom On Thursday, March 9, 2017 10:55 AM, Sean Owen wrote: I think a VOTE is over-thinkin

Re: Spark Improvement Proposals

2017-03-10 Thread Reynold Xin
We can just start using spip label and link to it. On Fri, Mar 10, 2017 at 9:18 AM, Cody Koeninger wrote: > So to be clear, if I translate that google doc to markup and submit a > PR, you will merge it? > > If we're just using "spip" label, that's probably fine, but we still > need shared filt

Re: Spark Improvement Proposals

2017-03-10 Thread Cody Koeninger
Can someone with filter share permissions can make a filter for open SPIP and one for closed SPIP and share it? e.g. project = SPARK AND status in (Open, Reopened, "In Progress") AND labels=SPIP ORDER BY createdDate DESC and another with the status closed equivalent I just made an open ticket w

Re: Spark Improvement Proposals

2017-03-10 Thread Cody Koeninger
So to be clear, if I translate that google doc to markup and submit a PR, you will merge it? If we're just using "spip" label, that's probably fine, but we still need shared filters for open and closed SPIPs so the page can link to them. I do not believe I have jira permissions to share filters,

Re: Spark Improvement Proposals

2017-03-10 Thread Cody Koeninger
I think it ought to be its own page, linked from the more / community menu dropdowns. We also need the jira tag, and for the page to clearly link to filters that show proposed / completed SPIPs On Fri, Mar 10, 2017 at 3:39 AM, Sean Owen wrote: > Alrighty, if nobody is objecting, and nobody calls

Re: Spark Improvement Proposals

2017-03-10 Thread Sean Owen
Alrighty, if nobody is objecting, and nobody calls for a VOTE, then, let's say this document is the SPIP 1.0 process. I think the next step is just to translate the text to some suitable location. I suggest adding it to https://github.com/apache/spark-website/blob/asf-site/contributing.md On Thu,

Re: Spark Improvement Proposals

2017-03-09 Thread Koert Kuipers
gonna end up with a stackoverflow on recursive votes here On Thu, Mar 9, 2017 at 1:17 PM, Mark Hamstra wrote: > -0 on voting on whether we need a vote. > > On Thu, Mar 9, 2017 at 9:00 AM, Reynold Xin wrote: > >> I'm fine without a vote. (are we voting on wether we need a vote?) >> >> >> On Thu,

Re: Spark Improvement Proposals

2017-03-09 Thread Mark Hamstra
-0 on voting on whether we need a vote. On Thu, Mar 9, 2017 at 9:00 AM, Reynold Xin wrote: > I'm fine without a vote. (are we voting on wether we need a vote?) > > > On Thu, Mar 9, 2017 at 8:55 AM, Sean Owen wrote: > >> I think a VOTE is over-thinking it, and is rarely used, but, can't hurt. >>

Re: Spark Improvement Proposals

2017-03-09 Thread vaquar khan
Many of us have issue with "shepherd role " , i think we should go with vote. Regards, Vaquar khan On Thu, Mar 9, 2017 at 11:00 AM, Reynold Xin wrote: > I'm fine without a vote. (are we voting on wether we need a vote?) > > > On Thu, Mar 9, 2017 at 8:55 AM, Sean Owen wrote: > >> I think a VOTE

Re: Spark Improvement Proposals

2017-03-09 Thread Reynold Xin
I'm fine without a vote. (are we voting on wether we need a vote?) On Thu, Mar 9, 2017 at 8:55 AM, Sean Owen wrote: > I think a VOTE is over-thinking it, and is rarely used, but, can't hurt. > Nah, anyone can call a vote. This really isn't that formal. We just want to > declare and document con

Re: Spark Improvement Proposals

2017-03-09 Thread Sean Owen
I think a VOTE is over-thinking it, and is rarely used, but, can't hurt. Nah, anyone can call a vote. This really isn't that formal. We just want to declare and document consensus. I think SPIP is just a remix of existing process anyway, and don't think it will actually do much anyway, which is wh

Re: Spark Improvement Proposals

2017-03-09 Thread Cody Koeninger
I started this idea as a fork with a merge-able change to docs. Reynold moved it to his google doc, and has suggested during this email thread that a vote should occur. If a vote needs to occur, I can't see anything on http://apache.org/foundation/voting.html suggesting that I can call for a vote,

Re: Spark Improvement Proposals

2017-03-07 Thread Sean Owen
Do we need a VOTE? heck I think anyone can call one, anyway. Pre-flight vote check: anyone have objections to the text as-is? See https://docs.google.com/document/d/1-Zdi_W-wtuxS9hTK0P9qb2x-nRanvXmnZ7SUi4qMljg/edit# If so let's hash out specific suggest changes. If not, then I think the next ste

Re: Spark Improvement Proposals

2017-03-07 Thread Cody Koeninger
Another week, another ping. Anyone on the PMC willing to call a vote on this? On Mon, Feb 27, 2017 at 3:08 PM, Ryan Blue wrote: > I'd like to see more discussion on the issues I raised. I don't think > there was a response for why voting is limited to PMC members. > > Tim was kind enough to rep

Re: Spark Improvement Proposals

2017-02-27 Thread Sean Owen
To me, no new process is being invented here, on purpose, and so we should just rely on whatever governs any large JIRA or vote, because SPIPs are really just guidance for making a big JIRA. http://apache.org/foundation/voting.html suggests that PMC members have the binding votes in general, and f

Re: Spark Improvement Proposals

2017-02-27 Thread Ryan Blue
I'd like to see more discussion on the issues I raised. I don't think there was a response for why voting is limited to PMC members. Tim was kind enough to reply with his rationale for a shepherd, but I don't think that it justifies failing proposals. I think it boiled down to "shepherds can be he

Re: Spark Improvement Proposals

2017-02-24 Thread Joseph Bradley
The current draft LGTM. I agree some of the various concerns may need to be addressed in the future, depending on how SPIPs progress in practice. If others agree, let's put it to a vote and revisit the proposal in a few months. Joseph On Fri, Feb 24, 2017 at 5:35 AM, Cody Koeninger wrote: > It'

Re: Spark Improvement Proposals

2017-02-24 Thread Cody Koeninger
It's been a week since any further discussion. Do PMC members think the current draft is OK to vote on? On Fri, Feb 17, 2017 at 10:41 PM, vaquar khan wrote: > I like document and happy to see SPIP draft version however i feel shepherd > role is again hurdle in process improvement ,It's like ever

Re: Spark Improvement Proposals

2017-02-17 Thread vaquar khan
I like document and happy to see SPIP draft version however i feel shepherd role is again hurdle in process improvement ,It's like everything depends only on shepherd . Also want to add point that SPIP should be time bound with define SLA else will defeats purpose. Regards, Vaquar khan On Thu,

Re: Spark Improvement Proposals

2017-02-16 Thread Ryan Blue
> [The shepherd] can advise on technical and procedural considerations for people outside the community The sentiment is good, but this doesn't justify requiring a shepherd for a proposal. There are plenty of people that wouldn't need this, would get feedback during discussion, or would ask a comm

Re: Spark Improvement Proposals

2017-02-16 Thread Sam Elamin
Hi Folks I thought id chime in as someone new to the process so feel free to disregard it if it doesn't make sense. I definitely agree that we need a new forum to identify or discuss changes as JIRA isnt exactly the best place to do that, its a Bug tracker first and foremost. For example I was c

Re: Spark Improvement Proposals

2017-02-16 Thread Tim Hunter
The doc looks good to me. Ryan, the role of the shepherd is to make sure that someone knowledgeable with Spark processes is involved: this person can advise on technical and procedural considerations for people outside the community. Also, if no one is willing to be a shepherd, the proposed idea i

Re: Spark Improvement Proposals

2017-02-16 Thread Cody Koeninger
Reynold, thanks, LGTM. Sean, great concerns. I agree that behavior is largely cultural and writing down a process won't necessarily solve any problems one way or the other. But one outwardly visible change I'm hoping for out of this a way for people who have a stake in Spark, but can't follow ji

Re: Spark Improvement Proposals

2017-02-16 Thread Sean Owen
The text seems fine to me. Really, this is not describing a fundamentally new process, which is good. We've always had JIRAs, we've always been able to call a VOTE for a big question. This just writes down a sensible set of guidelines for putting those two together when a major change is proposed.

Re: Spark Improvement Proposals

2017-02-16 Thread Ryan Blue
;>>>>>> >> >>> > To that end, the two biggest areas for improvements in >>>>>>>>>>> my opinion >>>>>>>>>>> >> >>> > are: >>>>>>>>>>> >

Re: Spark Improvement Proposals

2017-02-16 Thread Reynold Xin
n't follow >>>>>>>>>> closely, it is >>>>>>>>>> >> >>> > difficult to >>>>>>>>>> >> >>> > know what the important initiatives are. Even for people >>>>>>>>>> that do >>>>>>>>>> >> >>

Re: Spark Improvement Proposals

2017-02-14 Thread Cody Koeninger
;>>>>>> >> >>> > >>>>>>>>> >> >>> > 2. Solicit user (broadly defined, including developers >>>>>>>>> themselves) >>>>>>>>> >> >>> > input >>>>>>>>>

Re: Spark Improvement Proposals

2017-02-13 Thread Reynold Xin
;>>>> >> >>> > >>>>>>>> >> >>> > I've taken Cody's doc and edited it: >>>>>>>> >> >>> > >>>>>>>> >> >>> > >>>>>>>> >

Re: Spark Improvement Proposals

2017-02-11 Thread Xiao Li
recommended lazy >>>>>>> consensus >>>>>>> >> >>> > as >>>>>>> >> >>> > opposed to voting. The reason being in voting there can >>>>>>> easily be a >>>>>>> >> >>

Re: Spark Improvement Proposals

2017-02-11 Thread Cody Koeninger
>>> > things and linking them elsewhere simply having design docs >>>>>> and >>>>>> >> >>> > prototypes >>>>>> >> >>> > implementations in PRs is not something that has not worked >>&g

Re: Spark Improvement Proposals

2017-01-11 Thread Reynold Xin
orm and involve", rather than >>>>> just >>>>> >> >>> > "involve". SIPs should also have at least two emails that go >>>>> to >>>>> >> >>> > dev@. >>>>> >> >>>

Re: Spark Improvement Proposals

2017-01-05 Thread Tim Hunter
gt; >> >>> > On Tue, Nov 1, 2016 at 12:09 AM, Reynold Xin < >>>> r...@databricks.com> >>>> >> >>> > wrote: >>>> >> >>> >> >>>> >> >>> >> Most things looked OK to me too, although I do plan to take a >>>> >> >>> >> closer >&g

Re: Spark Improvement Proposals

2017-01-03 Thread Cody Koeninger
PM, Marcelo Vanzin >>> >> >>> >> >>> >> >>> >> wrote: >>> >> >>> >>> >>> >> >>> >>> The proposal looks OK to me. I assume, even though it's not >>> >> >>> >>> explicitly >>> >> >>> >>> called, that voting

Re: Spark Improvement Proposals

2017-01-03 Thread Imran Rashid
t;> >> >>> >>> candidate >> >> >>> >>> for a SIP, given the scope of the work. The document attached >> even >> >> >>> >>> somewhat matches the proposed format. So if anyone wants to try >> >> >&g

Re: Spark Improvement Proposals

2017-01-03 Thread Joseph Bradley
europe is over, are any committers > >> >>> >>> > interested > >> >>> >>> > in > >> >>> >>> > moving forward with this? > >> >>> >>> > > >> >>> >>&g

Re: Spark Improvement Proposals

2017-01-02 Thread Cody Koeninger
; >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-improvement-proposals.md >> >>> >>>

Re: Spark Improvement Proposals

2016-11-08 Thread Ryan Blue
;> >>> >> > >>> >>> >> > >>> >>> >> I didn't want to write "lets focus on Flink" or any other > >>> >>> >> framework. > >>> >>> >> The > >>> >>> >> idea with benchmarks was to show two

Re: Spark Improvement Proposals

2016-11-08 Thread Cody Koeninger
hat Spark is >>> >>> >> still on >>> >>> >> the >>> >>> >> top >>> >>> >> >>> >>> >> >>> >>> >> No more, no less. Benchmarks will be helpful, but I don't

Re: Spark Improvement Proposals

2016-11-07 Thread Reynold Xin
gt; >>> >> No more, no less. Benchmarks will be helpful, but I don't think >> >>> >> they're the >> >>> >> most important thing in Spark :) On the Spark main page there is >> still >> >>> >> chart >> >>> &g

Re: Spark Improvement Proposals

2016-11-07 Thread Reynold Xin
I, but much faster and optimized, comparable or > >>> >> even > >>> >> faster than other frameworks. > >>> >> > >>> >> > >>> >> About real-time streaming, I think it would be just good to see it > in > >&g

Re: Spark Improvement Proposals

2016-10-17 Thread Cody Koeninger
I think narrowly focusing on Flink or benchmarks is missing my point. My point is evolve or die. Spark's governance and organization is hampering its ability to evolve technologically, and it needs to change. On Sun, Oct 16, 2016 at 9:21 PM, Debasish Das wrote: > Thanks Cody for bringing up a v

Re: Spark Improvement Proposals(Internet mail)

2016-10-17 Thread 黄明
n be. --- Sincerely Andy 原始邮件 发件人: Debasish Das 收件人: Tomasz Gawęda 抄送: dev@spark.apache.org; Cody Koeninger 发送时间: 2016年10月17日(周一) 10:21 主题: Re: Spark Improvement Proposals(Internet mail) Thanks Cody for bringing up a valid point...I picked up Spark in 2014 as soon as I looked into it since comp

Re: Spark Improvement Proposals

2016-10-16 Thread Debasish Das
Thanks Cody for bringing up a valid point...I picked up Spark in 2014 as soon as I looked into it since compared to writing Java map-reduce and Cascading code, Spark made writing distributed code fun...But now as we went deeper with Spark and real-time streaming use-case gets more prominent, I thin

Re: Spark Improvement Proposals

2016-10-16 Thread Tomasz Gawęda
Hi everyone, I'm quite late with my answer, but I think my suggestions may help a little bit. :) Many technical and organizational topics were mentioned, but I want to focus on these negative posts about Spark and about "haters" I really like Spark. Easy of use, speed, very good community - it'

Re: Spark Improvement Proposals

2016-10-12 Thread kant kodali
> First we can always have other people suggest SIPs but mark >> them as >> >>>> >> > “unreviewed” and have committers basically move them forward. >> The >> >>>> >> > problem is >> >>>> >> > that wri

Re: Spark Improvement Proposals

2016-10-11 Thread Ryan Blue
;> >> > > >>>> >> > As for strategy, in many cases implementation strategy can affect > >>>> >> > the > >>>> >> > goals. > >>>> >> > I will give a small example: In the current structur

Re: Spark Improvement Proposals

2016-10-10 Thread Cody Koeninger
t;> >> >>> >>>> On Fri, Oct 7, 2016 at 3:58 PM, Stavros Kontopoulos >>>> >> >>> >>>> <[hidden email]> wrote: >>>> >> >>> >>>>> >>>> >> >>> >>>>> +1 to the

Re: Spark Improvement Proposals

2016-10-10 Thread Mark Hamstra
n a set of all distinct >>> values. >>> >> > One >>> >> > way to implement this would be to make the set into a map and have >>> the >>> >> > value >>> >> > contain the last time seen. Multiplying it acros

Re: Spark Improvement Proposals

2016-10-10 Thread Mark Hamstra
I'm not a fan of the SEP acronym. Besides it prior established meaning of "Somebody else's problem", the are other inappropriate or offensive connotations such as this Australian slang that often gets shortened to just "sep": http://www.urbandictionary.com/define.php?term=Seppo On Sun, Oct 9, 20

Re: Spark Improvement Proposals

2016-10-10 Thread Mark Hamstra
y, it is easy for whoever goes to the design > >> > document to not think about these cases. Furthermore, it might be > >> > decided > >> > that these cases are rare enough so that the strategy is still good > >> > enough > >> > but how would w

Re: Spark Improvement Proposals

2016-10-10 Thread Cody Koeninger
gt;> problem is >>>>>>>>> that writing a good document takes time. This way we can leverage >>>>>>>>> non >>>>>>>>> committers to do some of this work (it is just another way to >>>>>>>>> contribute). &

Re: Spark Improvement Proposals

2016-10-10 Thread Steve Loughran
gt;>>>>> >>>>>>>> >>>>>>>> As for strategy, in many cases implementation strategy can affect >>>>>>>> the >>>>>>>> goals. >>>>>>>> I will give a small example: In the curren

Re: Spark Improvement Proposals

2016-10-10 Thread Matei Zaharia
mall example: In the current structured streaming >>>>>>> strategy, >>>>>>> we group by the time to achieve a sliding window. This is definitely >>>>>>> an >>>>>>> implementation decision and not a goal. However, I can think of >&

Re: Spark Improvement Proposals

2016-10-10 Thread Cody Koeninger
the time inside their calculation >> >>> > buffer. >> >>> > For example, let’s say we want to return a set of all distinct >> >>> > values. >> >>> > One >> >>> > way to implement this would be to make the set into

Re: Spark Improvement Proposals

2016-10-10 Thread Ryan Blue
> >>> > on > >>> > the type of aggregations and their performance which does affect the > >>> > goal. > >>> > Without adding the strategy, it is easy for whoever goes to the > design > >>> > document to not think about these

Re: Spark Improvement Proposals

2016-10-10 Thread Cody Koeninger
ions and their performance which does affect the >>> > goal. >>> > Without adding the strategy, it is easy for whoever goes to the design >>> > document to not think about these cases. Furthermore, it might be >>> > decided >>> > that these ca

Re: Spark Improvement Proposals

2016-10-10 Thread Ryan Blue
gt;> > I believe this example is exactly what Cody was talking about. Since >> many >> > times implementation strategies have a large effect on the goal, we >> should >> > have it discussed when discussing the goals. In addition, while it is >> often >

Re: Spark Improvement Proposals

2016-10-10 Thread Cody Koeninger
hat the strategy is still good >> > enough >> > but how would we know it without user feedback? >> > >> > I believe this example is exactly what Cody was talking about. Since >> > many >> > times implementation strategies have a large effect o

Re: Spark Improvement Proposals

2016-10-10 Thread Ryan Blue
le is exactly what Cody was talking about. Since many > > times implementation strategies have a large effect on the goal, we > should > > have it discussed when discussing the goals. In addition, while it is > often > > easy to throw out completely infeasible goals, it is often muc

Re: Spark Improvement Proposals

2016-10-10 Thread Cody Koeninger
gt; Assaf. > > > > From: Cody Koeninger-2 [via Apache Spark Developers List] > [mailto:ml-node+[hidden email]] > Sent: Monday, October 10, 2016 2:25 AM > To: Mendelson, Assaf > Subject: Re: Spark Improvement Proposals > > > > Only committers should formally submit S

RE: Spark Improvement Proposals

2016-10-09 Thread assaf.mendelson
ia Apache Spark Developers List] [mailto:ml-node+s1001551n19359...@n3.nabble.com] Sent: Monday, October 10, 2016 2:25 AM To: Mendelson, Assaf Subject: Re: Spark Improvement Proposals Only committers should formally submit SIPs because in an apache project only commiters have explicit political power. If a

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
Only committers should formally submit SIPs because in an apache project only commiters have explicit political power. If a user can't find a commiter willing to sponsor an SIP idea, they have no way to get the idea passed in any case. If I can't find a committer to sponsor this meta-SIP idea, I'

Re: Spark Improvement Proposals

2016-10-09 Thread Nicholas Chammas
On Sun, Oct 9, 2016 at 5:19 PM Cody Koeninger wrote: > Regarding name, if the SIP overlap is a concern, we can pick a different > name. > > My tongue in cheek suggestion would be > > Spark Lightweight Improvement process (SPARKLI) > If others share my minor concern about the SIP name, I propose

Re: Spark Improvement Proposals

2016-10-09 Thread Matei Zaharia
Well, I think there are a few things here that don't make sense. First, why should only committers submit SIPs? Development in the project should be open to all contributors, whether they're committers or not. Second, I think unrealistic goals can be found just by inspecting the goals, and I'm n

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
Yeah, I've looked at KIPs and Scala SIPs. I'm reluctant to use the Kafka structured streaming as an example because of the pre-existing conflict around it. If Michael or another committer wanted to put it forth as an example, I'd participate in good faith though. On Sun, Oct 9, 2016 at 5:07 PM,

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
Users instead of people, sure. Commiters and contributors are (or at least should be) a subset of users. Non goals, sure. I don't care what the name is, but we need to clearly say e.g. 'no we are not maintaining compatibility with XYZ right now'. API, what I care most about is whether it allows

Re: Spark Improvement Proposals

2016-10-09 Thread Ofir Manor
This is a great discussion! Maybe you could have a look at Kafka's process - it also uses Rejected Alternatives and I personally find it very clear actually (the link also leads to all KIPs): https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals Cody - maybe you could take

Re: Spark Improvement Proposals

2016-10-09 Thread Matei Zaharia
Yup, this is the stuff that I found unclear. Thanks for clarifying here, but we should also clarify it in the writeup. In particular: - Goals needs to be about user-facing behavior ("people" is broad) - I'd rename Rejected Goals to Non-Goals. Otherwise someone will dig up one of these and say "

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
Regarding name, if the SIP overlap is a concern, we can pick a different name. My tongue in cheek suggestion would be Spark Lightweight Improvement process (SPARKLI) On Sun, Oct 9, 2016 at 4:14 PM, Cody Koeninger wrote: > So to focus the discussion on the specific strategy I'm suggesting, > docum

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
So to focus the discussion on the specific strategy I'm suggesting, documented at https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-improvement-proposals.md "Goals: What must this allow people to do, that they can't currently?" Is it unclear that this is focusing specifically on people-

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
If there's confusion there, the document is specifically what I'm proposing. The email is just by way of introduction. On Sun, Oct 9, 2016 at 3:47 PM, Nicholas Chammas wrote: > Oh, hmm… I guess I’m a little confused on the relation between Cody’s > email and the document he linked to, which say

Re: Spark Improvement Proposals

2016-10-09 Thread Nicholas Chammas
Oh, hmm… I guess I’m a little confused on the relation between Cody’s email and the document he linked to, which says: https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-improvement-proposals.md#when SIPs should be used for significant user-facing or cross-cutting changes, not day-to-day

Re: Spark Improvement Proposals

2016-10-09 Thread Matei Zaharia
Yup, but the example you gave is for alternatives about *user-facing behavior*, not implementation. The current SIP doc describes "strategy" more as implementation strategy. I'm just saying there are different possible goals for these types of docs. BTW, PEPs and Scala SIPs focus primarily on u

Re: Spark Improvement Proposals

2016-10-09 Thread Nicholas Chammas
- Rejected strategies: I personally wouldn’t put this, because what’s the point of voting to reject a strategy before you’ve really begun designing and implementing something? What if you discover that the strategy is actually better when you start doing stuff? I would guess the point

Re: Spark Improvement Proposals

2016-10-09 Thread Matei Zaharia
Hi Cody, I think this would be a lot more concrete if we had a more detailed template for SIPs. Right now, it's not super clear what's in scope -- e.g. are they a way to solicit feedback on the user-facing behavior or on the internals? "Goals" can cover both things. I've been thinking of SIPs

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
Here's my specific proposal (meta-proposal?) Spark Improvement Proposals (SIP) Background: The current problem is that design and implementation of large features are often done in private, before soliciting user feedback. When feedback is solicited, it is often as to detailed design specifics

Re: Spark Improvement Proposals

2016-10-08 Thread vaquar khan
+1 for SIP lebles,waiting for Reynolds detailed proposal . Regards, Vaquar khan On 8 Oct 2016 16:22, "Matei Zaharia" wrote: > Sounds good. Just to comment on the compatibility part: > > > I meant changing public user interfaces. I think the first design is > > unlikely to be right, because it'

Re: Spark Improvement Proposals

2016-10-08 Thread Matei Zaharia
Sounds good. Just to comment on the compatibility part: > I meant changing public user interfaces. I think the first design is > unlikely to be right, because it's done at a time when you have the > least information. As a user, I find it considerably more frustrating > to be unable to use a too

Re: Spark Improvement Proposals

2016-10-07 Thread Reynold Xin
Alright looks like there are quite a bit of support. We should wait to hear from more people too. To push this forward, Cody and I will be working together in the next couple of weeks to come up with a concrete, detailed proposal on what this entails, and then we can discuss this the specific prop

Re: Spark Improvement Proposals

2016-10-07 Thread Cody Koeninger
Yeah, in case it wasn't clear, I was talking about SIPs for major user-facing or cross-cutting changes, not minor feature adds. On Fri, Oct 7, 2016 at 3:58 PM, Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > +1 to the SIP label as long as it does not slow down things and it targ

Re: Spark Improvement Proposals

2016-10-07 Thread Cody Koeninger
+1 to adding an SIP label and linking it from the website. I think it needs - template that focuses it towards soliciting user goals / non goals - clear resolution as to which strategy was chosen to pursue. I'd recommend a vote. Matei asked me to clarify what I meant by changing interfaces, I t

Re: Spark Improvement Proposals

2016-10-07 Thread Reynold Xin
I like the lightweight proposal to add a SIP label. During Spark 2.0 development, Tom (Graves) and I suggested using wiki to track the list of major changes, but that never really materialized due to the overhead. Adding a SIP label on major JIRAs and then link to them prominently on the Spark web

Re: Spark Improvement Proposals

2016-10-07 Thread Hyukjin Kwon
I am glad that it was not only what I was thinking. I also do agree with Holden, Sean and Cody. All I wanted to say were all said. 2016-10-08 1:16 GMT+09:00 Holden Karau : > First off, thanks Cody for taking the time to put together these proposals > - I think it has kicked off some wonderful d

Re: Spark Improvement Proposals

2016-10-07 Thread Matei Zaharia
For the improvement proposals, I think one major point was to make them really visible to users who are not contributors, so we should do more than sending stuff to dev@. One very lightweight idea is to have a new type of JIRA called a SIP and have a link to a filter that shows all such JIRAs fr

Re: Spark Improvement Proposals

2016-10-07 Thread Reynold Xin
I called Cody last night and talked about some of the topics in his email. It became clear to me Cody genuinely cares about the project. Some of the frustrations come from the success of the project itself becoming very "hot", and it is difficult to get clarity from people who don't dedicate all t

Re: Spark Improvement Proposals

2016-10-07 Thread Nicholas Chammas
There are several important discussions happening simultaneously. Should we perhaps split them up into separate threads? Otherwise it’s really difficult to follow. It seems like the discussion about having a more formal “Spark Improvement Proposal” process should take priority here. Other discuss

Re: Spark Improvement Proposals

2016-10-07 Thread Matei Zaharia
I think people misunderstood my comment about trolls a bit -- I'm not saying to just dismiss what people say, but to focus on what improves the project instead of being upset that people criticize stuff. This stuff happens all the time to any project in a "hot" area, as Sean said. I don't think

Re: Spark Improvement Proposals

2016-10-07 Thread Holden Karau
First off, thanks Cody for taking the time to put together these proposals - I think it has kicked off some wonderful discussion. I think dismissing people's complaints with Spark as largely trolls does us a disservice, it’s important for us to recognize our own shortcomings - otherwise we are bli

Re: Spark Improvement Proposals

2016-10-07 Thread Cody Koeninger
Sean, that was very eloquently put, and I 100% agree. If I ever meet you in person, I'll buy you multiple rounds of beverages of your choice ;) This is probably reiterating some of what you said in a less clear manner, but I'll throw more of my 2 cents in. - Design. Yes, design by committee doesn

Re: Spark Improvement Proposals

2016-10-07 Thread Sean Owen
Suggestion actions way at the bottom. On Fri, Oct 7, 2016 at 5:14 AM Matei Zaharia wrote: since March. But it's true that other things such as the Kafka source for it didn't have as much design on JIRA. Nonetheless, this component is still early on and there's still a lot of time to change it, w

Re: Spark Improvement Proposals

2016-10-06 Thread Xiao Li
Let us continue to improve Apache Spark! I volunteer to go through all the SQL-related open JIRAs. Xiao Li 2016-10-06 21:14 GMT-07:00 Matei Zaharia : > Hey Cody, > > Thanks for bringing these things up. You're talking about quite a few > different things here, but let me get to them each in tur

Re: Spark Improvement Proposals

2016-10-06 Thread Matei Zaharia
Hey Cody, Thanks for bringing these things up. You're talking about quite a few different things here, but let me get to them each in turn. 1) About technical / design discussion -- I fully agree that everything big should go through a lot of review, and I like the idea of a more formal way to

Re: Spark Improvement Proposals

2016-10-06 Thread DW @ Gmail
I was there, too. I agree with Cody's assessments and recommendations Dean Sent from my rotary phone. > On Oct 6, 2016, at 9:51 PM, Cody Koeninger wrote: > > I love Spark. 3 or 4 years ago it was the first distributed computing > environment that felt usable, and the community was welcoming