Re: Assigning issue

Done -- have fun! :) On Mon, Oct 14, 2013 at 12:50 AM, karthik tunga wrote: > Hi, > > I was looking at some starter issues. How do I assign an issue to myself ? > I was looking at > SPARK-627 . > Its a simple change to the scripts, so I thoug

Re: appId is no longer in the command line args for StandaloneExecutor

Looks like the appId thing was removed accidentally quite a while ago: https://github.com/apache/incubator-spark/commit/85a35c68401e171df0b72b172a689d8c4e412199 and has gone unnoticed since. I have no objections to adding it back in... On Tue, Nov 5, 2013 at 2:36 PM, Reynold Xin wrote: > +aaron

Re: SPARK-942

By the way, there are a few places one can look for logs while testing: Unit test runner logs (should contain driver and worker logs): core/target/unit-tests.log Executor logs: work/app-* This should help find the root exception when you see one caught by the DAGScheduler, such as in this case.

Re: Mesos not working on head of master

Hey Evan, Sorry about the issue with the Mesos urls. I submitted the patch that broke it originally -- at the time, I did not know about the zk:// urls mapping to Mesos. I went ahead and started PR #217 to correct this, which also includes a unit

IntelliJ Scala Import Organizer Plugin

Hi guys, just wanted to share a little plugin I wrote for IntelliJ to help auto-organize Scala imports. Anyone who has submitted a patch to Spark has probably felt the exhilaration of manually sorting and bucketing your imports. Well, now you can let your IDE have some fun! It's in the plugin repo

Re: IMPORTANT: Spark mailing lists moving to Apache by September 1st

I'd be fine with one-way mirrors here (Apache threads being reflected in Google groups) -- I have no idea how one is supposed to navigate the Apache list to look for historic threads. On Thu, Dec 19, 2013 at 7:58 PM, Mike Potts wrote: > Thanks very much for the prompt and comprehensive reply!

Re: Any suggestion about JIRA 1006 "MLlib ALS gets stack overflow with too many iterations"?

I'm an idiot, but which part of the DAGScheduler is recursive here? Seems like processEvent shouldn't have inherently recursive properties. On Sat, Jan 25, 2014 at 9:57 PM, Reynold Xin wrote: > It seems to me fixing DAGScheduler to make it not recursive is the better > solution here, given the

Re: [VOTE] Graduation of Apache Spark

be it further > > RESOLVED, that the office of "Vice President, Apache Spark" be > > and hereby is created, the person holding such office to serve > > at the direction of the Board of Directors as the chair of the > > Apache Spark Project, and to have primary re

Re: Problems while moving from 0.8.0 to 0.8.1

Looks like your MyCustomKeyType.equals() method doesn't correctly handle a null argument. In general, the contract of equals is to return false if called with a null argument, which this code currently relies on. This could still be patched for the sake of backwards compatibility with similarly in

Re: Problems while moving from 0.8.0 to 0.8.1

lid to use it, it just may be unintuitive if you expect never to have null references of MyCustomKeyType hanging around. On Mon, Jan 27, 2014 at 12:12 PM, Archit Thakur wrote: > On Tue, Jan 28, 2014 at 1:25 AM, Aaron Davidson > wrote: > > > Looks like your MyCustomKeyType.eq

Re: Notice: JIRA messages will be forwarded to this list

Also, it may be intentional but the only PR-level comments seem to get forwarded. Comments on code are not sent. On Fri, Feb 7, 2014 at 12:31 PM, Shivaram Venkataraman < shiva...@eecs.berkeley.edu> wrote: > All the emails I get from github seem to have the same subject line > "[GitHub] incubator

Proposal: Clarifying minor points of Scala style

There are a few bits of the Scala style that are underspecified by both the Scala style guide and our own supplemental notes. Often, this leads to inconsistent formatting within the codebas

Re: Proposal: Clarifying minor points of Scala style

ormat ? Can we just stick > to the above for new functions ? > > Thanks > Shivaram > > On Mon, Feb 10, 2014 at 11:33 AM, Reynold Xin wrote: > > +1 on both > > > > > > On Mon, Feb 10, 2014 at 1:34 AM, Aaron Davidson > wrote: > > > >> T

Re: Proposal: Clarifying minor points of Scala style

my proposal - Essentially we can just have two styles: > >> The entire function + parameterList + return type fits in one line or > >> when it doesn't we wrap parameters into lines. > >> I agree that it makes the code a more verbose, but it'll make code

Re: [VOTE] Graduation of Apache Spark from the Incubator

nt, Apache Spark" be and hereby is created, > > > the person holding such office to serve at the direction of > > > the Board of Directors as the chair of the Apache Spark > > > Project, and to have primary responsibility for management > > > of th

Re: proposal: replace lift-json with spray-json

I am not sure I fully understand this reasoning. I imagine that lift-json is only one of hundreds of packages that would have to be built if you wanted to build all of Spark's transitive dependencies from source. Additionally, to make sure I understand the impact -- this is only intended to simpli

Re: [GitHub] incubator-spark pull request: SPARK-1078: Replace lift-json with j...

My apologies, my intention was that it is fine for the next minor *or*major release, regardless of what comes next. I only wanted to distinguish that from the next maintenance release, since my understanding is that we wish to avoid changing dependencies during maintenance releases. On Tue, Feb 1

Re: proposal: replace lift-json with spray-json

Will, thanks for the clarifications. I think Spark's main use-case is "warm, small inputs" right now, but the change seems reasonable to me nevertheless. Paul, do you know if there are any issues relevant to Spark that we need from 2.3.2? We would also have to wait for json4s to release a new vers

Re: proposal: replace lift-json with spray-json

ble. > > Best. > -- Paul > > -- > p...@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/ > > > On Wed, Feb 12, 2014 at 10:38 AM, Aaron Davidson > wrote: > > > Will, thanks for the clarifications. I think Spark's main use-case is > > &qu

Re: proposal: replace lift-json with spray-json

x > > Jackson series and Scala 2.10, but I think it makes sense to for Spark to > > just use the released version and then update when a json4s release is > > available. > > > > Best. > > -- Paul > > > > -- > > p...@mult.ifario.us | Multifarious, In

Re: Proposal: Clarifying minor points of Scala style

t it has some evil implications with regards to serialization and shadowing during inheritance. It's especially concerning when a field starts out as a constructor parameter and during a later patch becomes a field, and now we're serializing it. On Mon, Feb 10, 2014 at 9:00 PM, Aaron Dav

Re: Proposal: Clarifying minor points of Scala style

t;5: aload_0 > >6: invokespecial #23 // Method > java/lang/Object."":()V > >9: return > > } > > > ...however, compare Foo3 to what you get when `b` is `private[this] val`: > > > public class dummy.Foo5 { > &g

Re: Proposal: Clarifying minor points of Scala style

on't have a good answer, just more things to consider. > > > > On Wed, Feb 12, 2014 at 2:14 PM, Aaron Davidson > wrote: > > > Thanks for the clarification, Mark. So "private val" generates the Scala > > getter whereas "private[this] val" does not

Re: Signal/Noise Ratio

This is due, unfortunately, to Apache policies that all development-related discussion should take place on the dev list. As we are attempting to graduate from an incubating project to an Apache top level project, there were some concerns raised about GitHub, and the fastest solution to avoid confl

Re: coding style discussion: explicit return type in public APIs

One slight concern regarding primitive types -- in particular, Ints and Longs can have semantic differences when it comes to overflow, so it's often good to know what type of variable you're returning. Perhaps it is sufficient to say that Int is the "default" numeric type, and that other types shou

Re: Signal/Noise Ratio

han Jewett wrote: > > > Hi Aaron, > > > > Off-list message here. Can you point me to this policy? Due to some > > previous experiences here, I'm under the impression that it doesn't > exist. > > I can't find it on the Apache website. > > &

Re: Signal/Noise Ratio

;> > To: "dev@spark.incubator.apache.org" > >> > Subject: Re: Signal/Noise Ratio > >> > > >> > >FYI I submitted an ASF INFRA ticket on granting the AMPLab Jenkins > >> > >permission to use the github commit status API. > >> >