Re: What's the best practice for developing new features for spark ?
I personally build with SBT and run Spark on YARN with IntelliJ. You need to connect to remote JVMs with a remote debugger. You also need to do similar, if you use Python, because it will launch a JVM on the driver aswell. On Wed, Aug 19, 2015 at 2:10 PM canan chen wrote: > Thanks Ted. I notice another thread about running spark programmatically > (client mode for standalone and yarn). Would it be much easier to debug > spark if is is possible ? Hasn't anyone thought about it ? > > On Wed, Aug 19, 2015 at 5:50 PM, Ted Yu wrote: > >> See this thread: >> >> >> http://search-hadoop.com/m/q3RTtdZv0d1btRHl/Spark+build+module&subj=Building+Spark+Building+just+one+module+ >> >> >> >> > On Aug 19, 2015, at 1:44 AM, canan chen wrote: >> > >> > I want to work on one jira, but it is not easy to do unit test, because >> it involves different components especially UI. spark building is pretty >> slow, I don't want to build it each time to test my code change. I am >> wondering how other people do ? Is there any experience can share ? Thanks >> > >> > >> > >
Re: What's the best practice for developing new features for spark ?
Thanks Ted. I notice another thread about running spark programmatically (client mode for standalone and yarn). Would it be much easier to debug spark if is is possible ? Hasn't anyone thought about it ? On Wed, Aug 19, 2015 at 5:50 PM, Ted Yu wrote: > See this thread: > > > http://search-hadoop.com/m/q3RTtdZv0d1btRHl/Spark+build+module&subj=Building+Spark+Building+just+one+module+ > > > > > On Aug 19, 2015, at 1:44 AM, canan chen wrote: > > > > I want to work on one jira, but it is not easy to do unit test, because > it involves different components especially UI. spark building is pretty > slow, I don't want to build it each time to test my code change. I am > wondering how other people do ? Is there any experience can share ? Thanks > > > > >
Re: What's the best practice for developing new features for spark ?
See this thread: http://search-hadoop.com/m/q3RTtdZv0d1btRHl/Spark+build+module&subj=Building+Spark+Building+just+one+module+ > On Aug 19, 2015, at 1:44 AM, canan chen wrote: > > I want to work on one jira, but it is not easy to do unit test, because it > involves different components especially UI. spark building is pretty slow, I > don't want to build it each time to test my code change. I am wondering how > other people do ? Is there any experience can share ? Thanks > > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
What's the best practice for developing new features for spark ?
I want to work on one jira, but it is not easy to do unit test, because it involves different components especially UI. spark building is pretty slow, I don't want to build it each time to test my code change. I am wondering how other people do ? Is there any experience can share ? Thanks
