I usually just use the existing launch scripts to create a correctly sized cluster then just:
1. Copy spark/conf/* to /tmp 2. rm -rf spark/ 3. Checkout and build new spark from github (you'll need to rename incubator-spark to spark) 4. Copy conf files back 5. /root/spark-ec2/copy-dir --delete /root/spark - Patrick On Sat, Jan 4, 2014 at 9:14 PM, Aureliano Buendia <[email protected]> wrote: > So I should just lanuch an AMI from one of > https://github.com/mesos/spark-ec2/tree/v2/ami-list and build the > development version on it? Is it just a simple matter of copying the target > to a directory, or does it need a lot of configurations? > > > > On Sun, Jan 5, 2014 at 5:03 AM, Patrick Wendell <[email protected]> wrote: >> >> I meant you'll need to build your own version of Spark. Typically we >> do this by launching an existing AMI and then just building a new >> version of spark and copying it to the slaves.... >> >> - Patrick >> >> On Sat, Jan 4, 2014 at 8:44 PM, Patrick Wendell <[email protected]> >> wrote: >> > You'll have to build your own. Also there are some packaging >> > differences in master (some bin/ scripts moved to sbin/) just to give >> > you a heads up. >> > >> > On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <[email protected]> >> > wrote: >> >> Good to know the next release will be on scala 2.10. >> >> >> >> Meanwhile, does using the master branch mean that I will have to build >> >> my >> >> own AMI when launching an ec2 cluster? Also, is there a nightly binary >> >> build >> >> maven repo available for spark? >> >> >> >> >> >> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <[email protected]> >> >> wrote: >> >>> >> >>> Scala 2.10.3 support was recently merged into master (#259). The >> >>> branch is >> >>> probably not as stable as 0.8.1, but things "should" work. >> >>> >> >>> The 2.10 branch should be deleted, the only issue is there are some >> >>> outstanding PRs against that branch that haven't been moved to master. >> >>> >> >>> >> >>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia >> >>> <[email protected]> >> >>> wrote: >> >>>> >> >>>> Hi, >> >>>> >> >>>> I was going to give https://github.com/scala/pickling a try on spark >> >>>> to >> >>>> see how it would compare with kryo. Unfortunately, it only works with >> >>>> scala >> >>>> 2.10.3. >> >>>> >> >>>> - Is there a time line for spark to work with scala 2.10? >> >>>> >> >>>> - Is the 2.10 branch as stable as 2.9? >> >>>> >> >>>> - What's blocking spark to work with 2.10? >> >>> >> >>> >> >> > >
