Hey All, Just a couple notes. I recently posted a shell script for creating the AMI's from a clean Amazon Linux AMI.
https://github.com/mesos/spark-ec2/blob/v3/create_image.sh I think I will update the AMI's soon to get the most recent security updates. For spark-ec2's purpose this is probably sufficient (we'll only need to re-create them every few months). However, it would be cool if someone wanted to tackle providing a more general mechanism for defining Spark-friendly "images" that can be used more generally. I had thought that docker might be a good way to go for something like this - but maybe this packer thing is good too. For one thing, if we had a standard image we could use it to create containers for running Spark's unit test, which would be really cool. This would help a lot with random issues around port and filesystem contention we have for unit tests. I'm not sure if the long term place for this would be inside the spark codebase or a community library or what. But it would definitely be very valuable to have if someone wanted to take it on. - Patrick On Fri, Oct 3, 2014 at 5:20 PM, Nicholas Chammas <nicholas.cham...@gmail.com> wrote: > FYI: There is an existing issue -- SPARK-3314 > <https://issues.apache.org/jira/browse/SPARK-3314> -- about scripting the > creation of Spark AMIs. > > With Packer, it looks like we may be able to script the creation of > multiple image types (VMWare, GCE, AMI, Docker, etc...) at once from a > single Packer template. That's very cool. > > I'll be looking into this. > > Nick > > > On Thu, Oct 2, 2014 at 8:23 PM, Nicholas Chammas <nicholas.cham...@gmail.com >> wrote: > >> Thanks for the update, Nate. I'm looking forward to seeing how these >> projects turn out. >> >> David, Packer looks very, very interesting. I'm gonna look into it more >> next week. >> >> Nick >> >> >> On Thu, Oct 2, 2014 at 8:00 PM, Nate D'Amico <n...@reactor8.com> wrote: >> >>> Bit of progress on our end, bit of lagging as well. Our guy leading >>> effort got little bogged down on client project to update hive/sql testbed >>> to latest spark/sparkSQL, also launching public service so we have been bit >>> scattered recently. >>> >>> Will have some more updates probably after next week. We are planning on >>> taking our client work around hive/spark, plus taking over the bigtop >>> automation work to modernize and get that fit for human consumption outside >>> or org. All our work and puppet modules will be open sourced, documented, >>> hopefully start to rally some other folks around effort that find it useful >>> >>> Side note, another effort we are looking into is gradle tests/support. >>> We have been leveraging serverspec for some basic infrastructure tests, but >>> with bigtop switching over to gradle builds/testing setup in 0.8 we want to >>> include support for that in our own efforts, probably some stuff that can >>> be learned and leveraged in spark world for repeatable/tested infrastructure >>> >>> If anyone has any specific automation questions to your environment you >>> can drop me a line directly.., will try to help out best I can. Else will >>> post update to dev list once we get on top of our own product release and >>> the bigtop work >>> >>> Nate >>> >>> >>> -----Original Message----- >>> From: David Rowe [mailto:davidr...@gmail.com] >>> Sent: Thursday, October 02, 2014 4:44 PM >>> To: Nicholas Chammas >>> Cc: dev; Shivaram Venkataraman >>> Subject: Re: EC2 clusters ready in launch time + 30 seconds >>> >>> I think this is exactly what packer is for. See e.g. >>> http://www.packer.io/intro/getting-started/build-image.html >>> >>> On a related note, the current AMI for hvm systems (e.g. m3.*, r3.*) has >>> a bad package for httpd, whcih causes ganglia not to start. For some reason >>> I can't get access to the raw AMI to fix it. >>> >>> On Fri, Oct 3, 2014 at 9:30 AM, Nicholas Chammas < >>> nicholas.cham...@gmail.com >>> > wrote: >>> >>> > Is there perhaps a way to define an AMI programmatically? Like, a >>> > collection of base AMI id + list of required stuff to be installed + >>> > list of required configuration changes. I'm guessing that's what >>> > people use things like Puppet, Ansible, or maybe also AWS >>> CloudFormation for, right? >>> > >>> > If we could do something like that, then with every new release of >>> > Spark we could quickly and easily create new AMIs that have everything >>> we need. >>> > spark-ec2 would only have to bring up the instances and do a minimal >>> > amount of configuration, and the only thing we'd need to track in the >>> > Spark repo is the code that defines what goes on the AMI, as well as a >>> > list of the AMI ids specific to each release. >>> > >>> > I'm just thinking out loud here. Does this make sense? >>> > >>> > Nate, >>> > >>> > Any progress on your end with this work? >>> > >>> > Nick >>> > >>> > >>> > On Sun, Jul 13, 2014 at 8:53 PM, Shivaram Venkataraman < >>> > shiva...@eecs.berkeley.edu> wrote: >>> > >>> > > It should be possible to improve cluster launch time if we are >>> > > careful about what commands we run during setup. One way to do this >>> > > would be to walk down the list of things we do for cluster >>> > > initialization and see if there is anything we can do make things >>> > > faster. Unfortunately this might >>> > be >>> > > pretty time consuming, but I don't know of a better strategy. The >>> > > place >>> > to >>> > > start would be the setup.sh file at >>> > > https://github.com/mesos/spark-ec2/blob/v3/setup.sh >>> > > >>> > > Here are some things that take a lot of time and could be improved: >>> > > 1. Creating swap partitions on all machines. We could check if there >>> > > is a way to get EC2 to always mount a swap partition 2. Copying / >>> > > syncing things across slaves. The copy-dir script is called too many >>> > > times right now and each time it pauses for a few milliseconds >>> > > between slaves [1]. This could be improved by removing unnecessary >>> > > copies 3. We could make less frequently used modules like Tachyon, >>> > > persistent >>> > hdfs >>> > > not a part of the default setup. >>> > > >>> > > [1] https://github.com/mesos/spark-ec2/blob/v3/copy-dir.sh#L42 >>> > > >>> > > Thanks >>> > > Shivaram >>> > > >>> > > >>> > > >>> > > >>> > > On Sat, Jul 12, 2014 at 7:02 PM, Nicholas Chammas < >>> > > nicholas.cham...@gmail.com> wrote: >>> > > >>> > > > On Thu, Jul 10, 2014 at 8:10 PM, Nate D'Amico <n...@reactor8.com> >>> > wrote: >>> > > > >>> > > > > Starting to work through some automation/config stuff for spark >>> > > > > stack >>> > > on >>> > > > > EC2 with a project, will be focusing the work through the apache >>> > bigtop >>> > > > > effort to start, can then share with spark community directly as >>> > things >>> > > > > progress if people are interested >>> > > > >>> > > > >>> > > > Let us know how that goes. I'm definitely interested in hearing >>> more. >>> > > > >>> > > > Nick >>> > > > >>> > > >>> > >>> >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
