Hi, Sorry for the late reply, this kind of slipped under my radar.
On 01/13/2014 12:12 PM, Aureliano Buendia wrote: > On Mon, Jan 13, 2014 at 5:59 PM, Josh Rosen <[email protected]> wrote: > >> If you'd like to use Spark with Docker, the AMPLab's Docker scripts might >> be a nice starting point: >> > > Is it used in production? Not as far as I know. But I believe that quite a few people have used it to try out Spark quickly without using any cloud provider. Also someone apparently used it successfully on Azure: http://govindkanshi.wordpress.com/2013/12/09/spark-on-azure-using-docker-works/ Note that the scripts assume that the cluster runs on a single fat box. Before adding inter-node setup inside the scripts (via iptables, route, etc.) I would wait a bit for the next few Docker releases. I have heard that there are some exciting things coming up that would make this much easier to implement. Andre >> >> https://amplab.cs.berkeley.edu/2013/10/23/got-a-minute-spin-up-a-spark-cluster-on-your-laptop-with-docker/ >> https://github.com/amplab/docker-scripts >> >> >> On Mon, Jan 13, 2014 at 8:51 AM, Mayur Rustagi >> <[email protected]>wrote: >> >>> They are in active development but themselves ask not be used in >>> production.I can create one to play around I guess. >>> Regards >>> Mayur >>> >>> Mayur Rustagi >>> Ph: +919632149971 >>> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com >>> https://twitter.com/mayur_rustagi >>> >>> >>> >>> On Mon, Jan 13, 2014 at 9:40 PM, Aureliano Buendia >>> <[email protected]>wrote: >>> >>>> Why not using docker for spark? GCE recently supports custom kernels >>>> allowing docker to work on it. >>>> >>>> A docket image can be shared between GCE and EC2. >>>> >>>> >>>> On Mon, Jan 13, 2014 at 8:56 AM, Mayur Rustagi >>>> <[email protected]>wrote: >>>> >>>>> Hi, >>>>> I have setup a basic guide here. >>>>> >>>>> >>>>> http://docs.sigmoidanalytics.com/index.php/How_to_Install_Spark_on_Google_Compute_Engine >>>>> >>>>> GCE unfortunately doesnt provide a easy way to manage images( It has >>>>> interface to download the tar.gz image and then reload it, I didnt find it >>>>> as convinient as AMI). If you create one let me know and I'll update the >>>>> guide accordingly. If you create a sharable image let me know i'll host it >>>>> on S3 and make it available for all. >>>>> >>>>> Regards >>>>> Mayur >>>>> >>>>> >>>>> Mayur Rustagi >>>>> Ph: +919632149971 >>>>> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com >>>>> https://twitter.com/mayur_rustagi >>>>> >>>>> >>>>> >>>>> On Mon, Jan 13, 2014 at 11:01 AM, Debasish Das < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Aureliano, >>>>>> >>>>>> Look for google compute engine scripts from typesafe repo. They >>>>>> recently tested Akka Cluster on 2400 nodes from Google Compute Engine. >>>>>> You >>>>>> should be able to reuse the scripts. >>>>>> >>>>>> Thanks. >>>>>> Deb >>>>>> >>>>>> >>>>>> On Sun, Jan 12, 2014 at 8:00 PM, Aureliano Buendia < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Has anyone worked on a script similar to spark-ec2 for google compute >>>>>>> engine? >>>>>>> >>>>>>> Google compute engine claims that they have faster instance start up >>>>>>> time, and that together with by minute charging makes it a desirable >>>>>>> choice >>>>>>> for spark. >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
