By the way, here is a small example of how it is done with Spark and Docker in a non-clustered environment deployment: http://blog.sequenceiq.com/blog/2015/01/09/spark-1-2-0-docker/
Looks like they also have "spark-submit" script similar to the one we are discussing in this thread. D. On Sat, Jan 17, 2015 at 6:43 AM, Dmitriy Setrakyan <[email protected]> wrote: > I understand that Fabric would be used on the user side, not on the EC2 > side. Can someone explain what advantage does Fabric give over regular SCP > call? > > On Sat, Jan 17, 2015 at 2:31 AM, Valentin Kulichenko < > [email protected]> wrote: > >> +1 for Fabric. I was looking into several tools some time ago and liked >> this one most. Very easy to use and completely suites our needs. >> On Jan 17, 2015 2:24 AM, "Sergi Vladykin" <[email protected]> >> wrote: >> >> > Personally I prefer using Fabric fabfile.org for this kind of tasks. >> It is >> > in Python and Amazon has Python API as well, so creating such a tool >> should >> > be pretty straightforward. >> > >> > regards, >> > Sergi Vladykin >> > >> > 2015-01-17 6:17 GMT+03:00 Dmitriy Setrakyan <[email protected]>: >> > >> > > I would like to start brainstorming a possibility of automating >> starting >> > > fully configured Ignite instances on AWS cloud. >> > > >> > > The problem is that simply creating an Ignite AMI and storing it on >> AWS >> > is >> > > not very convenient as users will always have to add their own >> > > configuration and JAR files to Ignite prior to its start. Ideally I >> would >> > > like Ignite EC2 instances be able to get the new configuration and >> JARs >> > > automatically without forcing users to create new images every time >> they >> > > need to change a line in the configuration file. >> > > >> > > A possible way to support it would be to create a shell script which >> will >> > > SCP the new configuration and JARs into the Ignite EC2 instances. Each >> > > Ignite EC2 instance should have a Puppet recipe which will monitor >> that >> > > either the configuration or JAR files changed and will (re)start the >> > Ignite >> > > process with new settings. >> > > >> > > To summarize, the process should be as follows: >> > > >> > > - We create Ignite AMI images for every Ignite release. >> > > - User executes Ignite EC2 shell script (let's call it ignite-ec2.sh) >> and >> > > tells it which configuration and JARs to deploy to the EC2 instances >> and >> > > how many instances to start. >> > > - Ideally we should also allow to automatically run examples and see >> the >> > > output. >> > > >> > > Please speak up if you can think of a better way to automate this >> > process. >> > > >> > > D. >> > > >> > >> > >
