I manage my Mesos clusters via CloudFormation. Upgrades are pretty straightforward -- just apply an updated template (and/or parameters) to an existing stack. For now, I manually terminate instances (one at a time) and let the auto scaling groups bring up replacements with the new configuration.
This should be automated with an UpdatePolicy <http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-updatepolicy.html>, I just haven't taken the time yet to implement and test it. By the way, the templates are open source, so feel free to use or steal from them: https://github.com/thefactory/cloudformation-mesos As Dick mentioned, however, this still doesn't cover ad-hoc maintenance. But it should make provisioning much less of a headache :) On Wed, Jul 16, 2014 at 4:35 AM, Nayeem Syed <nay...@cronycle.com> wrote: > you mean ad-hoc maintenance formations wont help? > > probably should have it deployed using formations initially and use > user-data to customize the server with specific installs etc? > > > On Wed, Jul 16, 2014 at 12:15 PM, Dick Davies <d...@hellooperator.net> > wrote: > >> For provisioning yes , for ad-hoc maintenance tasks won't help at all. >> >> On 16 July 2014 11:29, Nayeem Syed <nay...@cronycle.com> wrote: >> > Thanks for those! I will give it a try to get some deployment through >> > ansible. >> > >> > I was also wondering if Cloudformation might be good for this? As it >> clears >> > up the things very cleanly when you remove the formation? Though I find >> > their JSON file very difficult to navigate and their Update Feature >> doesnt >> > seem to work too well.. >> > >> > >> > On Wed, Jul 16, 2014 at 10:46 AM, Dick Davies <d...@hellooperator.net> >> > wrote: >> >> >> >> I'd like to show you my playbooks, but unfortunately they're for a >> client >> >> - I can vouch for it being very easy to add nodes to a cluster etc. if >> you >> >> just have to edit an 'inventory' file and add IPs into the correct >> groups. >> >> >> >> (NB: puppet and chef will automate your infrastructure too, it's just >> >> they're >> >> not as useful for things like rolling deployments in my experience >> because >> >> they're agent based, so it's harder to control when each server will >> >> update and >> >> restart services). >> >> >> >> A quick Google found: >> >> >> >> >> >> >> http://blog.michaelhamrah.com/2014/06/setting-up-a-multi-node-mesos-cluster-running-docker-haproxy-and-marathon-with-ansible/ >> >> >> >> which might be useful. >> >> >> >> The play books linked from that post are for bootstrapping a cluster, >> but >> >> it's >> >> pretty simple to add a second playbook to manage rolling deploys etc. >> >> There's some Ansible examples of rolling deploys (not Mesos specific) >> >> at : >> >> >> >> http://docs.ansible.com/guide_rolling_upgrade.html >> >> >> >> >> >> On 15 July 2014 14:41, Nayeem Syed <nay...@cronycle.com> wrote: >> >> > thanks! >> >> > >> >> > do you have some examples of how you are using it with ansible? i >> dont >> >> > have >> >> > specific preferences, whatever works really. >> >> > >> >> > >> >> > On Tue, Jul 15, 2014 at 2:35 PM, Dick Davies <d...@hellooperator.net >> > >> >> > wrote: >> >> >> >> >> >> You want a rolling restart i'd guess, unless you want downtime for >> some >> >> >> reason. >> >> >> >> >> >> We use Ansible, it's pretty nice. >> >> >> >> >> >> On 15 July 2014 10:47, Nayeem Syed <nay...@cronycle.com> wrote: >> >> >> > whats the best way to update mesos master instances. eg I want to >> >> >> > update >> >> >> > things in there, install new frameworks, but at the moment I am >> >> >> > ssh'ing >> >> >> > to >> >> >> > the instances and installing them one by one. that feels wrong, >> >> >> > shouldnt >> >> >> > it >> >> >> > be done in parallel to all the instances? >> >> >> > >> >> >> > what do people currently do to keep all the masters in sync? >> >> > >> >> > >> > >> > >> > >
