On Mon, Sep 12, 2016 at 03:26:02PM -0700, Clint Byrum wrote: :Excerpts from Mathieu Gagné's message of 2016-09-12 17:55:34 -0400:
:> I also think that small team with small deployments has little :> incentive to invest in *heavy* automation (to help themselves) and/or :> tools to delegate its operation to a 3rd party or team. Your :> deployment isn't "big enough" so you feel it's not worth the :> investment because "I can manage those just fine" or "There isn't :> enough compute nodes to automate their installation", etc. :> :> Once you hit a certain size, you need to have those in place. Without :> those tools, you will feel overwhelm by the task, feel like you are :> the only ones capable of managing/operating the infra and can't :> delegate to anyone. :> : :^^ This, so very this. Yes! Hopefully I'm preaching to the choir here, but I do still hear about people running with little to no deployment automation We basicly got this for 'free' when depolying OpenStack since we'd had automated network installation and config management for more than a decade before OpenStack existed (though not the same stack the whole time) so most of the scafolding was already done. In the immediate context of OpenStack staffing it's very hard to determine howmuch of ongoing deployment and config management work is OpenStack costs since most of it covers a much wider field. My first OpenStack deploy was Essex on Ununtu12.04 on 60 nodes. Went from 'Let's try and install OpenStack' to early adopters using it to complete a tight deadline in less than a week with just one person working on it. This was 99% because we already had Puppet setup for config managemnt and already had a customized auto deployment for Ubuntu across many different types of systems (different functions and different administrative subdomains) so adding another couple types was a well worn path. This happened to be the most mature (if I can use that word for Essex) and tested pairing at the time so it was relatively easy to leverage other people's work and plug it into our preexisting infrastructure and work flows. If I had to manually install all those nodes and scp around lovingly hand crafted configs I don't think it would ever have worked. Today in OpenStack land most distros and managemnt tools have pretty good coverage. You are never too small for automation. We have some 'one off' services that are easier to manually poke than go through updating config managemnt. These tend to drift away from reproducibility as they age and become terrible burdens to upgrade since making a replica to test on becomes guess work or sloppy 'restore form back up and make sure you fix all reference to IP and DNS name where you need ot but not where you shouldn't" I'm as guilty as anyone for this, but if you don't automate even if it's unique system you are going to have a bad time. Conversely pre-OpenStack I built up a 5 server cluster that was the core internal services for a local start up (with some Xen based virtualizaton on top so it was more than 5 servers from the outside but not many more). The first thing I built was the install/management server and used that to build the 'real services' they asked for. Now if I were a good consultant perhaps I'd have worked less efficiently to get more billing later, but I really wanted to be one and done. I was and this made hand off to their Sysadmin when they did hire one very smooth as everything that was running was recreatable and how it was created was relatively easy to inspect which mostly removes the 'one systems person with everything in their head' single point of failure. I'd guess given the small size adding the automation layers probably cost 20% of the project time -vs- just making it work by hand. But I'm confident it saved way more than that in transition costs and ongoing operations costs when they expanded, and really who wants to be just 5 servers in a closet? -Jon _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
