Re: [Openstack-operators] Openstack team size vs's deployment size

[Jonathan D. Proulx]

On Mon, Sep 12, 2016 at 03:26:02PM -0700, Clint Byrum wrote:
:Excerpts from Mathieu Gagné's message of 2016-09-12 17:55:34 -0400:

:> I also think that small team with small deployments has little
:> incentive to invest in *heavy* automation (to help themselves) and/or
:> tools to delegate its operation to a 3rd party or team. Your
:> deployment isn't "big enough" so you feel it's not worth the
:> investment because "I can manage those just fine" or "There isn't
:> enough compute nodes to automate their installation", etc.
:> 
:> Once you hit a certain size, you need to have those in place. Without
:> those tools, you will feel overwhelm by the task, feel like you are
:> the only ones capable of managing/operating the infra and can't
:> delegate to anyone.
:> 
:
:^^ This, so very this.

Yes! Hopefully I'm preaching to the choir here, but I do still hear
about people running with little to no deployment automation 

We basicly got this for 'free' when depolying OpenStack since we'd had
automated network installation and config management for more than a
decade before OpenStack existed (though not the same stack the whole
time) so most of the scafolding was already done. In the immediate
context of OpenStack staffing it's very hard to determine howmuch of
ongoing deployment and config management work is OpenStack costs since
most of it covers a much wider field.

My first OpenStack deploy was Essex on Ununtu12.04 on 60 nodes. Went
from 'Let's try and install OpenStack' to early adopters using it to
complete a tight deadline in less than a week with just one person
working on it.  

This was 99% because we already had Puppet setup for config managemnt
and already had a customized auto deployment for Ubuntu across many
different types of systems (different functions and different
administrative subdomains) so adding another couple types was a well
worn path.

This happened to be the most mature (if I can use that word for Essex)
and tested pairing at the time so it was relatively easy to leverage
other people's work and plug it into our preexisting infrastructure
and work flows. If I had to manually install all those nodes and scp
around lovingly hand crafted configs I don't think it would ever have
worked. Today in OpenStack land most distros and managemnt tools have
pretty good coverage.

You are never too small for automation.  

We have some 'one off' services that are easier to manually poke than
go through updating config managemnt. These tend to drift away from
reproducibility as they age and become terrible burdens to upgrade
since making a replica to test on becomes guess work or sloppy
'restore form back up and make sure you fix all reference to IP and
DNS name where you need ot but not where you shouldn't"

I'm as guilty as anyone for this, but if you don't automate even if
it's unique system you are going to have a bad time.

Conversely pre-OpenStack I built up a 5 server cluster that was the
core internal services for a local start up (with some Xen based
virtualizaton on top so it was more than 5 servers from the outside
but not many more). The first thing I built was the install/management
server and used that to build the 'real services' they asked for.  Now
if I were a good consultant perhaps I'd have worked less efficiently
to get more billing later, but I really wanted to be one and done.  I
was and this made hand off to their Sysadmin when they did hire one
very smooth as everything that was running was recreatable and how it
was created was relatively easy to inspect which mostly removes the
'one systems person with everything in their head' single point of
failure.

I'd guess given the small size adding the automation layers probably
cost 20% of the project time -vs- just making it work by hand.  But
I'm confident it saved way more than that in transition costs and
ongoing operations costs when they expanded, and really who wants to
be just 5 servers in a closet?

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[Clint Byrum]

Excerpts from Mathieu Gagné's message of 2016-09-12 17:55:34 -0400:
> On Fri, Sep 9, 2016 at 6:41 PM, Mathieu Gagné  wrote:
> > On Wed, Sep 7, 2016 at 6:59 PM, Kris G. Lindgren  
> > wrote:
> >>
> >> I was hoping to poll other operators to see what their average team size
> >> vs’s deployment size is, as I am trying to use this in an internal company
> >> discussion.
> >
> > It is difficult to come up with numbers without context as not all
> > team are equally created. But I think we are currently facing the same
> > situation where you feel the team can't keep up with the amount of
> > work.
> 
> I also think that small team with small deployments has little
> incentive to invest in *heavy* automation (to help themselves) and/or
> tools to delegate its operation to a 3rd party or team. Your
> deployment isn't "big enough" so you feel it's not worth the
> investment because "I can manage those just fine" or "There isn't
> enough compute nodes to automate their installation", etc.
> 
> Once you hit a certain size, you need to have those in place. Without
> those tools, you will feel overwhelm by the task, feel like you are
> the only ones capable of managing/operating the infra and can't
> delegate to anyone.
> 

^^ This, so very this.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[Mathieu Gagné]

On Fri, Sep 9, 2016 at 6:41 PM, Mathieu Gagné  wrote:
> On Wed, Sep 7, 2016 at 6:59 PM, Kris G. Lindgren  
> wrote:
>>
>> I was hoping to poll other operators to see what their average team size
>> vs’s deployment size is, as I am trying to use this in an internal company
>> discussion.
>
> It is difficult to come up with numbers without context as not all
> team are equally created. But I think we are currently facing the same
> situation where you feel the team can't keep up with the amount of
> work.

I also think that small team with small deployments has little
incentive to invest in *heavy* automation (to help themselves) and/or
tools to delegate its operation to a 3rd party or team. Your
deployment isn't "big enough" so you feel it's not worth the
investment because "I can manage those just fine" or "There isn't
enough compute nodes to automate their installation", etc.

Once you hit a certain size, you need to have those in place. Without
those tools, you will feel overwhelm by the task, feel like you are
the only ones capable of managing/operating the infra and can't
delegate to anyone.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[Fox, Kevin M]

I'd also add it depends on feature set of the cloud. If you have extra 
services, or your users keep asking for more and more openstack features to be 
added to the cloud (dnsaas, dbaas, hadoopaas, coeaas,) then the ratio is much 
higher then say, with just a basic cloud with vmaas & naas.

Thanks,
Kevin

From: Clint Byrum [cl...@fewbar.com]
Sent: Monday, September 12, 2016 2:26 PM
To: openstack-operators
Subject: Re: [Openstack-operators] Openstack team size vs's deployment size

Excerpts from gustavo panizzo (gfa)'s message of 2016-09-09 22:07:49 +0800:
> On Thu, Sep 08, 2016 at 03:52:42PM +, Kris G. Lindgren wrote:
> > I completely agree about the general rule of thumb.  I am only looking at 
> > the team that specifically supports openstack.  For us frontend support for 
> > public clouds is handled by another team/org all together.
>
> in my previous job the ratio was 1 openstack guy / 300 prod hv and
> ~ 50 non prod hypervisors (non prod clouds).
>
> we had 5 different clouds, 2 biggest clouds shared keystone and
> glance (same dc, different buildings, almost lan latency). the biggest cloud 
> had 2 regions
> (different connectivity on same dc building)
>
> a different team took care of the underlying hw, live migrations (when
> necessary but usually escalated to the openstack team) and install the
> computes running a single salt stack command. another team developed a
> in-house horizon replacement
>
> that job definitively burned me, i'd say that the sweet spot is about
> 1 person every 200 hv, but if your regions are very similar and you have
> control of the whole stack (i didn't) i think 1 person every 300 hv is
> doable
>
> we only used nova, neutron (provider networks), glance and keystone.
>

This ratio is entirely dependent on the point of the cloud, and where
one's priorities lie.

If you have a moderately sized private cloud (< 200 hv's) with no AZ's,
and 1 region, and an uptime SLA of 99.99%, I'd expect 1 FTE would be
plenty. But larger clouds that expect to continuously grow should try to
drive it down as low as possible so that value can be extracted from the
equipment in as small a batch size as possible. The higher that ratio,
the more value we're getting from the servers themselves.

Basically, I'd expect this to scale logarithmically, with clouds in the
1 - 500 hv range being similar in total cost, but everything above that
leveling off in total cost growth, but continuing a more or less linear
path up in value proposition. The only way we get there is to attack
complexity with a vengeance.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[Clint Byrum]

Excerpts from gustavo panizzo (gfa)'s message of 2016-09-09 22:07:49 +0800:
> On Thu, Sep 08, 2016 at 03:52:42PM +, Kris G. Lindgren wrote:
> > I completely agree about the general rule of thumb.  I am only looking at 
> > the team that specifically supports openstack.  For us frontend support for 
> > public clouds is handled by another team/org all together.
> 
> in my previous job the ratio was 1 openstack guy / 300 prod hv and
> ~ 50 non prod hypervisors (non prod clouds).
> 
> we had 5 different clouds, 2 biggest clouds shared keystone and
> glance (same dc, different buildings, almost lan latency). the biggest cloud 
> had 2 regions
> (different connectivity on same dc building)
> 
> a different team took care of the underlying hw, live migrations (when
> necessary but usually escalated to the openstack team) and install the
> computes running a single salt stack command. another team developed a
> in-house horizon replacement
> 
> that job definitively burned me, i'd say that the sweet spot is about
> 1 person every 200 hv, but if your regions are very similar and you have
> control of the whole stack (i didn't) i think 1 person every 300 hv is
> doable
> 
> we only used nova, neutron (provider networks), glance and keystone.
> 

This ratio is entirely dependent on the point of the cloud, and where
one's priorities lie.

If you have a moderately sized private cloud (< 200 hv's) with no AZ's,
and 1 region, and an uptime SLA of 99.99%, I'd expect 1 FTE would be
plenty. But larger clouds that expect to continuously grow should try to
drive it down as low as possible so that value can be extracted from the
equipment in as small a batch size as possible. The higher that ratio,
the more value we're getting from the servers themselves.

Basically, I'd expect this to scale logarithmically, with clouds in the
1 - 500 hv range being similar in total cost, but everything above that
leveling off in total cost growth, but continuing a more or less linear
path up in value proposition. The only way we get there is to attack
complexity with a vengeance.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[Silence Dogood]

I want desperately to see a failed deployments talk at summit.  I'd be glad
to contribute but we'd need info on a variety of failure states.

On Sep 12, 2016 1:05 PM, "Jonathan D. Proulx"  wrote:

>
> I agree this would make a very interesting OPs session.
>
> As many have poitned out it's difficult to really quantify in a
> comparable way given the range of roles and approches.
>
> We have 1 region (going on 2) with 100 hypervisors providing a basic
> IaaS service (agian looking to expand into various XaaS overlays).
> Typically running 600-900 VMs in support of several hundred users &
> some randomly shifting number of projects (164 had instances run in
> the past month, though some were only single test instances).
>
> We have zero dedicate staff for OpenStack so coming upt with FTE
> numbers is guess work at best.
>
> For front end user support, answering questions and using the admin
> interface for adds and changes we probably spend about 0.5 FTE (Across
> a team of 6-7 people)
>
> For backend stuff for both OpenStack and Ceph probably 1 FTE, maybe a
> bit more, mostly handled by 2 individuals also included in the count
> above.
>
> This is workable in ideal conditions but far too thin and fragile in
> real conditions.  One of the two core OpenStack people moved and it
> took 6mo to get a replacemnet. This made us loose an Upgrade cycle as
> there was insufficient staffing to test and implement an upgrate to
> Liberty during the gap.  This along with otehr bits that didn't happen
> accumulated a lot of technical debt in a short time.  We are digging
> out but 1 year later we'll be back at current software version but
> with zero progress on previously planned service expantions.
>
> -Jon
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[Jonathan D. Proulx]

I agree this would make a very interesting OPs session.

As many have poitned out it's difficult to really quantify in a
comparable way given the range of roles and approches.

We have 1 region (going on 2) with 100 hypervisors providing a basic
IaaS service (agian looking to expand into various XaaS overlays).
Typically running 600-900 VMs in support of several hundred users &
some randomly shifting number of projects (164 had instances run in
the past month, though some were only single test instances).

We have zero dedicate staff for OpenStack so coming upt with FTE
numbers is guess work at best.

For front end user support, answering questions and using the admin
interface for adds and changes we probably spend about 0.5 FTE (Across
a team of 6-7 people)

For backend stuff for both OpenStack and Ceph probably 1 FTE, maybe a
bit more, mostly handled by 2 individuals also included in the count
above.

This is workable in ideal conditions but far too thin and fragile in
real conditions.  One of the two core OpenStack people moved and it
took 6mo to get a replacemnet. This made us loose an Upgrade cycle as
there was insufficient staffing to test and implement an upgrate to
Liberty during the gap.  This along with otehr bits that didn't happen
accumulated a lot of technical debt in a short time.  We are digging
out but 1 year later we'll be back at current software version but
with zero progress on previously planned service expantions.

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[Mathieu Gagné]

On Wed, Sep 7, 2016 at 6:59 PM, Kris G. Lindgren  wrote:
>
> I was hoping to poll other operators to see what their average team size
> vs’s deployment size is, as I am trying to use this in an internal company
> discussion.

It is difficult to come up with numbers without context as not all
team are equally created. But I think we are currently facing the same
situation where you feel the team can't keep up with the amount of
work. I will describe our life as the operational "OpenStack" team and
how we plan on address our issues. Maybe you will find useful
information.

Our team is composed of ~ 7 people. This team is responsible for
everything "IT" except "corporative" services such as Exchange, Active
directory or telephony.
We also have multiple development teams, one being more or less
dedicated to everything related to baremetal: network automation,
Ironic development, etc.
We also have a dedicated network team, hands and eyes,
assembly/provisioning team, etc. (we don't wire/rack the servers
ourselves)

So if you ask "How many people per compute nodes?", as you can see,
the number might be very low as we aren't 100% dedicated to OpenStack.

In our team, each member has a set of competences/specialities.
(hardware, backup, monitoring, OpenStack, HA and more) This means
there are "experts" in our team which are responsible for those
domains of expertises. This also means that there might be only one
guy knowing how to package OpenStack or one fully understanding how
the logging system is installed.

We have a lot of OpenStack related tasks:
* Maintain the "deployer" (mainly written in Puppet)
* OpenStack packages (based on Ubuntu packages)
* OpenStack bugs fixing and features
* Debug delivery issues (baremetal, performance, misconfiguration, etc.)
* Capacity management (new compute nodes, storage nodes)
* New deployments (regions)
* Maintain other related systems such as monitoring, logging, etc.
* POC with new OpenStack projects

Our team also happens to be exposed to multiple stakeholders who all
want us to work on their needs/problems right away. This includes
other developers, network team, customer support, provisioning, and
many more. It is difficult to prioritize all those requests. This
causes a lot of planning challenges which we have yet to fully
address. This is something we will be working on very shortly.

The main challenge that comes up from all this is difficulty to keep
up with technical debt payment.

Technical debt makes everything above much more difficult:
* your logging system is not setup as you would like it, rendering it useless
* your monitoring system has stability issues which you can't find
time to address, resulting in even less sleep.
* upgrading OpenStack becomes a task you postpone as long as you can
due to other higher priorities

So you want to address those to make your life easier and hopefully
improve your life quality.

Based on my experience, the business will never give you 6 months
full-time to fix your technical debts. They will want features, new
regions, more capacity, etc. And operational problems still need to be
addressed right away when they happen.

What we are going through now is putting on paper our pain points
(technical debts) and communicate those to upper management. As long
as you cannot quantify and describe your "misery" (technical debts),
it's hard for the business to justify hiring new people, offer you the
opportunity to work on your internal tools or fully understand your
situation.

To summarize:
* Identify why you would want more people (technical debts, too many
ops tasks, etc.)
* See if your team can address those issues themselves
* Or use those info to justify hiring more people to work on those items
* Or delegate operational tasks to other teams if possible
* Or build tools to make the above possible.

Cheers

--
Mathieu

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[gustavo panizzo (gfa)]

On Thu, Sep 08, 2016 at 03:52:42PM +, Kris G. Lindgren wrote:
> I completely agree about the general rule of thumb.  I am only looking at the 
> team that specifically supports openstack.  For us frontend support for 
> public clouds is handled by another team/org all together.

in my previous job the ratio was 1 openstack guy / 300 prod hv and
~ 50 non prod hypervisors (non prod clouds).

we had 5 different clouds, 2 biggest clouds shared keystone and
glance (same dc, different buildings, almost lan latency). the biggest cloud 
had 2 regions
(different connectivity on same dc building)

a different team took care of the underlying hw, live migrations (when
necessary but usually escalated to the openstack team) and install the
computes running a single salt stack command. another team developed a
in-house horizon replacement

that job definitively burned me, i'd say that the sweet spot is about
1 person every 200 hv, but if your regions are very similar and you have
control of the whole stack (i didn't) i think 1 person every 300 hv is
doable

we only used nova, neutron (provider networks), glance and keystone.


--
1AE0 322E B8F7 4717 BDEA BF1D 44BB 1BA7 9F6C 6333

keybase: https://keybase.io/gfa

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[Tim Bell]

It is a difficult calculation to make for me:


-  Shared services like Facilities/Hardware repair/Network…

-  Expectation for new functions since upstream has made them available

-  The user support effort has increased significantly as more users 
come online

-  Application cloud-readiness is another factor, are the applications 
able to handle infrastructure outages.

Would be a very interesting topic for an Ops session at Barcelona and I suspect 
the larger deployments are aiming to approach O(1) staffing, i.e. no problem to 
add another X% resources to the cloud with only a <<X% increase in staff but 
understanding the best practices and some advice (such as < 1 FTE will be 
challenging) would be helpful.

Tim

From: "Kris G. Lindgren" <klindg...@godaddy.com>
Date: Thursday 8 September 2016 at 17:52
To: "Van Leeuwen, Robert" <rovanleeu...@ebay.com>, openstack-operators 
<openstack-operators@lists.openstack.org>
Subject: Re: [Openstack-operators] Openstack team size vs's deployment size

I completely agree about the general rule of thumb.  I am only looking at the 
team that specifically supports openstack.  For us frontend support for public 
clouds is handled by another team/org all together.

So what I am looking at is the people who are in charge of the care/feeding of 
the openstack system specifically.
Which means: people who do the dev work on openstack, community participation, 
the integrations work, the capacity monitoring and server additions, the 
responding to alarms/monitoring around openstack, the POC of new features.  Any 
automation work specifically around openstack.  Any testing specifically done 
against openstack.

For us we have both private and public clouds in 4 different regions.
We try to maintain either N or N-1 release cadence.
We build our own packages.
We have lots of existing legacy systems that we need to integrate with.
We have a number of local patches that we carry (the majority around cellsv1).

So given that would you be willing to share your compute node/engineer ratio?

___
Kris Lindgren
Senior Linux Systems Engineer
GoDaddy

From: "Van Leeuwen, Robert" <rovanleeu...@ebay.com>
Date: Thursday, September 8, 2016 at 1:33 AM
To: "Kris G. Lindgren" <klindg...@godaddy.com>, OpenStack Operators 
<openstack-operators@lists.openstack.org>
Subject: Re: [Openstack-operators] Openstack team size vs's deployment size

> I was hoping to poll other operators to see what their average team size vs’s 
> deployment size is,
>  as I am trying to use this in an internal company discussion.
> Right now we are in the order of ~260 Compute servers per Openstack 
> Dev/Engineer.
> So trying to see how we compare with other openstack installs, particularly 
> those running with more than a few hundred compute nodes.

In my opinion it highly depends on too many things to have general rule of 
thumb.
Just a few things that I think would impact required team size:
* How many regions you have: setting up and managing a region usually takes 
more time then adding computes to an existing region
* How often do you want/need to upgrade
* Are you offering more then “core IAAS services” e.g. designate/trove/…
* What supporting things do you need around your cloud and who manage e.g. 
networking, setting up dns / repositories / authentication systems  etc
* What kind of SDN are you using/ how it needs to be integrated existing 
networks
* What kind of hardware you are rolling and what is the average size of the 
instances. E.G. hosting 1000 tiny instances on a 768GB / 88 core hypervisor 
will probably create more support tickets then 10 large instances on a low-spec 
hypervisor.
* How you handle storage ceph/san/local?
* Do you need live-migration when doing maintenance or are you allowed to bring 
down an availability zone
* Are you building your own packages / Using vendor packages
* The level of support the users expect and which team is taking care of that

In my private cloud experience rolling compute nodes and the controllers are 
not the bulk of the work.
The time goes in all the things that you need around the cloud and 
customizations that takes time.

It might be a bit different for public cloud providers where you might deliver 
as-is and do not need any integrations.
But you might need other things like very reliable billing and good automation 
around misbehaving users.


Cheer,
Robert van Leeuwen
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[Kris G. Lindgren]

I completely agree about the general rule of thumb.  I am only looking at the 
team that specifically supports openstack.  For us frontend support for public 
clouds is handled by another team/org all together.

So what I am looking at is the people who are in charge of the care/feeding of 
the openstack system specifically.
Which means: people who do the dev work on openstack, community participation, 
the integrations work, the capacity monitoring and server additions, the 
responding to alarms/monitoring around openstack, the POC of new features.  Any 
automation work specifically around openstack.  Any testing specifically done 
against openstack.

For us we have both private and public clouds in 4 different regions.
We try to maintain either N or N-1 release cadence.
We build our own packages.
We have lots of existing legacy systems that we need to integrate with.
We have a number of local patches that we carry (the majority around cellsv1).

So given that would you be willing to share your compute node/engineer ratio?

___
Kris Lindgren
Senior Linux Systems Engineer
GoDaddy

From: "Van Leeuwen, Robert" <rovanleeu...@ebay.com>
Date: Thursday, September 8, 2016 at 1:33 AM
To: "Kris G. Lindgren" <klindg...@godaddy.com>, OpenStack Operators 
<openstack-operators@lists.openstack.org>
Subject: Re: [Openstack-operators] Openstack team size vs's deployment size

> I was hoping to poll other operators to see what their average team size vs’s 
> deployment size is,
>  as I am trying to use this in an internal company discussion.
> Right now we are in the order of ~260 Compute servers per Openstack 
> Dev/Engineer.
> So trying to see how we compare with other openstack installs, particularly 
> those running with more than a few hundred compute nodes.

In my opinion it highly depends on too many things to have general rule of 
thumb.
Just a few things that I think would impact required team size:
* How many regions you have: setting up and managing a region usually takes 
more time then adding computes to an existing region
* How often do you want/need to upgrade
* Are you offering more then “core IAAS services” e.g. designate/trove/…
* What supporting things do you need around your cloud and who manage e.g. 
networking, setting up dns / repositories / authentication systems  etc
* What kind of SDN are you using/ how it needs to be integrated existing 
networks
* What kind of hardware you are rolling and what is the average size of the 
instances. E.G. hosting 1000 tiny instances on a 768GB / 88 core hypervisor 
will probably create more support tickets then 10 large instances on a low-spec 
hypervisor.
* How you handle storage ceph/san/local?
* Do you need live-migration when doing maintenance or are you allowed to bring 
down an availability zone
* Are you building your own packages / Using vendor packages
* The level of support the users expect and which team is taking care of that

In my private cloud experience rolling compute nodes and the controllers are 
not the bulk of the work.
The time goes in all the things that you need around the cloud and 
customizations that takes time.

It might be a bit different for public cloud providers where you might deliver 
as-is and do not need any integrations.
But you might need other things like very reliable billing and good automation 
around misbehaving users.


Cheer,
Robert van Leeuwen
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[Van Leeuwen, Robert]

> I was hoping to poll other operators to see what their average team size vs’s 
> deployment size is,
>  as I am trying to use this in an internal company discussion.
> Right now we are in the order of ~260 Compute servers per Openstack 
> Dev/Engineer.
> So trying to see how we compare with other openstack installs, particularly 
> those running with more than a few hundred compute nodes.

In my opinion it highly depends on too many things to have general rule of 
thumb.
Just a few things that I think would impact required team size:
* How many regions you have: setting up and managing a region usually takes 
more time then adding computes to an existing region
* How often do you want/need to upgrade
* Are you offering more then “core IAAS services” e.g. designate/trove/…
* What supporting things do you need around your cloud and who manage e.g. 
networking, setting up dns / repositories / authentication systems  etc
* What kind of SDN are you using/ how it needs to be integrated existing 
networks
* What kind of hardware you are rolling and what is the average size of the 
instances. E.G. hosting 1000 tiny instances on a 768GB / 88 core hypervisor 
will probably create more support tickets then 10 large instances on a low-spec 
hypervisor.
* How you handle storage ceph/san/local?
* Do you need live-migration when doing maintenance or are you allowed to bring 
down an availability zone
* Are you building your own packages / Using vendor packages
* The level of support the users expect and which team is taking care of that

In my private cloud experience rolling compute nodes and the controllers are 
not the bulk of the work.
The time goes in all the things that you need around the cloud and 
customizations that takes time.

It might be a bit different for public cloud providers where you might deliver 
as-is and do not need any integrations.
But you might need other things like very reliable billing and good automation 
around misbehaving users.


Cheer,
Robert van Leeuwen
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[xav]

On Wed, 2016-09-07 at 22:59 +, Kris G. Lindgren wrote:
> Hello all,
> 
>  
> 
> I was hoping to poll other operators to see what their average team
> size vs’s deployment size is, as I am trying to use this in an
> internal company discussion.  Right now we are in the order of ~260
> Compute servers per Openstack Dev/Engineer.  So trying to see how we
> compare with other openstack installs, particularly those running with
> more than a few hundred compute nodes.
> 
>  
Similar discussions for me, but at the opposite end of the scale.

I think a large part of that discussion is around the demarcation
between operator tasks and sysadmin tasks.  If ops are doing purely
OpenStack work or just as an escalation for an operations center, and
others handle the network and OS, then it's a different story. Our
current team is close to 1:10 FTE:compute, but at such a small scale
there's a practical minimum for keeping the lights on regardless of the
scale.  Devs are a separate picture, currently double the number of ops,
and there's customer support/sales folks too.

We're about to open a fresh region and that will load the ops well
beyond capacity, so are hiring another.  The extra region alone doesn't
justify the hire, but the income allows us to do so and eventually we'll
hit a critical mass.  I really don't want to derail the conversation,
but would also love to hear about smaller deployments and how that
balance works.
> 
> ___
> 
> 
> Kris Lindgren
> 
> 
> Senior Linux Systems Engineer
> 
> 
> GoDaddy
> 
> 
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators




___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack team size vs's deployment size

[Silence Dogood]

early days was 2 full time 2 part time for a cluster size of a couple
hundred.

On Wed, Sep 7, 2016 at 6:59 PM, Kris G. Lindgren 
wrote:

> Hello all,
>
>
>
> I was hoping to poll other operators to see what their average team size
> vs’s deployment size is, as I am trying to use this in an internal company
> discussion.  Right now we are in the order of ~260 Compute servers per
> Openstack Dev/Engineer.  So trying to see how we compare with other
> openstack installs, particularly those running with more than a few hundred
> compute nodes.
>
>
>
> ___
>
> Kris Lindgren
>
> Senior Linux Systems Engineer
>
> GoDaddy
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators