Re: [OpenStack-Infra] opensource infra: server sizes

[Donny Davis]

>If they're not listed anywhere, I suppose that nobody follows up on the
sizing until something breaks?

I have been introspecting the infra mirror on FortNebula to determine if
there is any last little bit of performance to be gained.
Its also publicly tracked here

https://grafana.fortnebula.com/d/9MMqh8HWk/openstack-utilization



On Mon, Aug 19, 2019 at 12:57 PM Clark Boylan  wrote:

> On Mon, Aug 19, 2019, at 9:36 AM, Shadi Akiki wrote:
> > > the sizing details for control plane servers are not really listed
> anywhere
> >
> > If they're not listed anywhere, I suppose that nobody follows up on the
> > sizing until something breaks?
> >
> > > Sometimes the cloud provider will provide us with custom flavors, or
> ask us to use a particular variant
> >
> > Why would they ask for a particular variant? Is it because these
> > resources are donated by the cloud provider?
> > That's my best guess to justify this.
>
> Yes, resources are donated by the cloud providers. I think providers tend
> to use special flavors to control how are resources are scheduled.
>
> >
> > > for something not load-balanced that requires production downtime,
> it's a very manual process
> >
> > Is it also the case that the load-balancing settings are not recorded
> > anywhere?
> > eg minimum and maximum number of machines in a load-balancing cluster,
> > machine flavor
>
> Our load balancers don't currently do auto scaling. But configs do live in
> config management. For example:
> https://opendev.org/opendev/system-config/src/branch/master/playbooks/group_vars/gitea-lb.yaml
>
> There are longer term plans to host that service in kubernetes which could
> make use of scaling, but gitea the software isn't capable of running in  a
> true cluster yet.
>
> >
> > > It's just a trade-off between development time to create and maintain
> > > that, and how often we actually start control-plane servers.
> >
> > Are the control-plane servers the only cloud cost aspect to outweigh
> > the development costs?
> > I'm surprised there isn't already a tool out there that interfaces with
> > rrdtool and/or cacti to help with this.
> > rrdtool seems to have been around since 20 years now [1] [2]
>
> The expectation from cacti is likely that you'll use the same snmp MIB
> data that cacti uses to build its graphs to query sizing info directly  if
> you want it. Rather than expecting cacti or rrdtool to expose that to you.
>
> >
> > [1] https://tobi.oetiker.ch/webtools/appreciators.txt
> > [2]
> >
> https://github.com/oetiker/rrdtool-1.x/commit/37fc663811528ddf3ded4fe236ea26f4f76fa32d#diff-dee0aab09da2b4d69b6722a85037700d
> > --
> > Shadi Akiki
> > Founder & CEO, AutofitCloud
> > https://autofitcloud.com/
> > +1 813 579 4935
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] opensource infra: server sizes

[Clark Boylan]

On Mon, Aug 19, 2019, at 9:36 AM, Shadi Akiki wrote:
> > the sizing details for control plane servers are not really listed anywhere
> 
> If they're not listed anywhere, I suppose that nobody follows up on the 
> sizing until something breaks?
> 
> > Sometimes the cloud provider will provide us with custom flavors, or ask us 
> > to use a particular variant
> 
> Why would they ask for a particular variant? Is it because these 
> resources are donated by the cloud provider?
> That's my best guess to justify this.

Yes, resources are donated by the cloud providers. I think providers tend to 
use special flavors to control how are resources are scheduled.

> 
> > for something not load-balanced that requires production downtime, it's a 
> > very manual process
> 
> Is it also the case that the load-balancing settings are not recorded 
> anywhere?
> eg minimum and maximum number of machines in a load-balancing cluster, 
> machine flavor

Our load balancers don't currently do auto scaling. But configs do live in 
config management. For example: 
https://opendev.org/opendev/system-config/src/branch/master/playbooks/group_vars/gitea-lb.yaml

There are longer term plans to host that service in kubernetes which could make 
use of scaling, but gitea the software isn't capable of running in  a true 
cluster yet.

> 
> > It's just a trade-off between development time to create and maintain
> > that, and how often we actually start control-plane servers.
> 
> Are the control-plane servers the only cloud cost aspect to outweigh 
> the development costs?
> I'm surprised there isn't already a tool out there that interfaces with 
> rrdtool and/or cacti to help with this.
> rrdtool seems to have been around since 20 years now [1] [2]

The expectation from cacti is likely that you'll use the same snmp MIB data 
that cacti uses to build its graphs to query sizing info directly  if you want 
it. Rather than expecting cacti or rrdtool to expose that to you.

> 
> [1] https://tobi.oetiker.ch/webtools/appreciators.txt
> [2] 
> https://github.com/oetiker/rrdtool-1.x/commit/37fc663811528ddf3ded4fe236ea26f4f76fa32d#diff-dee0aab09da2b4d69b6722a85037700d
> --
> Shadi Akiki
> Founder & CEO, AutofitCloud
> https://autofitcloud.com/
> +1 813 579 4935

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] opensource infra: server sizes

[Shadi Akiki]

>
> the sizing details for control plane servers are not really listed anywhere


If they're not listed anywhere, I suppose that nobody follows up on the
sizing until something breaks?

Sometimes the cloud provider will provide us with custom flavors, or ask us
> to use a particular variant


Why would they ask for a particular variant? Is it because these resources
are donated by the cloud provider?
That's my best guess to justify this.

for something not load-balanced that requires production downtime, it's a
> very manual process


Is it also the case that the load-balancing settings are not recorded
anywhere?
eg minimum and maximum number of machines in a load-balancing cluster,
machine flavor

It's just a trade-off between development time to create and maintain
> that, and how often we actually start control-plane servers.
>

Are the control-plane servers the only cloud cost aspect to outweigh the
development costs?
I'm surprised there isn't already a tool out there that interfaces with
rrdtool and/or cacti to help with this.
rrdtool seems to have been around since 20 years now [1] [2]

[1]  https://tobi.oetiker.ch/webtools/appreciators.txt
[2]
https://github.com/oetiker/rrdtool-1.x/commit/37fc663811528ddf3ded4fe236ea26f4f76fa32d#diff-dee0aab09da2b4d69b6722a85037700d
--
Shadi Akiki
Founder & CEO, AutofitCloud
https://autofitcloud.com/
+1 813 579 4935


On Thu, Aug 15, 2019 at 3:15 AM Ian Wienand  wrote:

> On Tue, Aug 13, 2019 at 11:44:36AM +0200, Shadi Akiki wrote:
> > 2- how the allocated resource can be downsized (which I was hoping to
> find
> > in the opendev/system-config 
> >  repo)
>
> You are correct that the sizing details for control plane servers are
> not really listed anywhere.
>
> This is really an artifact of us manually creating control-plane
> servers.  When we create a new control-plane server, we use the launch
> tooling in [1] where you will see we manually select a flavor size.
> This is dependent on the cloud we launch the server in and the flavors
> they provide us.
>
> There isn't really a strict rule on what flavor is chosen; it's more
> art than science :) Basically the smallest for what seems appropriate
> for what the server is doing.
>
> After the server is created the exact flavor used is not recorded
> separately (i.e. other than querying nova directly).  So there is no
> central YAML file or anything with the server and the flavor it was
> created with.  Sometimes the cloud provider will provide us with
> custom flavors, or ask us to use a particular variant.
>
> So in terms of resizing the servers, we are limited to the flavors
> provided to us by the providers, which varies.  In terms of the
> practicality of resizing, as I'm sure you know this can be harder or
> easier depending on a big variety of things from the provider.  We
> have resized servers before when it becomes clear they're not
> performing (recently adding swap to the gitea servers comes to mind).
> Depending on the type of service it varies; for something not
> load-balanced that requires production downtime, it's a very manual
> process.
>
> Nobody is opposed to making any of this more programatic, I'm sure.
> It's just a trade-off between development time to create and maintain
> that, and how often we actually start control-plane servers.
>
> In terms of ask.o.o, that is a "8 GB Performance" flavor, as defined
> by RAX's flavors.  This was rebuilt when we upgraded it to Xenial as
> an 8GB node (from 4GB) as investigation at the time showed 4GB was a
> bit tight [2].  8GB is the next quanta up of flavor provided by RAX
> over 4GB.
>
> I hope this helps!
>
> -i
>
> [1] https://opendev.org/opendev/system-config/src/branch/master/launch
> [2]
> http://lists.openstack.org/pipermail/openstack-dev/2018-April/129078.html
>
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] opensource infra: server sizes

[Shadi Akiki]

Thanks Clark and Mohammed.
What I'm looking for is potential savings in resources behind the published
openstack infra.
For example, if I focus on the ask.openstack.org service (powering the Q
website), I'm looking for
1- whether or not the ask.openstack.org server is oversized. This is
answered by cacti showing
- max cpu 40% over 2 years here

- traffic being at 20% what it was 2 years ago here

- used memory at 25% here

2- how the allocated resource can be downsized (which I was hoping to find
in the opendev/system-config 
 repo)
- the only related info I could file (doesn't contain resource sizing
info) are
- inventory/groups.yaml

- inventory/openstack.yaml

- hiera/common.yaml

- the only files in the repo with sizing info are (not about
ask.openstack.org)
- hiera/group/infracloud.yaml

 (all "cpu" entries are either "24" or empty)
- playbooks/clouds_layouts.yml

(only 2 entries on vcpu)

Where can I find an answer to point 2?
--
Shadi Akiki
Founder & CEO, AutofitCloud
https://autofitcloud.com/
+1 813 579 4935


On Mon, Aug 12, 2019 at 6:47 PM Mohammed Naser  wrote:

> You can see them listed here:
>
> https://docs.openstack.org/infra/system-config/contribute-cloud.html
>
> On Mon, Aug 12, 2019 at 12:45 PM Shadi Akiki 
> wrote:
> >
> > Hello. I've been going through the opensource infrastructure
> configurations at https://docs.openstack.org/infra/system-config/
> > and linked from https://opensourceinfra.org/
> >
> > I don't see any information about server sizes.
> > Is this something that is not interesting to share as part of the
> opensource infrastructure initiative?
> > --
> > Shadi Akiki
> > Founder & CEO, AutofitCloud
> > https://autofitcloud.com/
> > +1 813 579 4935
> > ___
> > OpenStack-Infra mailing list
> > OpenStack-Infra@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
>
> --
> Mohammed Naser — vexxhost
> -
> D. 514-316-8872
> D. 800-910-1726 ext. 200
> E. mna...@vexxhost.com
> W. http://vexxhost.com
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] opensource infra: server sizes

[Clark Boylan]

On Mon, Aug 12, 2019, at 9:43 AM, Shadi Akiki wrote:
> Hello. I've been going through the opensource infrastructure 
> configurations at https://docs.openstack.org/infra/system-config/
> and linked from https://opensourceinfra.org/
> 
> I don't see any information about server sizes.
> Is this something that is not interesting to share as part of the 
> opensource infrastructure initiative?

Our test node sizing is documented here [0]. This has remained fairly static as 
we try to ensure the test sizes are small enough that developers have the 
opportunity to replicate results locally without needing a datacenter.

For the control plane those server sizes tend not to remain fixed over time as 
demand rises and falls. You can see a current snapshot of server sizing on our 
cacti server [1]. These values may change as we find the servers are too small 
or too big though.

[0] https://docs.openstack.org/infra/manual/testing.html
[1] http://cacti.openstack.org/cacti/graph_view.php

Hope this helps,
Clark

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] opensource infra: server sizes

[Mohammed Naser]

You can see them listed here:

https://docs.openstack.org/infra/system-config/contribute-cloud.html

On Mon, Aug 12, 2019 at 12:45 PM Shadi Akiki  wrote:
>
> Hello. I've been going through the opensource infrastructure configurations 
> at https://docs.openstack.org/infra/system-config/
> and linked from https://opensourceinfra.org/
>
> I don't see any information about server sizes.
> Is this something that is not interesting to share as part of the opensource 
> infrastructure initiative?
> --
> Shadi Akiki
> Founder & CEO, AutofitCloud
> https://autofitcloud.com/
> +1 813 579 4935
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra



-- 
Mohammed Naser — vexxhost
-
D. 514-316-8872
D. 800-910-1726 ext. 200
E. mna...@vexxhost.com
W. http://vexxhost.com

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra