Re: [Openstack-operators] nova-placement-api tuning

[Belmiro Moreira]

Hi,
with Ocata upgrade we decided to run local placements (one service per
cellV1) because we were nervous about possible scalability issues but
specially the increase of the schedule time. Fortunately, this is now been
address with the placement-req-filter work.

We started slowly to aggregate our local placements into a the central one
(required for cellsV2).
Currently we have >7000 compute nodes (>40k requests per minute) into this
central placement. Still ~2000 compute nodes to go.

Some lessons so far...
- Scale keystone accordingly when enabling placement.
- Don't forget to configure memcache for keystone_authtoken.
- Change apache mpm default from prefork to event/worker.
- Increase the WSGI number of processes/threads considering where placement
is running.
- Have enough placement nodes considering your number of requests.
- Monitor the request time. This impacts VM scheduling. Also, depending how
it's configured the LB can also start removing placement nodes.
- DB could be a bottleneck.

We are still learning how to have a stable placement at scale.
It would be great if others can share their experiences.


Belmiro
CERN

On Thu, Mar 29, 2018 at 10:19 AM, Chris Dent  wrote:

> On Wed, 28 Mar 2018, iain MacDonnell wrote:
>
> Looking for recommendations on tuning of nova-placement-api. I have a few
>> moderately-sized deployments (~200 nodes, ~4k instances), currently on
>> Ocata, and instance creation is getting very slow as they fill up.
>>
>
> This should be well within the capabilities of an appropriately
> installed placement service, so I reckon something is weird about
> your installation. More within.
>
> $ time curl http://apihost:8778/
>> {"error": {"message": "The request you have made requires
>> authentication.", "code": 401, "title": "Unauthorized"}}
>> real0m20.656s
>> user0m0.003s
>> sys 0m0.001s
>>
>
> This is good choice for trying to determine what's up because it
> avoids any interaction with the database and most of the stack of
> code: the web server answers, runs a very small percentage of the
> placement python stack and kicks out the 401. So this mostly
> indicates that socket accept is taking forever.
>
> nova-placement-api is running under mod_wsgi with the "standard"(?)
>> config, i.e.:
>>
>
> Do you recall where this configuration comes from? The settings for
> WSGIDaemonProcess are not very good and if there is some packaging
> or documentation that is settings this way it would be good to find
> it and fix it.
>
> Depending on what else is on the host running placement I'd boost
> processes to number of cores divided by 2, 3 or 4 and boost threads to
> around 25. Or you can leave 'threads' off and it will default to 15
> (at least in recent versions of mod wsgi).
>
> With the settings a below you're basically saying that you want to
> handle 3 connections at a time, which isn't great, since each of
> your compute-nodes wants to talk to placement multiple times a
> minute (even when nothing is happening).
>
> Tweaking the number of processes versus the number of threads
> depends on whether it appears that the processes are cpu or I/O
> bound. More threads helps when things are I/O bound.
>
> ...
>>  WSGIProcessGroup nova-placement-api
>>  WSGIApplicationGroup %{GLOBAL}
>>  WSGIPassAuthorization On
>>  WSGIDaemonProcess nova-placement-api processes=3 threads=1 user=nova
>> group=nova
>>  WSGIScriptAlias / /usr/bin/nova-placement-api
>> ...
>>
>
> [snip]
>
> Other suggestions? I'm looking at things like turning off
>> scheduler_tracks_instance_changes, since affinity scheduling is not
>> needed (at least so-far), but not sure that that will help with placement
>> load (seems like it might, though?)
>>
>
> This won't impact the placement service itself.
>
> A while back I did some experiments with trying to overload
> placement by using the fake virt driver in devstack and wrote it up
> at https://anticdent.org/placement-scale-fun.html
>
> The gist was that with a properly tuned placement service it was
> other parts of the system that suffered first.
>
> --
> Chris Dent   ٩◔̯◔۶   https://anticdent.org/
> freenode: cdent tw: @anticdent
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Custom libvirt fragment for instance type?

[Belmiro Moreira]

Hi Jonathan,
this was introduced in Pike.

Belmiro

On Tue, 16 Jan 2018 at 22:48, Jonathan Proulx  wrote:

> On Tue, Jan 16, 2018 at 03:49:25PM -0500, Jonathan Proulx wrote:
> :On Tue, Jan 16, 2018 at 08:42:00PM +, Tim Bell wrote:
> ::If you want to hide the VM signature, you can use the
> img_hide_hypervisor_id property (
> https://docs.openstack.org/python-glanceclient/latest/cli/property-keys.html
> )
> :
> :Thanks Tim, I believe that's the magic I was looking for.
>
> Unfortunately settign that doesn't appear to do anything way back here
> in Mitaka land :(
>
> Oh well reason 128478 that I need to take that series of leaps I
> suppose.
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] reminder: ops meetups team meeting on #openstack-operators

[Belmiro Moreira]

Hi,
can we add this meeting into the official IRC meetings page?
https://wiki.openstack.org/wiki/Meetings
http://eavesdrop.openstack.org/

thanks,
Belmiro



On Tue, 24 Oct 2017 at 15:51, Chris Morgan  wrote:

> Next meeting in about 10 minutes from now
>
> Chris
>
> --
> Chris Morgan 
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Libvirt CPU map (host-model)

[Belmiro Moreira]

Hi Paul,
yes, this is exactly what we are observing.
Thanks for the bugzilla pointer.

We now also opened a ticket through the RedHat support.

thanks,
Belmiro

On Mon, Oct 9, 2017 at 12:37 PM, Paul Browne <pf...@cam.ac.uk> wrote:

> Hello Belmiro,
>
> We ran into this issue recently, similarly upgrading a RHEL7.3 OpenStack
> Platform Overcloud to RHEL7.4 and in the process upgrading libvirtd.
>
> For instances that were spawned prior to this upgrade, we see the CPU
> flags [1] , but for new instance workload the CPU flags [2]. Notably the
> CMT=disabled flag is present in [1] but absent in [2]
>
> This similarly prevents live migration of the older spawned instances, as
> the CMT=disabled flag is rejected.
>
> A RH bugzilla [3] was opened on the issue which attracted a lot of really
> good contributions from libvirt maintainers. The one sure-fire workaround
> we'd found is just to cold-boot the instance again, starting it under the
> new libvirtd. But from that BZ there is also a slightly more hack-ish
> workaround to hand-edit the running domain XML and clear the offending CMT
> flag (comment 12 on that BZ).
>
> Hope this helps some,
>
> Thanks,
> Paul Browne
>
> [1] https://pastebin.com/JshWi6i3
> [2] https://pastebin.com/5b8cAanP
> [3] https://bugzilla.redhat.com/show_bug.cgi?id=1495171
>
> On 9 October 2017 at 04:59, Belmiro Moreira <moreira.belmiro.email.lists@
> gmail.com> wrote:
>
>> Hi,
>> the CPU model that we expose to the guest VMs varies considering the
>> compute node use case.
>> We use "cpu_mode=host-passthrough" for the compute nodes that run batch
>> processing VMs and "cpu_mode=host-model" for the compute nodes for service
>> VMs. The reason to have "cpu_mode=host-model" is because we assumed that
>> new CPUs (in the libvirt map) will continue to support previous features
>> allowing for live migration when we need to move the VMs to a new CPU
>> generation.
>>
>> We recently upgraded from CentOS7.3 (libvirt 2.0.0) to CentOS7.4 (libvirt
>> 3.2.0) and noticed that now libvirt maps a slightly different CPU for the
>> guests. For example, still "Haswell no-TSX" but no mention to the feature
>> "cmt". This blocks suspended VMs to restore and live migrate.
>>
>> Has anyone experienced this same problem?
>>
>> We are thinking in few solutions but none of them are nice (downgrade
>> libvirt? hard reboot instances? ...)
>>
>> thanks,
>> Belmiro
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>
>
> --
> ***
> Paul Browne
> Research Computing Platforms
> University Information Services
> Roger Needham Building
> JJ Thompson Avenue
> University of Cambridge
> Cambridge
> United Kingdom
> E-Mail: pf...@cam.ac.uk
> Tel: 0044-1223-746548
> ***
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Libvirt CPU map (host-model)

[Belmiro Moreira]

Hi,
the CPU model that we expose to the guest VMs varies considering the
compute node use case.
We use "cpu_mode=host-passthrough" for the compute nodes that run batch
processing VMs and "cpu_mode=host-model" for the compute nodes for service
VMs. The reason to have "cpu_mode=host-model" is because we assumed that
new CPUs (in the libvirt map) will continue to support previous features
allowing for live migration when we need to move the VMs to a new CPU
generation.

We recently upgraded from CentOS7.3 (libvirt 2.0.0) to CentOS7.4 (libvirt
3.2.0) and noticed that now libvirt maps a slightly different CPU for the
guests. For example, still "Haswell no-TSX" but no mention to the feature
"cmt". This blocks suspended VMs to restore and live migrate.

Has anyone experienced this same problem?

We are thinking in few solutions but none of them are nice (downgrade
libvirt? hard reboot instances? ...)

thanks,
Belmiro
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Should we allow passing new user_data during rebuild?

[Belmiro Moreira]

In our cloud rebuild is the only way for a user to keep the same IP.
Unfortunately, we don't offer floating IPs, yet.
Also, we use the user_data to bootstrap some actions in new instances
(puppet, ...).
Considering all the use-cases for rebuild it would be great if the
user_data can be updated at rebuild time.

On Wed, Oct 4, 2017 at 5:15 PM, Chris Friesen 
wrote:

> On 10/03/2017 11:12 AM, Clint Byrum wrote:
>
> My personal opinion is that rebuild is an anti-pattern for cloud, and
>> should be frozen and deprecated. It does nothing but complicate Nova
>> and present challenges for scaling.
>>
>> That said, if it must stay as a feature, I don't think updating the
>> user_data should be a priority. At that point, you've basically created an
>> entirely new server, and you can already do that by creating an entirely
>> new server.
>>
>
> If you've got a whole heat stack with multiple resources, and you realize
> that you messed up one thing in the template and one of your servers has
> the wrong personality/user_data, it can be useful to be able to rebuild
> that one server without affecting anything else in the stack.  That's just
> a convenience though.
>
> Chris
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova] Queens PTG recap - everything else

[Belmiro Moreira]

Hi Matt,
thanks for these great summaries.

I didn't find any mention to nested quotas.
Was it discussed in the PTG? and what can we expect for Queens?

thanks,
Belmiro
CERN

On Mon, Sep 18, 2017 at 11:58 PM, Matt Riedemann 
wrote:

> There was a whole lot of other stuff discussed at the PTG. The details are
> in [1]. I won't go into everything here, so I'm just highlighting some of
> the more concrete items that had owners or TODOs.
>
> Ironic
> --
>
> The Ironic team came over on Wednesday afternoon. We talked a bit, had
> some laughs, it was a good time. Since I don't speak fluent baremetal,
> Dmitry Tantsur is going to recap those discussions in the mailing list.
> Thanks again, Dmitry.
>
> Privsep
> ---
>
> Michael Still has been going hog wild converting the nova libvirt driver
> code to use privsep instead of rootwrap. He has a series of changes tracked
> under this blueprint [2]. Most of the discussion was a refresh on privsep
> and a recap of what's already been merged and some discussion on
> outstanding patches. The goal for Queens is to get the entire libvirt
> driver converted and also try to get all of nova-compute converted, but we
> want to limit that to getting things merged early in the release to flush
> out bugs since a lot of these are weird, possibly untested code paths.
> There was also discussion of a kind of privsep heartbeat daemon to tell if
> it's running (even though it's not a separate service) but this is
> complicated and is not something we'll pursue for Queens.
>
> Websockify security proxy framework
> ---
>
> This is a long-standing security hardening feature [3] which has changed
> hands a few times and hasn't gotten much review. Sean Dague and Melanie
> Witt agreed to focus on reviewing this for Queens.
>
> Certificate validation
> --
>
> This is another item that's been discussed since at least the Ocata summit
> but hasn't made much progress. Sean Dague agreed to help review this, and
> Eric Fried said he knew someone that could help review the security aspects
> of this change. Sean also suggested scheduling a hangout so the John
> Hopkins University team working on this can give a primer on the feature
> and what to look out for during review. We also suggested getting a
> scenario test written for this in the barbican tempest plugin, which runs
> as an experimental queue job for nova.
>
> Notifications
> -
>
> Given the state of the Searchlight project and how we don't plan on using
> Searchlight as a global proxy for the compute REST API, we are not going to
> work on parity with versioned notifications there. There are some cleanups
> we still need to do in Nova for versioned notifications from a performance
> perspective. We also agreed that we aren't going to consider deprecating
> legacy unversioned notifications until we have parity with the versioned
> notifications, especially given legacy unversioned notification consumers
> have not yet moved to using the versioned notifications.
>
> vGPU support
> 
>
> This depends on nested resource providers (like lots of other things). It
> was not clear from the discussion if this is static or dynamic support,
> e.g. can we hot plug vGPUs using Cyborg? I assume we will not support hot
> plugging at first. We also need improved functional testing of this space
> before we can make big changes.
>
> Preemptible (spot) instances
> -
>
> This was continuing the discussion from the Boston forum session [5]. The
> major issue in Nova is that we don't want Nova to be in charge of
> orchestrating preempting instances when a request comes in for a "paid"
> instance. We agreed to start small where you can't burst over quota. Blazar
> also delivered some reservation features in Pike [6] which sound like they
> can be built on here, which also sound like expiration policies. Someone
> will have to prototype an external (to nova) "reaper" which will cull the
> preemptible instances based on some configurable policy. Honestly the notes
> here are confusing so we're going to need someone to drive this forward.
> That might mean picking up John Garbutt's draft spec for this (link not
> available right now).
>
> Driver updates
> --
>
> Various teams from IBM gave updates on plans for their drivers in Queens.
>
> PowerVM (in tree): the team is proposing a few more capabilities to the
> driver in Queens. Details are in the spec [7].
>
> zDPM (out of tree): this out of tree driver has had two releases (ocata
> and pike) and is working on 3rd party CI. One issue they have with Tempest
> is they can only boot from volume.
>
> zVM (out of tree): the team is working on refactoring some code into a
> library, similar to os-xenapi, os-powervm and oslo.vmware. They have CI
> running but are not yet reporting against nova changes.
>
> Endpoint discovery
> --
>
> This is carry-over work 

Re: [Openstack-operators] [nova][scheduler] Anyone relying on the host_subset_size config option?

[Belmiro Moreira]

This option is useful in large deployments.

Our scheduler strategy is to "pack", however we are not interested in this
strategy per individual compute node but per sets of them. One of the
advantages is that when a user creates consecutive instances in the same
AVZ it's unlikely that they will be started in the same compute node.

Also, a problem in the "best" compute node doesn't block completely the
creation of new instances when not using "retry".



Belmiro

--

CERN

On Fri, May 26, 2017 at 7:17 PM, Edward Leafe  wrote:

> [resending to include the operators list]
>
> The host_subset_size configuration option was added to the scheduler to
> help eliminate race conditions when two requests for a similar VM would be
> processed close together, since the scheduler’s algorithm would select the
> same host in both cases, leading to a race and a likely failure to build
> for the second request. By randomly choosing from the top N hosts, the
> likelihood of a race would be reduced, leading to fewer failed builds.
>
> Current changes in the scheduling process now have the scheduler claiming
> the resources as soon as it selects a host. So in the case above with 2
> similar requests close together, the first request will claim successfully,
> but the second will fail *while still in the scheduler*. Upon failing the
> claim, the scheduler will simply pick the next host in its weighed list
> until it finds one that it can claim the resources from. So the
> host_subset_size configuration option is no longer needed.
>
> However, we have heard that some operators are relying on this option to
> help spread instances across their hosts, rather than using the RAM
> weigher. My question is: will removing this randomness from the scheduling
> process hurt any operators out there? Or can we safely remove that logic?
>
>
> -- Ed Leafe
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [keystone][nova][cinder][glance][neutron][horizon][policy] defining admin-ness

[Belmiro Moreira]

Hi,
thanks for bringing this into discussion in the Operators list.

Option 1 and 2 and not complementary but complety different.
So, considering "Option 2" and the goal to target it for Queens I would
prefer not going into a migration path in
Pike and then again in Queens.

Belmiro


On Fri, May 26, 2017 at 2:52 AM, joehuang  wrote:

> I think a option 2 is better.
>
> Best Regards
> Chaoyi Huang (joehuang)
> --
> *From:* Lance Bragstad [lbrags...@gmail.com]
> *Sent:* 25 May 2017 3:47
> *To:* OpenStack Development Mailing List (not for usage questions);
> openstack-operators@lists.openstack.org
> *Subject:* Re: [openstack-dev] [keystone][nova][cinder][
> glance][neutron][horizon][policy] defining admin-ness
>
> I'd like to fill in a little more context here. I see three options with
> the current two proposals.
>
> *Option 1*
>
> Use a special admin project to denote elevated privileges. For those
> unfamiliar with the approach, it would rely on every deployment having an
> "admin" project defined in configuration [0].
>
> *How it works:*
>
> Role assignments on this project represent global scope which is denoted
> by a boolean attribute in the token response. A user with an 'admin' role
> assignment on this project is equivalent to the global or cloud
> administrator. Ideally, if a user has a 'reader' role assignment on the
> admin project, they could have access to list everything within the
> deployment, pending all the proper changes are made across the various
> services. The workflow requires a special project for any sort of elevated
> privilege.
>
> Pros:
> - Almost all the work is done to make keystone understand the admin
> project, there are already several patches in review to other projects to
> consume this
> - Operators can create roles and assign them to the admin_project as
> needed after the upgrade to give proper global scope to their users
>
> Cons:
> - All global assignments are linked back to a single project
> - Describing the flow is confusing because in order to give someone global
> access you have to give them a role assignment on a very specific project,
> which seems like an anti-pattern
> - We currently don't allow some things to exist in the global sense (i.e.
> I can't launch instances without tenancy), the admin project could own
> resources
> - What happens if the admin project disappears?
> - Tooling or scripts will be written around the admin project, instead of
> treating all projects equally
>
> *Option 2*
>
> Implement global role assignments in keystone.
>
> *How it works:*
>
> Role assignments in keystone can be scoped to global context. Users can
> then ask for a globally scoped token
>
> Pros:
> - This approach represents a more accurate long term vision for role
> assignments (at least how we understand it today)
> - Operators can create global roles and assign them as needed after the
> upgrade to give proper global scope to their users
> - It's easier to explain global scope using global role assignments
> instead of a special project
> - token.is_global = True and token.role = 'reader' is easier to understand
> than token.is_admin_project = True and token.role = 'reader'
> - A global token can't be associated to a project, making it harder for
> operations that require a project to consume a global token (i.e. I
> shouldn't be able to launch an instance with a globally scoped token)
>
> Cons:
> - We need to start from scratch implementing global scope in keystone,
> steps for this are detailed in the spec
>
> *Option 3*
>
> We do option one and then follow it up with option two.
>
> *How it works:*
>
> We implement option one and continue solving the admin-ness issues in Pike
> by helping projects consume and enforce it. We then target the
> implementation of global roles for Queens.
>
> Pros:
> - If we make the interface in oslo.context for global roles consistent,
> then consuming projects shouldn't know the difference between using the
> admin_project or a global role assignment
>
> Cons:
> - It's more work and we're already strapped for resources
> - We've told operators that the admin_project is a thing but after Queens
> they will be able to do real global role assignments, so they should now
> migrate *again*
> - We have to support two paths for solving the same problem in keystone,
> more maintenance and more testing to ensure they both behave exactly the
> same way
>   - This can get more complicated for projects dedicated to testing policy
> and RBAC, like Patrole
>
>
> Looking for feedback here as to which one is preferred given timing and
> payoff, specifically from operators who would be doing the migrations to
> implement and maintain proper scope in their deployments.
>
> Thanks for reading!
>
>
> [0] https://github.com/openstack/keystone/blob/
> 3d033df1c0fdc6cc9d2b02a702efca286371f2bd/etc/keystone.conf.
> sample#L2334-L2342
>
> On Wed, May 24, 2017 at 10:35 AM, Lance Bragstad 

Re: [Openstack-operators] [openstack-dev] [Nova] [Cells] Stupid question: Cells v2 & AZs

[Belmiro Moreira]

Hi David,
AVZs are basically aggregates.
In cells_v2 aggregates are defined in the cell_api, so it will be possible
to have
multiple AVZs per cell and AVZs that spread between different cells.

Belmiro

On Wed, May 24, 2017 at 5:14 AM, David Medberry 
wrote:

> Hi Devs and Implementers,
>
> A question came up tonight in the Colorado OpenStack meetup regarding
> cells v2 and availability zones.
>
> Can a cell contain multiple AZs? (I assume this is yes.)
>
> Can an AZ contain mutliple cells (I assumed this is no, but now in
> thinking about it, that's probably not right.)
>
> What's the proper way to think about this? In general, I'm considering AZs
> primarily as a fault zone type of mechanism (though they can be used in
> other ways.)
>
> Is there a clear diagram/documentation about this?
>
> And consider this to be an Ocata/Pike and later only type of question.
>
> Thanks.
>
> -dave
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Successful nova-network to Neutron Migration

[Belmiro Moreira]

Hi Joe,
congrats.

Can you also make available your scripts changes for IPv6?
The more the better for any site that is still working in the migration,
like us :)

thanks,
Belmiro

On Sat, May 20, 2017 at 6:51 PM, Joe Topjian  wrote:

> Hi all,
>
> There probably aren't a lot of people in this situation nowadays, but for
> those that are, I wanted to report a successful nova-network to Neutron
> migration.
>
> We used NeCTAR's migration scripts which can be found here:
>
> https://github.com/NeCTAR-RC/novanet2neutron
>
> These scripts allowed us to do an in-place upgrade with almost no
> downtime. There was probably an hour or two of network downtime, but all
> instances stayed up and running. There were also a handful of instances
> that needed a hard reboot and some that had to give up their Floating IP to
> Neutron. All acceptable, IMO.
>
> We modified them to suit our environment, specifically by adding support
> for IPv6 and Floating IPs. In addition, we leaned on our existing Puppet
> environment to deploy certain  Nova and Neutron settings in phases.
>
> But we wouldn't have been able to do this migration without these scripts,
> so to Sam and the rest of the NeCTAR crew: thank you all very much!
>
> Joe
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] nova mitaka->newton online_data_migrations

[Belmiro Moreira]

Hi Saverio,
when not using "max_count" the message "Running batches of 50 until complete"
is always printed.
If you are not getting any error and no more output the migrations should
have finished.

Unfortunately, there is not such message "All Done" when the
online_data_migrations
finish.

You can check manually if everything was correctly migrated, looking into
the DB.
These methods are triggered by the "online_data_migrations":
db.pcidevice_online_data_migration
db.computenode_uuids_online_data_migration
db.aggregate_uuids_online_data_migration

cheers,

Belmiro
CERN

On Tue, Mar 7, 2017 at 8:34 AM, Saverio Proto  wrote:

> Hello there,
>
> before doing the upgrade, when still in Mitaka (tag 13.1.3), I am
> running the folllwing command:
>
> nova-manage db online_data_migrations
>
> I get this output:
>
> Option "verbose" from group "DEFAULT" is deprecated for removal.  Its
> value may be silently ignored in the future.
> Option "notification_topics" from group "DEFAULT" is deprecated. Use
> option "topics" from group "oslo_messaging_notifications".
> Running batches of 50 until complete
>
> I always get the same output.
>  I also tried :
> nova-manage db online_data_migrations --max-count 5000
>
> In this case I only get the warnings about the option without 'Running
> batches of 50 until complete'.
>
> When I do not use max-count I will always get 'Running batches of 50
> until complete'.
>
> How do I understand when all the online_data_migrations are done ?
>
> I never get anything printed like 'all done'.
>
> thank you
>
> Saverio
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] FYI: live_migration_progress_timeout will default to 0 and be deprecated in Ocata

[Belmiro Moreira]

+1
We use "block-migration" and we needed to disable this timeout.

Belmiro
CERN

On Thu, Feb 9, 2017 at 5:29 PM, Matt Riedemann  wrote:

> This is just a heads up to anyone running with this since Liberty, there
> is a patch [1] that will go into Ocata which deprecates the
> live_migration_progress_timeout config option used in the libvirt driver.
> The default value will also change to 0, effectively disabling the feature.
> See the patch and related bug report for more details.
>
> If you've seen issues trying to use this feature, and can confirm this or
> think it's actually wrong, please speak up soon (final Nova ocata tag is on
> 2/16).
>
> [1] https://review.openstack.org/#/c/429798/
>
> --
>
> Thanks,
>
> Matt Riedemann
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova] How to expose the compute node local disks to instances

[Belmiro Moreira]

Hi Sean,
thanks for the suggestions.

On Thu, 8 Dec 2016 at 23:10, Sean McGinnis <sean.mcgin...@gmx.com> wrote:

> Hi Belmiro,
>
> In Cinder there is the "raw disk device" driver that has been used by some
> for Hadoop and similar applications. That may be something to look at, but
> with a big warning.
>
> That being that it is deprecated in the Ocata release and will be removed
> in Pike.
>
> The reason it is going to be removed has a couple reasons though. One big
> one is that it is unable to meet the "minimum feature set" that we require
> for a Cinder driver. There are just too many restrictions on what can be
> done when you're using raw disk.
>
> The other reason was that after running a set of tests, it we really didn't
> see that much of a difference in uing the raw device driver vs. configuring
> the LVM driver appropriately. I don't have the full details of those tests
> handy at the moment, but I'm sure we can track down some specifics if
> needed.
>
> So I would recommend connecting your JBODs to some compute hosts and using
> that disk space to back LVM, then configure Cinder with the LVM driver to
> use that pool of space to present volumes to your compute instances.
>
> Thanks,
> Sean
>
> On Thu, Dec 08, 2016 at 07:46:35PM +0100, Belmiro Moreira wrote:
> > Hi,
> >
> > we have a set of disk servers (JBOD) that we would like to integrate into
> > our cloud to run   applications like Hadoop and Spark.
> >
> > Using file disks for storage and a huge "/var/lib/nova" is not an option
> > for these use cases so we would like to expose the local drives directly
> to
> > the VMs as ephemeral disks.
> >
> >
> >
> > Is there already something in Nova that allows this?
> >
> >
> >
> > thanks,
> >
> > Belmiro
> >
> > CERN
>
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-dev] [nova] How to expose the compute node local disks to instances

[Belmiro Moreira]

Hi,

we have a set of disk servers (JBOD) that we would like to integrate into
our cloud to run   applications like Hadoop and Spark.

Using file disks for storage and a huge "/var/lib/nova" is not an option
for these use cases so we would like to expose the local drives directly to
the VMs as ephemeral disks.



Is there already something in Nova that allows this?



thanks,

Belmiro

CERN
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How to tune scheduling for "Insufficient compute resources" (race conditions ?)

[Belmiro Moreira]

How many nova-schedulers are you running?
You can hit this issue when multiple nova-schedulers select the same
compute node for different instances.

Belmiro

On Wed, Nov 30, 2016 at 3:56 PM, Massimo Sgaravatto <
massimo.sgarava...@gmail.com> wrote:

> Hi all
>
> I have a problem with scheduling in our Mitaka Cloud,
> Basically when there are a lot of requests for new instances, some of them
> fail because "Failed to compute_task_build_instances: Exceeded maximum
> number of retries". And the failures are because "Insufficient compute
> resources: Free memory 2879.50 MB < requested
>  8192 MB" [*]
>
> But there are compute nodes with enough memory that could serve such
> requests.
>
> In the conductor log I also see messages reporting that "Function
> 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted
> interval by xxx sec" [**]
>
>
> My understanding is that:
>
> - VM a is scheduled to a certain compute node
> - the scheduler chooses the same compute node for VM b before the info for
> that compute node is updated (so the 'size' of VM a is not taken into
> account)
>
> Does this make sense or am I totally wrong ?
>
> Any hints about how to cope with such scenarios, besides increasing
>  scheduler_max_attempts ?
>
> scheduler_default_filters is set to:
>
> scheduler_default_filters = AggregateInstanceExtraSpecsFilter,
> AggregateMultiTenancyIsolation,RetryFilter,AvailabilityZoneFilter,
> RamFilter,CoreFilter,AggregateRamFilter,AggregateCoreFilter,ComputeFilter,
> ComputeCapabilitiesFilter,ImagePropertiesFilter,
> ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter
>
>
> Thanks a lot, Massimo
>
> [*]
>
> 2016-11-30 15:10:20.233 25140 WARNING nova.scheduler.utils
> [req-ec8c0bdc-b413-4cab-b925-eb8f11212049 840c96b6fb1e4972beaa3d30ade10cc7
> d27fe2becea94a3e980fb9f66e2f29
> 1a - - -] Failed to compute_task_build_instances: Exceeded maximum number
> of retries. Exceeded max scheduling attempts 5 for instance
> 314eccd0-fc73-446f-8138-7d8d3c
> 8644f7. Last exception: Insufficient compute resources: Free memory
> 2879.50 MB < requested 8192 MB.
> 2016-11-30 15:10:20.233 25140 WARNING nova.scheduler.utils
> [req-ec8c0bdc-b413-4cab-b925-eb8f11212049 840c96b6fb1e4972beaa3d30ade10cc7
> d27fe2becea94a3e980fb9f66e2f29
> 1a - - -] [instance: 314eccd0-fc73-446f-8138-7d8d3c8644f7] Setting
> instance to ERROR state.
>
>
> [**]
>
> 2016-11-30 15:10:48.873 25128 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 9.08 sec
> 2016-11-30 15:10:54.372 25142 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 9.33 sec
> 2016-11-30 15:10:54.375 25140 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 9.32 sec
> 2016-11-30 15:10:54.376 25129 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 9.30 sec
> 2016-11-30 15:10:54.381 25138 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 9.24 sec
> 2016-11-30 15:10:54.381 25139 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 9.28 sec
> 2016-11-30 15:10:54.382 25143 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 9.24 sec
> 2016-11-30 15:10:54.385 25141 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 9.11 sec
> 2016-11-30 15:11:01.964 25128 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 3.09 sec
> 2016-11-30 15:11:05.503 25142 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 1.13 sec
> 2016-11-30 15:11:05.506 25138 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 1.12 sec
> 2016-11-30 15:11:05.509 25139 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 1.13 sec
> 2016-11-30 15:11:05.512 25141 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 1.13 sec
> 2016-11-30 15:11:05.525 25143 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 1.14 sec
> 2016-11-30 15:11:05.526 25140 WARNING oslo.service.loopingcall [-]
> Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run
> outlasted interval by 1.15 sec
> 2016-11-30 15:11:05.529 25129 WARNING 

Re: [Openstack-operators] How do you handle purge of database tables ?

[Belmiro Moreira]

Hi,
we wrote this blog post a year ago but it still can be useful depending on
the
OpenStack version that you are running.

http://openstack-in-production.blogspot.ch/2015/05/purging-nova-databases-in-cell.html

Belmiro

On Thu, Jun 23, 2016 at 3:32 PM, Nick Jones 
wrote:

> > On Thu, 2016-06-23 at 09:55 +0100, Nick Jones wrote:
> >> Another vote of confidence for the script that Tim has mentioned with
> >> regards to clearing down Nova’s DB.  I blogged a bit about the process
> >> and the results here:
> >>
> >>
> >> http://dischord.org/2015/12/30/archiving-data-in-nova-s-database/
> >>
> >> —
> >
> >>
> >
> > In Nova, there's 'nova-manage db archive_deleted_rows' - is that the one
> > referred to in the blog post above that broke?
>
> That’s the one.
>
> —
>
> -Nick
> --
> DataCentred Limited registered in England and Wales no. 05611763
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [glance] Proposal for a mid-cycle virtual sync on operator issues

[Belmiro Moreira]

Hi Nikhil,
I'm interested in this discussion.

Initially you were proposing Thursday June 9th, 2016 at 2000UTC.
Are you suggesting to change also the date? Because in the new timeanddate
suggestions is 6/7 of June.

Belmiro

On Tue, May 31, 2016 at 6:13 PM, Nikhil Komawar 
wrote:

> Hey,
>
>
> Thanks for the feedback. 0800UTC is 4am EDT for some of the US Glancers :-)
>
>
> I request this time which may help the folks in Eastern and Central US
> time.
>
> http://www.timeanddate.com/worldclock/meetingdetails.html?year=2016=6=7=11=0=0=881=196=47=22=157=87=24=78
>
>
> If it still does not work, I may have to poll the folks in EMEA on how
> strong their intentions are for joining this call.  Because another time
> slot that works for folks in Australia & US might be too inconvenient
> for those in EMEA:
>
> http://www.timeanddate.com/worldclock/meetingdetails.html?year=2016=6=6=23=0=0=881=196=47=22=157=87=24=78
>
>
> Here's the map of cities that may be involved:
>
> http://www.timeanddate.com/worldclock/meetingtime.html?iso=20160607=881=196=47=22=157=87=24=78
>
>
> Please let me know which ones are possible and we can try to work around
> the times.
>
>
> On 5/31/16 2:54 AM, Blair Bethwaite wrote:
> > Hi Nikhil,
> >
> > 2000UTC might catch a few kiwis, but it's 6am everywhere on the east
> > coast of Australia, and even earlier out west. 0800UTC, on the other
> > hand, would be more sociable.
> >
> > On 26 May 2016 at 15:30, Nikhil Komawar  wrote:
> >> Thanks Sam. We purposefully chose that time to accommodate some of our
> >> community members from the Pacific. I'm assuming it's just your case
> >> that's not working out for that time? So, hopefully other Australian/NZ
> >> friends can join.
> >>
> >>
> >> On 5/26/16 12:59 AM, Sam Morrison wrote:
> >>> I’m hoping some people from the Large Deployment Team can come along.
> It’s not a good time for me in Australia but hoping someone else can join
> in.
> >>>
> >>> Sam
> >>>
> >>>
>  On 26 May 2016, at 2:16 AM, Nikhil Komawar 
> wrote:
> 
>  Hello,
> 
> 
>  Firstly, I would like to thank Fei Long for bringing up a few operator
>  centric issues to the Glance team. After chatting with him on IRC, we
>  realized that there may be more operators who would want to contribute
>  to the discussions to help us take some informed decisions.
> 
> 
>  So, I would like to call for a 2 hour sync for the Glance team along
>  with interested operators on Thursday June 9th, 2016 at 2000UTC.
> 
> 
>  If you are interested in participating please RSVP here [1], and
>  participate in the poll for the tool you'd prefer. I've also added a
>  section for Topics and provided a template to document the issues
> clearly.
> 
> 
>  Please be mindful of everyone's time and if you are proposing issue(s)
>  to be discussed, come prepared with well documented & referenced
> topic(s).
> 
> 
>  If you've feedback that you are not sure if appropriate for the
>  etherpad, you can reach me on irc (nick: nikhil).
> 
> 
>  [1]
> https://etherpad.openstack.org/p/newton-glance-and-ops-midcycle-sync
> 
>  --
> 
>  Thanks,
>  Nikhil Komawar
>  Newton PTL for OpenStack Glance
> 
> 
> 
> __
>  OpenStack Development Mailing List (not for usage questions)
>  Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> __
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> --
> >>
> >> Thanks,
> >> Nikhil
> >>
> >>
> >> ___
> >> OpenStack-operators mailing list
> >> OpenStack-operators@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
> >
>
> --
>
> Thanks,
> Nikhil
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Display all nova (or other service) configuration parameters

[Belmiro Moreira]

When a service starts with the log level configured to debug you can see
what options/values is using.

Belmiro

On Sat, May 7, 2016 at 10:17 PM, Sergio Cuellar Valdes <
scuell...@kionetworks.com> wrote:

> Hi everybody,
>
> How can you display all the values that has nova or other service and that
> has not been set in their configuration file?
>
> Thanks
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova][neutron] What are your cells networking use cases?

[Belmiro Moreira]

Hi,
thanks Carl for info about the DHCP plans.

Our DHCP concern is because currently the DHCP agent needs to be assigned
to a network and then it creates a port for each subnet.
In our infrastructure we only consider a network with several hundred
subnets.
By default the DHCP agent runs in the network node however when using
provider
networks and segmentation the DHCP requests will not reach it.

My understanding is that the plan is to have the DHCP agent per segment.
That is great.
But it will continue to create a port per subnet?
Looking into our usecase (only provided networks with segmentation) I don't
see why it
should create ports.

Belmiro

On Fri, Feb 26, 2016 at 2:55 AM, Sam Morrison  wrote:

>
> > On 26 Feb 2016, at 9:20 AM, Carl Baldwin  wrote:
> >
> > (resending with reply-all)
> >
> > The routed networks work will include a change to the DHCP scheduler
> > which will work something like this:
> >
> > 1. Neutron subnets will have optional affinity to a segment
> > 2. DHCP agents will (somewhat indirectly) report which segments to
> > which they are attached*.
> > 3. Where today, DHCP schedules networks to DHCP agents, tomorrow DHCP
> > will schedule each segment to an agent that can reach it.  This will
> > be predicated on 'enable_dhcp' being set on the subnets.
> >
> > There is an implicit assumption here that the operator will deploy a
> > DHCP agent in each of the segments.  This will be documented in the
> > guide.
>
> I assume you’re referring to https://review.openstack.org/#/c/205631/
> Really keen to get this in, we’re using this in prod and works well for us.
>
> Sam
>
>
>
>
> > Down the road, I really think we should continue to explore other
> > possibilities like DHCP relay or a DHCP responder on the compute host.
> > But, that should be considered an independent effort.
> >
> > Carl
> >
> > * they already do this by reporting physical_network in bridge mappings
> >
> > On Thu, Feb 25, 2016 at 11:30 AM, Tim Bell  wrote:
> >>
> >> The CERN guys had some concerns on how dhcp was working in a segment
> environment. I’ll leave them to give details.
> >>
> >> Tim
> >>
> >>
> >>
> >>
> >>
> >> On 25/02/16 14:53, "Andrew Laski"  wrote:
> >>
> >>>
> >>>
> >>> On Thu, Feb 25, 2016, at 05:01 AM, Tim Bell wrote:
> 
>  CERN info added.. Feel free to come back for more information if
> needed.
> >>>
> >>> An additional piece of information we're specifically interested in
> from
> >>> all cellsv1 deployments is around the networking control plane setup.
> Is
> >>> there a single nova-net/Neutron deployment per region that is shared
> >>> among cells? It appears that all cells users are splitting the network
> >>> data plane into clusters/segments, are similar things being done to the
> >>> control plane?
> >>>
> >>>
> 
>  Tim
> 
> 
> 
> 
>  On 24/02/16 22:47, "Edgar Magana"  wrote:
> 
> > It will be awesome if we can add this doc into the networking guide
> :-)
> >
> >
> > Edgar
> >
> >
> >
> >
> > On 2/24/16, 1:42 PM, "Matt Riedemann" 
> wrote:
> >
> >> The nova and neutron teams are trying to sort out existing
> deployment
> >> network scenarios for cells v1 so we can try and document some of
> that
> >> and get an idea if things change at all with cells v2.
> >>
> >> Therefore we're asking that deployers running cells please document
> >> anything you can in an etherpad [1].
> >>
> >> We'll try to distill that for upstream docs at some point and then
> use
> >> it as a reference when talking about cells v2 + networking.
> >>
> >> [1] https://etherpad.openstack.org/p/cells-networking-use-cases
> >>
> >> --
> >>
> >> Thanks,
> >>
> >> Matt Riedemann
> >>
> >>
> >> ___
> >> OpenStack-operators mailing list
> >> OpenStack-operators@lists.openstack.org
> >>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> > ___
> > OpenStack-operators mailing list
> > OpenStack-operators@lists.openstack.org
> >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>  ___
>  OpenStack-operators mailing list
>  OpenStack-operators@lists.openstack.org
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>  Email had 1 attachment:
>  + smime.p7s
>   4k (application/pkcs7-signature)
> >>>
> >>> ___
> >>> OpenStack-operators mailing list
> >>> OpenStack-operators@lists.openstack.org
> >>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >>
> >> ___
> >> 

[Openstack-operators] Operators Meetup Manchester - sessions schedule available?

[Belmiro Moreira]

Hi,
for the Ops Meetup in Manchester we have the following etherpad:
https://etherpad.openstack.org/p/MAN-ops-meetup but is there any schedule
already available for the sessions?

thanks,
Belmiro
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova][cells] Should flavors in the API DB for cells v2 be soft-deletable?

[Belmiro Moreira]

IMHO I think it's a great way to fix the URI problem.
+1

Belmiro

On Fri, Jan 8, 2016 at 3:23 PM, Sylvain Bauza  wrote:

>
>
> Le 08/01/2016 15:10, Andrew Laski a écrit :
>
>> On 01/08/16 at 12:43pm, John Garbutt wrote:
>>
>>> On 7 January 2016 at 19:59, Matt Riedemann 
>>> wrote:
>>>
 There is a cells v2 change up for review [1] which creates the flavor
 tables
 in the API DB.

 I noted that those table definitions include the soft-delete columns
 (deleted and deleted_at), which at the YVR summit and in other threads
 [2]
 we've said we're not doing anymore for new tables.

 The point raised to keep them soft-deletable is that the flavor API
 allows
 showing soft-deleted flavors if you know the id [3]. And you can get the
 flavor id for an instance (we always store the flavor info that was
 used to
 boot the instance).

 The question is, can we break that wrinkle in the API by not allow
 soft-deleting the flavor in the API DB?

>>>
>>> On balance, I think this is OK.
>>>
>>
>> There's an alternate approach to this that we can take which I'll outline
>> below.  But it should meet the needs of anyone currently looking up deleted
>> flavors.
>>
>>
>>
>>> Note that in the normal nova DB, if the admin archives/purges the
 instance_types table, the wrinkle is already broken because the
 soft-deleted
 flavor is now hard-deleted, but that's maybe not a great justification
 for
 consciously removing this support in the API DB.

>>>
>>> This is what won me over. All be it, reluctantly.
>>>
>>
>> I think this is good justification for removing the ability because
>> currently it is essentially undefined behavior. Requesting a deleted flavor
>> may or may not work and either response is correct.
>>
>>
>>> If we made the flavor soft-deletable in the API DB, one issue is we don't
 have an in-tree entrypoint for cleaning this up (there are no
 archive/purge
 CLIs for the API DB). We could always add that, but it's not there right
 now.

 Another thing that came up in the cells meeting this week is that if we
 didn't make the flavor soft-deletable, we could still show the flavor
 information for a given instance via the server GET API. However, that
 would
 be a microversion change to show the full flavor information for the
 server
 rather than just the flavor id.

>>>
>>> This is really only possible because we now store flavor info inside
>>> every instance object.
>>> I think before that, deleting the flavor would make some instance
>>> operations fail.
>>>
>>
>> Because we now store flavor info with each instance it opens up the
>> following possibility:
>>
>> Currently the flavor portion of an instance show request looks like
>> "flavor": {"id": "8", "links": [{"href": "https://example.com/flavors/8;,
>> "rel": "bookmark"}]}
>>
>> If that were instead changed to return 
>> "https://example.com/servers//flavor"
>> as the link and we exposed the flavor information stored with the instance
>> on that endpoint then we no longer need to expose deleted flavors.
>>
>> So for an instance booted with flavor 8 https://example.com/flavors/8
>> would be equivalent to https://example.com/servers//flavor except
>> that the latter would be available even if flavor 8 were deleted.
>>
>> Does that seem like an acceptable path for users?
>>
>>
> Yes, that's something we discussed on IRC yesterday which I think would be
> very good. It means a microversion but it means that operators would be
> very happy because if a flavor is removed, the flavor URI would still be
> working.
>
> +1000 to that.
>
> -Sylvain
>
>
>
>
>>> Thoughts? I'm cross-posting this to -dev and the -operators list to see
 what
 kind of user impact there would be if we didn't soft-delete flavors in
 the
 API DB (so you couldn't look up deleted flavors in the API).

>>>
>>> In balance, I think we should not allow soft_delete of flavors in the
>>> API DB.
>>>
>>> In a related note, and I thinking about a backlog spec looking at a
>>> flavor lifecycle. Thinking about early release, to production, then
>>> phasing out of flavors. I don't think soft delete is needed for that.
>>>
>>> Thanks,
>>> johnthetubaguy
>>>
>>> __
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> __
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> 

Re: [Openstack-operators] Confirm resize in Kilo with Cells

[Belmiro Moreira]

Hi Mathieu,
thanks for the related bugs.
But I'm observing this on 2015.1.1.

On Sun, Nov 22, 2015 at 12:58 AM, Mathieu Gagné <mga...@internap.com> wrote:

> On 2015-11-21 4:47 PM, Belmiro Moreira wrote:
> > Hi,
> > We are about to upgrade nova to kilo using cells and we noticed
> > the resize/migrate functionality is not working properly.
> >
> > The instance is correctly resized/migrated but fails to
> > “confirm resize” with the following trace:
> >
> > 2015-11-21 22:40:49.804 26786 ERROR nova.api.openstack.wsgi
> > [req-67f6a225-602c-426c-a33a-02c3f5e3093d belmiro
> > 40c7729b-c8e5-432f-81d5-a41543c8bf13] Exception handling resource:
> > 'NoneType' object has no attribute '__getitem__'
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi Traceback
> > (most recent call last):
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> > "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821,
> > in _process_stack
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
> > action_result = self.dispatch(meth, request, action_args)
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> > "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911,
> > in dispatch
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
> > method(req=request, **action_args)
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> >
> "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py",
> line
> > 823, in _action_confirm_resize
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
> > self.compute_api.confirm_resize(context, instance)
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> > "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 233, in
> wrapped
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
> > func(self, context, target, *args, **kwargs)
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> > "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 222, in
> inner
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
> > function(self, context, instance, *args, **kwargs)
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> > "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 250, in
> > _wrapped
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
> > fn(self, context, instance, *args, **kwargs)
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> > "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 203, in
> inner
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
> > f(self, context, instance, *args, **kw)
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> > "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2452, in
> > confirm_resize
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi deltas =
> > self._downsize_quota_delta(context, instance)
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> > "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2537, in
> > _downsize_quota_delta
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
> > API._resize_quota_delta(context, new_flavor, old_flavor, 1, -1)
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> > "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2504, in
> > _resize_quota_delta
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi if
> > compare * _quota_delta('vcpus') > 0:
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> > "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2501, in
> > _quota_delta
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
> > sense * (new_flavor[resource] - old_flavor[resource])
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi TypeError:
> > 'NoneType' object has no attribute '__getitem__'
> > 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
> >
> >
> >
> > I can't reproduce in master so this must have been fixed meanwhile,
> > however I can't identify the patch.
> >
> > Is someone experiencing the same problem? and know the commit that
> > fixed it?
> >
>
> I think those bugs are related and can be found in 2015.1.1:
> https://bugs.launchpad.net/nova/+bug/1459758
> https://bugs.launchpad.net/nova/+bug/1446082
>
> I know we had serious issues with flavor migrations not being properly
> done without those patches.
>
> --
> Mathieu
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Confirm resize in Kilo with Cells

[Belmiro Moreira]

Hi Matt,
I'm rolling kilo 2015.1.1

After resizing an instance (state: verify_resize) I have the following in
the instance_extra table:

API cell DB:
{"new": null, "old": null, "cur": {"nova_object.version": "1.1",
"nova_object.changes": ["root_gb", "name", "ephemeral_gb", "memory_mb",
"vcpus", "extra_specs", "swap", "rxtx_factor", "flavorid", "vcpu_weight",
"id"], "nova_object.name": "Flavor", "nova_object.data": {"root_gb": 20,
"name": "m1.small", "ephemeral_gb": 0, "memory_mb": 2048, "vcpus": 1,
"extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "flavorid": "2",
"vcpu_weight": null, "id": 5}, "nova_object.namespace": "nova"}}

and in the compute cell DB:
{"new": {"nova_object.version": "1.1", "nova_object.changes": ["deleted",
"ephemeral_gb", "updated_at", "disabled", "extra_specs", "rxtx_factor",
"is_public", "flavorid", "id", "root_gb", "deleted_at", "name",
"created_at", "memory_mb", "vcpus", "swap", "vcpu_weight"], "
nova_object.name": "Flavor", "nova_object.data": {"disabled": false,
"root_gb": 40, "name": "m1.medium", "flavorid": "3", "deleted": false,
"created_at": null, "ephemeral_gb": 0, "updated_at": null, "memory_mb":
4096, "vcpus": 2, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0,
"is_public": true, "deleted_at": null, "vcpu_weight": 0, "id": 1},
"nova_object.namespace": "nova"}, "old": {"nova_object.version": "1.1",
"nova_object.changes": ["root_gb", "name", "ephemeral_gb", "memory_mb",
"vcpus", "extra_specs", "swap", "rxtx_factor", "flavorid", "vcpu_weight",
"id"], "nova_object.name": "Flavor", "nova_object.data": {"root_gb": 20,
"name": "m1.small", "ephemeral_gb": 0, "memory_mb": 2048, "vcpus": 1,
"extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "flavorid": "2",
"vcpu_weight": null, "id": 5}, "nova_object.namespace": "nova"}, "cur":
{"nova_object.version": "1.1", "nova_object.changes": ["deleted",
"ephemeral_gb", "updated_at", "disabled", "extra_specs", "rxtx_factor",
"is_public", "flavorid", "id", "root_gb", "deleted_at", "name",
"created_at", "memory_mb", "vcpus", "swap", "vcpu_weight"], "
nova_object.name": "Flavor", "nova_object.data": {"disabled": false,
"root_gb": 40, "name": "m1.medium", "flavorid": "3", "deleted": false,
"created_at": null, "ephemeral_gb": 0, "updated_at": null, "memory_mb":
4096, "vcpus": 2, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0,
"is_public": true, "deleted_at": null, "vcpu_weight": 0, "id": 1},
"nova_object.namespace": "nova"}}


The quota is calculated in the API cell and the new flavor information is
missing there.
I don't see any relevant/related logs using DEBUG.

thanks,
Belmiro


On Sun, Nov 22, 2015 at 1:02 AM, Matt Riedemann <mrie...@linux.vnet.ibm.com>
wrote:

>
>
> On 11/21/2015 3:47 PM, Belmiro Moreira wrote:
>
>> Hi,
>> We are about to upgrade nova to kilo using cells and we noticed
>> the resize/migrate functionality is not working properly.
>>
>> The instance is correctly resized/migrated but fails to
>> “confirm resize” with the following trace:
>>
>> 2015-11-21 22:40:49.804 26786 ERROR nova.api.openstack.wsgi
>> [req-67f6a225-602c-426c-a33a-02c3f5e3093d belmiro
>> 40c7729b-c8e5-432f-81d5-a41543c8bf13] Exception handling resource:
>> 'NoneType' object has no attribute '__getitem__'
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi Traceback
>> (most recent call last):
&g

[Openstack-operators] Confirm resize in Kilo with Cells

[Belmiro Moreira]

Hi,
We are about to upgrade nova to kilo using cells and we noticed
the resize/migrate functionality is not working properly.

The instance is correctly resized/migrated but fails to
“confirm resize” with the following trace:

2015-11-21 22:40:49.804 26786 ERROR nova.api.openstack.wsgi
[req-67f6a225-602c-426c-a33a-02c3f5e3093d belmiro
40c7729b-c8e5-432f-81d5-a41543c8bf13] Exception handling resource:
'NoneType' object has no attribute '__getitem__'
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi Traceback (most
recent call last):
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821, in
_process_stack
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
action_result = self.dispatch(meth, request, action_args)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911, in
dispatch
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
method(req=request, **action_args)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py",
line 823, in _action_confirm_resize
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
self.compute_api.confirm_resize(context, instance)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 233, in wrapped
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
func(self, context, target, *args, **kwargs)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 222, in inner
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
function(self, context, instance, *args, **kwargs)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 250, in
_wrapped
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
fn(self, context, instance, *args, **kwargs)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 203, in inner
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
f(self, context, instance, *args, **kw)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2452, in
confirm_resize
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi deltas =
self._downsize_quota_delta(context, instance)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2537, in
_downsize_quota_delta
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
API._resize_quota_delta(context, new_flavor, old_flavor, 1, -1)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2504, in
_resize_quota_delta
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi if compare
* _quota_delta('vcpus') > 0:
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2501, in
_quota_delta
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
sense * (new_flavor[resource] - old_flavor[resource])
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi TypeError:
'NoneType' object has no attribute '__getitem__'
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi



I can't reproduce in master so this must have been fixed meanwhile,
however I can't identify the patch.

Is someone experiencing the same problem? and know the commit that
fixed it?

thanks,
Belmiro
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Confirm resize in Kilo with Cells

[Belmiro Moreira]

Hi,
We are about to upgrade nova to kilo using cells and we noticed
the resize/migrate functionality is not working properly.

The instance is correctly resized/migrated but fails to
“confirm resize” with the following trace:

2015-11-21 22:40:49.804 26786 ERROR nova.api.openstack.wsgi
[req-67f6a225-602c-426c-a33a-02c3f5e3093d belmiro
40c7729b-c8e5-432f-81d5-a41543c8bf13] Exception handling resource:
'NoneType' object has no attribute '__getitem__'
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi Traceback (most
recent call last):
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821, in
_process_stack
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
action_result = self.dispatch(meth, request, action_args)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911, in
dispatch
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
method(req=request, **action_args)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py",
line 823, in _action_confirm_resize
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
self.compute_api.confirm_resize(context, instance)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 233, in wrapped
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
func(self, context, target, *args, **kwargs)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 222, in inner
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
function(self, context, instance, *args, **kwargs)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 250, in
_wrapped
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
fn(self, context, instance, *args, **kwargs)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 203, in inner
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
f(self, context, instance, *args, **kw)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2452, in
confirm_resize
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi deltas =
self._downsize_quota_delta(context, instance)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2537, in
_downsize_quota_delta
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
API._resize_quota_delta(context, new_flavor, old_flavor, 1, -1)
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2504, in
_resize_quota_delta
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi if compare
* _quota_delta('vcpus') > 0:
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2501, in
_quota_delta
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return
sense * (new_flavor[resource] - old_flavor[resource])
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi TypeError:
'NoneType' object has no attribute '__getitem__'
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi



I can't reproduce in master so this must have been fixed meanwhile,
however I can't identify the patch.

Is someone experiencing the same problem? and know the commit that
fixed it?

thanks,
Belmiro
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Running mixed stuff Juno & Kilo , Was: cinder-api with rbd driver ignores ceph.conf

[Belmiro Moreira]

Hi Saverio,
we always upgrade one component at a time.
Cinder was one of the first components that we upgraded to kilo,
meaning that other components (glance, nova, ...) were running Juno.

We didn't have any problem with this setup.

Belmiro
CERN

On Tue, Nov 17, 2015 at 6:01 PM, Saverio Proto  wrote:

> Hello there,
>
> I need to quickly find a workaround to be able to use ceph object map
> features for cinder volumes with rbd backend.
>
> However, upgrading everything from Juno to Kilo will require a lot of
> time for testing and updating all my puppet modules.
>
> Do you think it is feasible to start updating just cinder to Kilo ?
> Will it work with the rest of the Juno components ?
>
> Has someone here experience in running mixed components between Juno and
> Kilo ?
>
> thanks
>
> Saverio
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [Scale][Performance] / compute_nodes ratio experience

[Belmiro Moreira]

Hi,
we are still running nova Juno and I don't see this performance issue.
(I can comment on Kilo next week).

Per cell, we have a node that runs conductor + other control plane services.
The number of conductor workers can change between 16 to 48.
We try to not have more than 200 compute nodes per cell.

Belmiro
CERN

On Wed, Nov 18, 2015 at 10:56 AM, Dina Belova  wrote:

> Dear operators,
>
> yesterday we (Performance Team) had weekly IRC meeting
> ,
> one of the things on board to discuss was nova-conductor performance issue
>  raised by
> Kris Lindgren (GoDaddy).
>
> There are still things to investigate and several suggestions about nature
> of this behaviour, but one of the questions/ideas raised was "If the
> conductor nodes to compute nodes ratio was ok?".
>
> In fact I never saw any recommendations on safe enough ratio for conductor
> nodes or networking nodes or whatever nodes vs compute ones.
>
> As providing OpenStack community with understandable recommendations and
> instructions on performant OpenStack cloud deployments is part of
> Performance Team mission, I'm kindly asking you to share your experience on
> safe cloud deployment ratio between various types of nodes you're having
> right now and the possible issues you observed (as an example: discussed
> GoDaddy's cloud is having 3 conductor boxes vs 250 computes in the cell,
> and there was an opinion that it's simply not enough and that's it).
>
> Thanks in advance!
>
> Cheers,
> Dina
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Informal Ops Meetup?

[Belmiro Moreira]

+1

Belmiro




On Thursday, 29 October 2015, Kris G. Lindgren 
wrote:

> We seem to have enough interest… so meeting time will be at 10am in the
> Prince room (if we get an actual room I will send an update).
>
> Does anyone have any ideas about what they want to talk about?  I am
> pretty much open to anything.  I started:
> https://etherpad.openstack.org/p/TYO-informal-ops-meetup  for tracking of
> some ideas/time/meeting place info.
>
> ___
> Kris Lindgren
> Senior Linux Systems Engineer
> GoDaddy
>
> From: Sam Morrison  >
> Date: Thursday, October 29, 2015 at 6:14 PM
> To: "openstack-operators@lists.openstack.org
> "
>  >
> Subject: Re: [Openstack-operators] Informal Ops Meetup?
>
> I’ll be there, talked to Tom too and he said there may be a room we can
> use else there is plenty of space around the dev lounge to use.
>
> See you tomorrow.
>
> Sam
>
>
> On 29 Oct 2015, at 6:02 PM, Xav Paice  > wrote:
>
> Suits me :)
>
> On 29 October 2015 at 16:39, Kris G. Lindgren  > wrote:
>
>> Hello all,
>>
>> I am not sure if you guys have looked at the schedule for Friday… but its
>> all working groups.  I was talking with a few other operators and the idea
>> came up around doing an informal ops meetup tomorrow.  So I wanted to float
>> this idea by the mailing list and see if anyone was interested in trying to
>> do an informal ops meet up tomorrow.
>>
>> ___
>> Kris Lindgren
>> Senior Linux Systems Engineer
>> GoDaddy
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [Large Deployments Team] Updated Agenda for Later Today

[Belmiro Moreira]

Hi,
just added our use-cases/patches to the etherpad.

Belmiro

On Fri, Jun 19, 2015 at 11:09 PM, Kris G. Lindgren klindg...@godaddy.com
wrote:

   Mike added our use case to the etherpad [1] today.  I talked it over
 with Carl Baldwin and he seemed ok with the format.  If you guys want to
 add your uses cases to the etherpad - please do, as now is the time to make
 your voice heard.

  Mike and I will be at the Neutron mid-cycle to discuss these use cases
 as well as the initial Neutron Segmentation ask [2].  I believe we will
 also be discussing the IP usages spec [3]

  [1] - https://etherpad.openstack.org/p/Network_Segmentation_Usecases
 [2] - https://bugs.launchpad.net/neutron/+bug/1458890
 [3] - https://review.openstack.org/#/c/180803/
  

 Kris Lindgren
 Senior Linux Systems Engineer
 GoDaddy, LLC.

   From: Sam Morrison sorri...@gmail.com
 Date: Thursday, June 18, 2015 at 5:44 PM
 To: Matt Van Winkle mvanw...@rackspace.com
 Cc: openstack-operators@lists.openstack.org 
 openstack-operators@lists.openstack.org
 Subject: Re: [Openstack-operators] [Large Deployments Team] Updated
 Agenda for Later Today

   The auto generated minutes are much appreciated!

  In terms of segmentation I think our use-cases are pretty much all the
 same so we should be able to get consensus on an ether pad.

  Sam


   On 18 Jun 2015, at 8:15 pm, Matt Van Winkle mvanw...@rackspace.com
 wrote:

  Hey folks,
 Our monthly meeting for LDT is already upon us.  I have updated the agenda
 - https://wiki.openstack.org/wiki/Meetings/LDT.  Today, we'll probably
 spend most of the time discussing some follow-up from our YVR session
 related to Neutron and segmentation.  Kris has done some great work
 following up on the matter with the devs and we should keep pushing on this.

  See you all at 16:00 UTC

  Thanks!
 Matt
  ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Random NUMA cell selection can leave NUMA cells unused

[Belmiro Moreira]

Hi,
I would like to raise your attention for the bug
https://bugs.launchpad.net/nova/+bug/1461777
since it can impact the efficiency of your cloud.
It affects Juno and Kilo deployments.

Belmiro
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] max_age and until_refresh for fixing Nova quotas

[Belmiro Moreira]

Hi,
I just posted in our operations blog how CERN is dealing with quotas
synchronization problem.

http://openstack-in-production.blogspot.fr/2015/03/nova-quota-usage-synchronization.html

Hope it helps,
cheers,
Belmiro



On Sat, Mar 21, 2015 at 12:55 AM, Sam Morrison sorri...@gmail.com wrote:

 I’d need to go through the logs to see exactly the occurrence maybe 10
 minutes was a bit much. We run the cron job every 10 mins.

 The cron job stopped a few weeks back and we had at least half a dozen
 projects with out of sync quotas within a few hours.

 Cheers,
 Sam


  On 21 Mar 2015, at 4:01 am, Mike Dorman mdor...@godaddy.com wrote:
 
  Hey Sam,
 
  When you say it occurs every 10 minutes, what exactly do you mean?  The
  quota refresh?  Or a quota getting out of sync?
 
  I am surprised you have max_age set so low.  I would think that would
  basically trigger a quota refresh on every single reservation for most
  users, right?
 
  Mike
 
 
 
 
 
  On 3/20/15, 4:18 AM, Sam Morrison sorri...@gmail.com wrote:
 
  We’ve had the following for a year or so but doesn’t help much, we still
  see it occurring every 10 mins or so.
 
  max_age = 10
  until_refresh = 5
  reservation_expire=600
 
  We have a cron job that runs every 10 mins that figures out what
 projects
  are out of sync and corrects them.
  We’ve always been scared of setting these to zero but we probably
 should.
 
  Sam
 
 
  On 15 Mar 2015, at 2:53 pm, Mike Dorman mdor...@godaddy.com wrote:
 
  Yeah the default is just ‘0’ for both, which disables the refresh.
 
 
 
  The one downside is that it may not be 100% transparent to the user.
  If
  the quota is already (incorrectly) too high, and exceeding the quota
  limit, the reservation that triggers the refresh will still fail.  I.e.
  the reservation is attempted based on the quota usage values _before_
  the
  refresh.  But then after that the quota should be fixed and it will
  work
  again on the next reservation.
 
  But my thinking is that most quota issues happen slowly over time.  If
  we
  are correcting them often and automatically, they hopefully never get
  to
  the point where they’re bad enough to manifest reservation errors to
  the
  user.
 
  I don’t have any information re: db load.  I assume it regenerates
  based
  on what’s in the instances or reservations table.  I imagine the load
  for
  doing a single refresh is probably comparable to doing a ‘nova list’.
 
  Mike
 
 
 
  On 3/14/15, 2:27 PM, Tim Bell tim.b...@cern.ch wrote:
 
  Interesting... what are the defaults ?
 
  Assuming no massive DB load, getting synced within a day would seem
  reasonable. Is the default no max age ?
 
  Tim
 
  -Original Message-
  From: Jesse Keating [mailto:j...@bluebox.net]
  Sent: 14 March 2015 16:59
  To: openstack-operators@lists.openstack.org
  Subject: Re: [Openstack-operators] max_age and until_refresh for
  fixing
  Nova
  quotas
 
  On 3/14/15 8:11 AM, Mike Dorman wrote:
  I did short write-up here http://t.co/Q5X1hTgJG1 if you are
  interested
  in the details.
 
 
  Thanks for sharing Matt! That's an excellent write up.
 
  --
  -jlk
 
  ___
  OpenStack-operators mailing list
  OpenStack-operators@lists.openstack.org
 
 
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
 
  ___
  OpenStack-operators mailing list
  OpenStack-operators@lists.openstack.org
 
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
  ___
  OpenStack-operators mailing list
  OpenStack-operators@lists.openstack.org
 
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
 
  ___
  OpenStack-operators mailing list
  OpenStack-operators@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Deprecation of in tree EC2 API in Nova for Kilo release

[Belmiro Moreira]

I completely agree with Tim and Daniel.
Also, deprecating nova EC2 API without having the community engaged with
the new stackforge “EC2 standalone service” can lead to a no EC2 support at
all.

On Thu, Jan 29, 2015 at 4:46 AM, Saju M sajup...@gmail.com wrote:

 I think, new EC2 API also uses EC2 API in the nova for basic operations.
 Only the new features like VPC is implemented in new EC2 API. So I think we
 should move it first.
 On Jan 29, 2015 5:42 AM, Alex Baretto shakamu...@gmail.com wrote:

 Hi Tim,

 Would you be able to share use cases/experiences with EC2 API to help
 with support for your proposals? I am also familiar with multiple OpenStack
 clouds using the EC2 API.

 Best,

 -Alex.

 On 28 January 2015 at 12:38, Tim Bell tim.b...@cern.ch wrote:

 There are currently 35% of openstack clouds from the survey using this
 feature. I would therefore propose that this be reviewed with the operators
 in detail before a rapid depreciation is performed. To the best of my
 knowledge, the new package is not yet in use in production.

 At CERN, we are highly dependent on this function and while it does not
 provide 100% compatibility, it works pretty well.

 I would therefore propose that the new EC2 API modules are validated in
 production and at scale before depreciating the existing functions. I think
 validating, packaging and deploying to a reasonable number of clouds and
 reviewing it with the operators is a viable target for Kilo. This can then
 be reviewed in one of the joint developer/operator sessions for feasibility.

 Tim
  -Original Message-
  From: Christian Berendt [mailto:christ...@berendt.io]
  Sent: 28 January 2015 21:29
  To: openstack-operators@lists.openstack.org
  Subject: Re: [Openstack-operators] Deprecation of in tree EC2 API in
 Nova for
  Kilo release
 
  On 01/28/2015 08:56 PM, Sean Dague wrote:
   Comments are welcomed, but we've attempted to get more people engaged
   to address these issues over the last 18 months, and never really had
   anyone step up. Without some real maintainers of this code in Nova
   (and tests somewhere in the community) it's really no longer viable.
 
  I think it is the right decision to remove the EC2 API in Nova.
 
  Christian.
 
  ___
  OpenStack-operators mailing list
  OpenStack-operators@lists.openstack.org
 
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators




 --
 Alex Barreto
 about.me/shakamunyi
 [image: Alex Barreto on about.me]
   http://about.me/shakamunyi
 http://about.me/shakamunyi
 (408) 442-4181 o | (415) 307-8594 m http://about.me/shakamunyi
 http://about.me/shakamunyishakamu...@gmail.com


 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Way to check compute - rabbitmq connectivity

[Belmiro Moreira]

Hi,
we had similar issues.
In our case, some times (not really a pattern here!) nova-compute didn't
consume messages even if everything was apparently happy.
We started monitoring the queues size and restarting nova-compute.

We are still using python-oslo-messaging-1.3.0.2, however the problem
disappeared when we upgraded to python-kombu-2.5.15 and
rabbitmq-server-3.3.5

Belmiro
---
CERN

On Fri, Jan 16, 2015 at 1:48 AM, Sam Morrison sorri...@gmail.com wrote:

 We’ve had a lot of issues with Icehouse related to rabbitMQ. Basically the
 change from openstack.rpc to oslo.messaging broke things. These things are
 now fixed in oslo.messaging version 1.5.1, there is still an issue with
 heartbeats and that patch is making it’s way through review process now.

 https://review.openstack.org/#/c/146047/

 Cheers,
 Sam


 On 16 Jan 2015, at 10:55 am, sridhar basam sridhar.ba...@gmail.com
 wrote:


 If you are using ha queues, use a version of rabbitmq  3.3.0. There was a
 change in that version where consumption on queues was automatically
 enabled when a master election for a queue happened. Previous versions only
 informed clients that they had to reconsume on a queue. It was the clients
 responsibility to start consumption on a queue.

 Make sure you enable tcp keepalives to a low enough value in case you have
 a firewall device in between your rabbit server and it's consumers.

 Monitor consumers on your rabbit infrastructure using 'rabbitmqctl
 list_queues name messages consumers'. Consumers on fanout queues is going
 to depend on the number of services of any type you have in your
 environment.

 Sri
  On Jan 15, 2015 6:27 PM, Michael Dorman mdor...@godaddy.com wrote:

   Here is the bug I’ve been tracking related to this for a while.  I
 haven’t really kept up to speed with it, so I don’t know the current status.

  https://bugs.launchpad.net/nova/+bug/856764


   From: Kris Lindgren klindg...@godaddy.com
 Date: Thursday, January 15, 2015 at 12:10 PM
 To: Gustavo Randich gustavo.rand...@gmail.com, OpenStack Operators 
 openstack-operators@lists.openstack.org
 Subject: Re: [Openstack-operators] Way to check compute - rabbitmq
 connectivity

   During the Atlanta ops meeting this topic came up and I specifically
 mentioned about adding a no-op or healthcheck ping to the rabbitmq stuff
 to both nova  neutron.  The dev's in the room looked at me like I was
 crazy, but it was so that we could exactly catch issues as you described.
 I am also interested if any one knows of a lightweight call that could be
 used to verify/confirm rabbitmq connectivity as well.  I haven't been able
 to devote time to dig into it.  Mainly because if one client is having
 issues - you will notice other clients are having similar/silent errors and
 a restart of all the things is the easiest way to fix, for us atleast.
  

 Kris Lindgren
 Senior Linux Systems Engineer
 GoDaddy, LLC.


   From: Gustavo Randich gustavo.rand...@gmail.com
 Date: Thursday, January 15, 2015 at 11:53 AM
 To: openstack-operators@lists.openstack.org 
 openstack-operators@lists.openstack.org
 Subject: Re: [Openstack-operators] Way to check compute - rabbitmq
 connectivity

Just to add one more background scenario, we also had similar
 problems trying to load balance rabbitmq via F5 Big IP LTM. For that reason
 we don't use it now. Our installation is a single rabbitmq instance and no
 intermediaries (albeit network switches). We use Folsom and Icehouse, the
 problem being perceived more in Icehouse nodes.

  We are already monitoring message queue size, but we would like to
 pinpoint in semi-realtime the specific hosts/racks/network paths
 experiencing the stale connection before a user complains about an
 operation being stuck, or even hosts with no such pending operations but
 already disconnected -- we also could diagnose possible network causes
 and avoid massive service restarting.

  So, for now, if someone knows about a cheap and quick openstack
 operation that triggers a message interchange between rabbitmq and
 nova-compute and a way of checking the result it would be great.




 On Thu, Jan 15, 2015 at 1:45 PM, Kris G. Lindgren klindg...@godaddy.com
 wrote:

  We did have an issue using celery  on an internal application that we
 wrote - but I believe it was fixed after much failover testing and code
 changes.  We also use logstash via rabbitmq and haven't noticed any issues
 there either.

  So this seems to be just openstack/oslo related.

  We have tried a number of different configurations - all of them had
 their issues.  We started out listing all the members in the cluster on the
 rabbit_hosts line.  This worked most of the time without issue, until we
 would restart one of the servers, then it seemed like the clients wouldn't
 figure out they were disconnected and reconnect to the next host.

  In an attempt to solve that we moved to using harpoxy to present a vip
 that we configured in the 

[Openstack-operators] New services disable reason

[Belmiro Moreira]

Hi,
as operators I would like to have your comments/suggestions on:
https://review.openstack.org/#/c/136645/1


With a large number of nodes several services are disabled because various
reasons (in our case mainly hardware interventions).
To help operations we use the disable reason as fast filter to identify
why the service is disabled.

At same time, we add several new nodes (nova-compute) per week.
At CERN to avoid adding a service when the daemon starts for the first time
nova is configured with:
enable_new_services=False
This is great, however no disable reason is set.
For us having services disabled with no reason specified creates additional
checks.

How are others keeping track of disabled services?


Belmiro
---
CERN
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] DB archive deleted rows

[Belmiro Moreira]

Hi,
our nova DBs are growing rapidly and it's time to start pruning them...

I'm trying the archive deleted rows however is not working and I'm
getting the following
warning in the logs: IntegrityError detected when archiving table
Searching about this problem I found the bug 
https://bugs.launchpad.net/nova/+bug/1183523;
which, if I understood correctly, means this functionality is broken for a
while...

How are other deployments dealing with growing DBs?

thanks,
Belmiro
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] InstanceInfoCacheNotFound: Info cache for instance X could not be found.

[Belmiro Moreira]

Hi,
I'm observing exactly the same problem.
But in my case it is happening every time a VM is deleted.

I'm using icehouse.

Any idea?

regards,
Belmiro



On Fri, Jul 18, 2014 at 10:22 AM, Alex Leonhardt aleonhardt...@gmail.com
wrote:

 Hi All,

 I keep seeing this in the logs when deleting an instance ( and it takes
 significantly longer than on other Hyper Visors ) :

 2014-07-18 09:16:50.453 23230 ERROR nova.virt.driver [-] Exception
 dispatching event nova.virt.event.LifecycleEvent object at 0x3382650:
 Info cache for instance {uuid goes here} could not be found.
 Traceback (most recent call last):

   File /usr/lib/python2.6/site-packages/nova/conductor/manager.py, line
 597, in _object_dispatch
 return getattr(target, method)(context, *args, **kwargs)

   File /usr/lib/python2.6/site-packages/nova/objects/base.py, line 151,
 in wrapper
 return fn(self, ctxt, *args, **kwargs)

   File /usr/lib/python2.6/site-packages/nova/objects/instance.py, line
 500, in refresh
 self.info_cache.refresh()

   File /usr/lib/python2.6/site-packages/nova/objects/base.py, line 151,
 in wrapper
 return fn(self, ctxt, *args, **kwargs)

   File
 /usr/lib/python2.6/site-packages/nova/objects/instance_info_cache.py,
 line 103, in refresh
 self.instance_uuid)

   File /usr/lib/python2.6/site-packages/nova/objects/base.py, line 112,
 in wrapper
 result = fn(cls, context, *args, **kwargs)

   File
 /usr/lib/python2.6/site-packages/nova/objects/instance_info_cache.py,
 line 70, in get_by_instance_uuid
 instance_uuid=instance_uuid)

 InstanceInfoCacheNotFound: Info cache for instance {uuid goes here} could
 not be found.


 I had a look around and there used to be a bug that got fixed, I cant seem
 to see this on happening on a different HV (lets call it stack1) that runs
 the same version of openstack, however, that HV (stack1) was provisioned
 with packstack vs the one reporting above (stack2) via my own puppet
 module/manifest ...

 When deleting the VM from stack2, it also takes significantly longer than
 on stack1 ...

 Anyone got any ideas ? I dont see the same happening on stack1.

 Thanks all!
 Alex


 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators