On 9/5/2018 10:03 AM, Mohammed Naser wrote:
On Wed, Sep 5, 2018 at 10:57 AM Matt Riedemann<mriede...@gmail.com>  wrote:
On 9/5/2018 8:47 AM, Mohammed Naser wrote:
Could placement not do what happened for a while when the nova_api
database was created?
Can you be more specific? I'm having a brain fart here and not
remembering what you are referring to with respect to the nova_api DB.
I think there was a period in time where the nova_api database was created
where entires would try to get pulled out from the original nova database and
then checking nova_api if it doesn't exist afterwards (or vice versa).  One
of the cases that this was done to deal with was for things like instance types
or flavours.

I don't know the exact details but I know that older instance types exist in
the nova db and the newer ones are sitting in nova_api.  Something along
those lines?

Yeah that more about supporting online data migrations *within* nova where new records were created in the API DB and old records would be looked up in both the API DB and then if not found there, in the cell (traditional nova DB). But you'd also be running the "nova-manage db online_data_migrations" CLI to force the migration of the records from the cell DB to the API DB.

With Placement split out of nova, we can't really do that. You could point placement at the nova_api DB so it can pull existing records, but it would continue to create new records in the nova_api DB rather than the placement DB and at some point you have to make that data migration. Maybe you were thinking something like have temporary fallback code in placement such that if a record isn't found in the placement database, it queries a configured nova_api database? That'd be a ton of work at this point, and if it was something we were going to do, we should have agreed on that in YVR several months ago, definitely pre-extraction.

I say this because I know that moving the database is a huge task for
us, considering how big it can be in certain cases for us, and it
means control plane outage too
I'm pretty sure you were in the room in YVR when we talked about how
operators were going to do the database migration and were mostly OK
with what was discussed, which was a lot will just copy and take the
downtime (I think CERN said around 10 minutes for them, but they aren't
a public cloud either), but others might do something more sophisticated
and nova shouldn't try to pick the best fit for all.
If we're provided the list of tables used by placement, we could considerably
make the downtime smaller because we don't have to pull in the other huge
tables like instances/build requests/etc

There are no instances records in the API DB, maybe you mean instance_mappings? But yes I get the point.

What happens if things like server deletes happen while the placement service
is down?

The DELETE /allocations/{consumer_id} requests from nova to placement will fail with some keystoneauth1 exception, but because of our old friend @safe_connect we likely won't fail the server delete because we squash the exception from KSA:


However, you'd still have allocations in placement against resource providers (compute nodes) for instances that no longer exist, which means you're available capacity for scheduling new requests is diminished until those bogus allocations are purged from placement, which will take some scripting.

In other words, not good things.

I'm definitely interested in what you do plan to do for the database
migration to minimize downtime.
At this point, I'm thinking turn off placement, setup the new one, do
the migration
of the placement-specific tables (this can be a straightforward documented task
OR it would be awesome if it was a placement command (something along
the lines of `placement-manage db import_from_nova`) which would import all
the right things

You wouldn't also stop nova-api while doing this? Otherwise you're going to get into the data/resource tracking mess described above which will require some post-migration cleanup scripting.

The idea of having a command would be*extremely*  useful for deployment tools
in automating the process and it also allows the placement team to selectively
decide what they want to onboard?

Just throwing ideas here.




OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Reply via email to