We had two design summit sessions for cells v2 at the Ocata summit in
Barcelona. The full etherpads are here:
https://etherpad.openstack.org/p/ocata-nova-summit-cellsv2-scheduler
https://etherpad.openstack.org/p/ocata-nova-summit-cellsv2-quotas
In the first session we mostly talked about the work items needed to
support multiple cells.
Scheduler interaction
---------------------
Andrew Laski had the spec up for this and there was some POC code
started, which Dan Smith is picking up. The main idea here is to create
the BuildRequest in nova-api, then call a new conductor method which
calls the scheduler to pick a host, and then conductor creates the
instance in the cell mapped to that host, and then conductor deletes the
BuildRequest. Once the instance is in that cell, it doesn't move out of
that cell via reschedule/rebuild/migration/evacuate. We might add
support for that later, but it's not something supported in the initial
cells v2 scheduler work.
Cell0
-----
Cell0 is the special database (using the same cell DB schema) where
instances that failed to build go to die. In Newton this was optional
but the API will pull instances from it when listing instances. However,
we aren't populating it on schedule failure yet so that's work that
needs to be done for Ocata. So far there are no patches up for this yet,
but it will most likely be worked in with the scheduler interaction series.
The listing instances problem
-----------------------------
In order to list instances we need to be able to page across cells,
which is going to be expensive for multiple large cells. Long-term we
want to use searchlight for this, but in the short term we're going to
do a simple merge sort in python. We also need to get the fixes in to
restrict the filter/sort keys for listing instances, Alex Xu and Kevin
Zheng are working on that. There are open questions on the long-term
plans of how things are going to work with searchlight, and Chris Dent
said he was interested in working on that. As a start, the searchlight
team has started reporting bugs against nova for gaps in the
notifications that nova sends out compared to the REST API. Balazs
Gibizer (gibi) who runs the notifications subteam in nova has already
started triaging those bugs.
Testing
-------
CI testing the multi-cell configuration should be relatively
straight-forward with a multinode job. We can have one node contain the
'control' bits like the API, conductor, scheduler, API/cell0/cell DBs,
along with a nova-compute service, and then another node that is just
running nova-compute. I've volunteered to work on that.
Dan and I have also started working on a series of changes to make
cellsv2 required in the Ocata CI jobs:
https://review.openstack.org/#/q/topic:ocata-requires-cellsv2
This consists of a nova database migration that fails if you haven't
created the cell0 database and run the simple_cell_setup command.
Upgrade
-------
Today in non-cells deployments we say to upgrade conductors first, then
API and finally computes. With cells v2 we want to do rolling upgrades
of the cells, which means upgrading conductors and then computes in the
cells, and finally the API. This allows you to only enable the latest
features in the API when the cells are all upgraded and ready to handle
those requests. This poses a bit of a problem for our CI tooling with
devstack/grenade though as we upgrade and start nova-api first. That's
going to require some changes as we get into the multi-cell testing
mentioned above.
Quotas
------
In the second design summit session on cells v2 we spent the entire time
talking about quotas, and thinking about ways to potentially redo the
quotas design when moving those to the API database.
Melanie Witt has a spec up which proposes that instead of doing the
reserve (API), do work, commit/rollback (compute) model, we move commits
to the API and have a process of (1) do work and then (2) reserve and
commit.
While talking about that in the session, there was some brainstorming on
doing limit checks differently in the API. Basically, do away with
reservations, do a quick DB query to check quota before an operation
begins, and if it's OK go forward. There will be races as tenants reach
the end of quota limits, but maybe this is OK. The upside is we
shouldn't get out of sync (which has been a problem for operators to
deal with), but there is a potential to race for the last usages which
might lead to overconsumption of resources.
There are some open questions around the 'fast and loose' approach like
are there things we need to count which aren't in the API database, and
can we do this efficiently for things that nova doesn't track in it's
database, like floating/fixed IPs in neutron?
--
Thanks,
Matt Riedemann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
