Update - you can take a look into comments about Mesos maintaining
upgrade process + my response to it that breaks the topic into a list of
OpenStack services (starting with a minimal list that is necessary to
build a working cloud) and their requirements in terms of data storage.
TL; DR: I still think in some cases we need to land containers on the
same slave after upgrade, please provide your feedback.
Thanks!
-marek
---------------------------------
[G.O] As I remember the spec for Mesos assumes that self-configuring
service will be used. There is another spec for oslo.config to support
remote configuration storages like ZooKeeper, ETCd, Consul. This
approach should simplify an upgrade process as most of the configuration
part will be done automatically by the service container itself. I
think, we need to discuss the ways how OpenStack service can be upgraded
and provide a baseline standards(requirements) for OpenStack services so
that OpenStack service code will support one or another way for
upgrades. Marathon framewok should support at least two ways of
upgrades:(https://mesosphere.github.io/marathon/docs/deployment-design-doc.html)
<https://mesosphere.github.io/marathon/docs/deployment-design-doc.html%29>
1) Rolling-Upgrade (Canary)
2) Green-Blue (A/B) upgrades
As an operator I should be able to select the specific version of
container which I want to roll-out to the existing cloud and I have to
be able to do a rolling-back operation in case of upgrade failure.
If we need to use volume based configuration storage then it should
rely on Mesos Volume management
(http://mesos.apache.org/documentation/latest/persistent-volume/)
<http://mesos.apache.org/documentation/latest/persistent-volume/%29>which is
not released yet as I know. Mesos/Marathon should be able to place
upgared container correctly and we should not define any contstrains for
that in the request. We still might use constraints but for providing
more flexible/complex rolling-upgrade process like upgrading only
specific number of instances at once.
[M.Z.] I agree in general about Mesos maintaing upgrades but in some
cases it's not about volumes but underlying (host-based) data (nova,
cinder).
Let's break it down into a list of OpenStack services (starting with a
minimal list that is necessary to build a working cloud) and their
requirements in terms of data storage:
1. nova: we need to ensure upgraded nova container is started on the
same slave so it can reconnect to hypervisor and see the VMs. Not a
candidate for Mesos Volumes (MVs).
2. cinder: must be on the same host to see block devices it uses for
storage. Not a candidate for MVs.
3. mariadb: must be on the same host to see directory it uses for data.
In the future when we'll use gallera perhaps it may be a group of hosts,
not just 1 host. Sounds like a candidate for MVs.
4. keystone: does not care about the host
5. neutron: does not care about the host
Additional notes:
- currently for neutron we mount "/lib/modules" and do modprobe from the
inside of the container to make neutron working - isnt' this wrong by
design? Slave host should be prepared beforehand and load necessary modules.
- since for now config files are stored on slaves in a well-known path
(this is provisioned by ansible) we can assume each slave is identical
in this regard so we can easily move services that do not use data
storage between slaves
* - also I don't see how we can avoid sticking to the same slave after
upgrade (whatever rolling or green/blue) without MVs for mariadb and
for nova & cinder/local disks at all.
* - I can see cinder container not being host-dependent as long as
it's not using local disks for storage (but ceph for example)
On 16.12.2015 18:16, Marek Zawadzki wrote:
Hello all,
I described use case and simple tool that I'd like to implement as a
first step in this topic - would you please
review it and provide me with feedback before I start coding?
Is the use-case realistic? Is this tool going to be useful given the
features I described? Any other comments?
https://etherpad.openstack.org/p/kolla_upgrading_containers
Thank you,
-marek zawadzki
--
Marek Zawadzki
Mirantis Kolla Team
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
