I've been looking into the startup constraints involved when launching Nova services with systemd using Type=notify (which causes systemd to wait for an explicit notification from the service before considering it to be "started". Some services (e.g., nova-conductor) will happily "start" even if the backing database is currently unavailable (and will enter a retry loop waiting for the database).
Other services -- specifically, nova-scheduler -- will block waiting for the database *before* providing systemd with the necessary notification. nova-scheduler blocks because it wants to initialize a list of available aggregates (in scheduler.host_manager.HostManager.__init__), which it gets by calling objects.AggregateList.get_all. Does it make sense to block service startup at this stage? The database disappearing during runtime isn't a hard error -- we will retry and reconnect when it comes back -- so should the same situation at startup be a hard error? As an operator, I am more interested in "did my configuration files parse correctly?" at startup, and would generally prefer the service to start (and permit any dependent services to start) even when the database isn't up (because that's probably a situation of which I am already aware). It would be relatively easy to have the scheduler lazy-load the list of aggregates on first references, rather than at __init__. I'm not familiar enough with the nova code to know if there would be any undesirable implications of this behavior. We're already punting initializing the list of instances to an asynchronous task in order to avoid blocking service startup. Does it make sense to permit nova-scheduler to complete service startup in the absence of the database (and then retry the connection in the background)? -- Lars Kellogg-Stedman <l...@redhat.com> | larsks @ {freenode,twitter,github} Cloud Engineering / OpenStack | http://blog.oddbit.com/
signature.asc
Description: PGP signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev