Hi Steve,
On 25/09/2018 10:51, Steven Hardy wrote:
Hi all,
After some discussions with bandini at the PTG, I've been taking a
look at this bug and how to solve it:
https://bugs.launchpad.net/tripleo/+bug/1792613
(Also more information in downstream bz1626140)
The problem is that we always run container bootstrap tasks (as well
as a bunch of update/upgrade tasks) on the bootstrap_nodeid, which by
default is always the overcloud-controller-0 node (regardless of which
services are running there).
This breaks a pattern we established a while ago for Composable HA,
where we' work out the bootstrap node by
$service_short_bootstrap_hostname, which means we always run on the
first node that has the service enabled (even if it spans more than
one Role).
This presents two problems:
1. service_config_settings only puts the per-service hieradata on
nodes where a service is enabled, hence data needed for the
bootstrapping (e.g keystone users etc) can be missing if, say,
keystone is running on some role that's not Controller (this, I think
is the root-cause of the bug/bz linked above)
2. Even when we by luck have the data needed to complete the bootstrap
tasks, we'll end up pulling service containers to nodes where the
service isn't running, potentially wasting both time and space.
I've been looking at solutions, and it seems like we either have to
generate per-service bootstrap_nodeid's (I have a patch to do this
https://review.openstack.org/605010), or perhaps we could just remove
all the bootstrap node id's, and switch to using hostnames instead?
Seems like that could be simpler, but wanted to check if there's
anything I'm missing?
I think we should recheck he initial assumptions, because based on my
testing:
* the bootstrap_nodeid is in fact a hostname already, despite its
deceptive name,
* it's not global, it is per-role.
From my env:
[root@overcloud-controller-2 ~]# hiera -c /etc/puppet/hiera.yaml
bootstrap_nodeid
overcloud-controller-0
[root@overcloud-novacompute-1 ~]# hiera -c /etc/puppet/hiera.yaml
bootstrap_nodeid
overcloud-novacompute-0
This makes me think the problems (1) and (2) as stated above shouldn't
be happening. The containers or tasks present in service definition
should be executed on all nodes where a service is present, and if they
additionally filter for bootstrap_nodeid, it would only pick one node
per role. So, the service *should* be deployed on the selected bootstrap
node, which means the service-specific hiera should be present there and
needless container downloading should not be happening, AFAICT.
However, thinking about it this way, we probably have a different
problem still:
(3) The actions which use bootstrap_nodeid check are not guaranteed to
execute once per service. In case the service is present on multiple
roles, the bootstrap_nodeid check succeeds once per such role.
Using per-service bootstrap host instead of per-role bootstrap host
sounds like going the right way then.
However, none of the above provides a solid explanation of what's really
happening in the LP/BZ mentioned above. Hopefully this info is at least
a piece of the puzzle.
Jirka
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
