On Tue, Sep 25, 2018 at 2:06 PM Jiří Stránský <ji...@redhat.com> wrote: > > Hi Steve,
Thanks for the reply - summary of our follow-up discussion on IRC below: > On 25/09/2018 10:51, Steven Hardy wrote: > > Hi all, > > > > After some discussions with bandini at the PTG, I've been taking a > > look at this bug and how to solve it: > > > > https://bugs.launchpad.net/tripleo/+bug/1792613 > > (Also more information in downstream bz1626140) > > > > The problem is that we always run container bootstrap tasks (as well > > as a bunch of update/upgrade tasks) on the bootstrap_nodeid, which by > > default is always the overcloud-controller-0 node (regardless of which > > services are running there). > > > > This breaks a pattern we established a while ago for Composable HA, > > where we' work out the bootstrap node by > > $service_short_bootstrap_hostname, which means we always run on the > > first node that has the service enabled (even if it spans more than > > one Role). > > > > This presents two problems: > > > > 1. service_config_settings only puts the per-service hieradata on > > nodes where a service is enabled, hence data needed for the > > bootstrapping (e.g keystone users etc) can be missing if, say, > > keystone is running on some role that's not Controller (this, I think > > is the root-cause of the bug/bz linked above) > > > > 2. Even when we by luck have the data needed to complete the bootstrap > > tasks, we'll end up pulling service containers to nodes where the > > service isn't running, potentially wasting both time and space. > > > > I've been looking at solutions, and it seems like we either have to > > generate per-service bootstrap_nodeid's (I have a patch to do this > > https://review.openstack.org/605010), or perhaps we could just remove > > all the bootstrap node id's, and switch to using hostnames instead? > > Seems like that could be simpler, but wanted to check if there's > > anything I'm missing? > > I think we should recheck he initial assumptions, because based on my > testing: > > * the bootstrap_nodeid is in fact a hostname already, despite its > deceptive name, > > * it's not global, it is per-role. > > From my env: > > [root@overcloud-controller-2 ~]# hiera -c /etc/puppet/hiera.yaml > bootstrap_nodeid > overcloud-controller-0 > > [root@overcloud-novacompute-1 ~]# hiera -c /etc/puppet/hiera.yaml > bootstrap_nodeid > overcloud-novacompute-0 > > This makes me think the problems (1) and (2) as stated above shouldn't > be happening. The containers or tasks present in service definition > should be executed on all nodes where a service is present, and if they > additionally filter for bootstrap_nodeid, it would only pick one node > per role. So, the service *should* be deployed on the selected bootstrap > node, which means the service-specific hiera should be present there and > needless container downloading should not be happening, AFAICT. Ah, I'd missed that because we have another different definition of the bootstrap node ID here: https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps.j2#L283 That one is global, because it only considers primary_role_name, which I think explains the problem described in the LP/BZ. > However, thinking about it this way, we probably have a different > problem still: > > (3) The actions which use bootstrap_nodeid check are not guaranteed to > execute once per service. In case the service is present on multiple > roles, the bootstrap_nodeid check succeeds once per such role. > > Using per-service bootstrap host instead of per-role bootstrap host > sounds like going the right way then. Yeah it seems like both definitions of the bootstrap node described above are wrong in different ways ;) > However, none of the above provides a solid explanation of what's really > happening in the LP/BZ mentioned above. Hopefully this info is at least > a piece of the puzzle. Yup, thanks for working through it - as mentioned above I think the problem is the docker-puppet.py conditional that runs the bootstrap tasks uses the deploy-steps.j2 global bootstrap ID, so it can run on potentially the wrong role. Unless there's other ideas, I think this will be a multi-step fix: 1. Replace all t-h-t references for bootstrap_nodeid with per-service bootstrap node names (I pushed https://review.openstack.org/#/c/605046/ which may make this easier to do in the ansible tasks) 2. Ensure puppet-tripleo does the same 3. Rework docker-puppet.py so it can read all the per-service bootstrap nodes (and filter to only run when appropriate), or perhaps figure out a way to do the filtering in the ansible tasks before running docker-puppet.py 4. Remove all bootstrap_node* references from the tripleo-ansible-inventory code, to enforce using the per-service values Any other ideas or suggestions of things I've missed welcome! :) Steve __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
