On Tue, Sep 25, 2018 at 2:06 PM Jiří Stránský wrote:
>
> Hi Steve,
Thanks for the reply - summary of our follow-up discussion on IRC below:
> On 25/09/2018 10:51, Steven Hardy wrote:
> > Hi all,
> >
> > After some discussions with bandini at the PTG, I've been taking a
> > look at this bug and how to solve it:
> >
> > https://bugs.launchpad.net/tripleo/+bug/1792613
> > (Also more information in downstream bz1626140)
> >
> > The problem is that we always run container bootstrap tasks (as well
> > as a bunch of update/upgrade tasks) on the bootstrap_nodeid, which by
> > default is always the overcloud-controller-0 node (regardless of which
> > services are running there).
> >
> > This breaks a pattern we established a while ago for Composable HA,
> > where we' work out the bootstrap node by
> > $service_short_bootstrap_hostname, which means we always run on the
> > first node that has the service enabled (even if it spans more than
> > one Role).
> >
> > This presents two problems:
> >
> > 1. service_config_settings only puts the per-service hieradata on
> > nodes where a service is enabled, hence data needed for the
> > bootstrapping (e.g keystone users etc) can be missing if, say,
> > keystone is running on some role that's not Controller (this, I think
> > is the root-cause of the bug/bz linked above)
> >
> > 2. Even when we by luck have the data needed to complete the bootstrap
> > tasks, we'll end up pulling service containers to nodes where the
> > service isn't running, potentially wasting both time and space.
> >
> > I've been looking at solutions, and it seems like we either have to
> > generate per-service bootstrap_nodeid's (I have a patch to do this
> > https://review.openstack.org/605010), or perhaps we could just remove
> > all the bootstrap node id's, and switch to using hostnames instead?
> > Seems like that could be simpler, but wanted to check if there's
> > anything I'm missing?
>
> I think we should recheck he initial assumptions, because based on my
> testing:
>
> * the bootstrap_nodeid is in fact a hostname already, despite its
> deceptive name,
>
> * it's not global, it is per-role.
>
> From my env:
>
> [root@overcloud-controller-2 ~]# hiera -c /etc/puppet/hiera.yaml
> bootstrap_nodeid
> overcloud-controller-0
>
> [root@overcloud-novacompute-1 ~]# hiera -c /etc/puppet/hiera.yaml
> bootstrap_nodeid
> overcloud-novacompute-0
>
> This makes me think the problems (1) and (2) as stated above shouldn't
> be happening. The containers or tasks present in service definition
> should be executed on all nodes where a service is present, and if they
> additionally filter for bootstrap_nodeid, it would only pick one node
> per role. So, the service *should* be deployed on the selected bootstrap
> node, which means the service-specific hiera should be present there and
> needless container downloading should not be happening, AFAICT.
Ah, I'd missed that because we have another different definition of
the bootstrap node ID here:
https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps.j2#L283
That one is global, because it only considers primary_role_name, which
I think explains the problem described in the LP/BZ.
> However, thinking about it this way, we probably have a different
> problem still:
>
> (3) The actions which use bootstrap_nodeid check are not guaranteed to
> execute once per service. In case the service is present on multiple
> roles, the bootstrap_nodeid check succeeds once per such role.
>
> Using per-service bootstrap host instead of per-role bootstrap host
> sounds like going the right way then.
Yeah it seems like both definitions of the bootstrap node described
above are wrong in different ways ;)
> However, none of the above provides a solid explanation of what's really
> happening in the LP/BZ mentioned above. Hopefully this info is at least
> a piece of the puzzle.
Yup, thanks for working through it - as mentioned above I think the
problem is the docker-puppet.py conditional that runs the bootstrap
tasks uses the deploy-steps.j2 global bootstrap ID, so it can run on
potentially the wrong role.
Unless there's other ideas, I think this will be a multi-step fix:
1. Replace all t-h-t references for bootstrap_nodeid with per-service
bootstrap node names (I pushed
https://review.openstack.org/#/c/605046/ which may make this easier to
do in the ansible tasks)
2. Ensure puppet-tripleo does the same
3. Rework docker-puppet.py so it can read all the per-service
bootstrap nodes (and filter to only run when appropriate), or perhaps
figure out a way to do the filtering in the ansible tasks before
running docker-puppet.py
4. Remove all bootstrap_node* references from the
tripleo-ansible-inventory code, to enforce using the per-service
values
Any other ideas or suggestions of things I've missed welcome! :)
Steve
__
OpenStack Development Mailing List (not for usage questions)
