Re: [OpenStack-Infra] Hostnames

[James E. Blair]

Clark Boylan  writes:

> On Sun, Jan 7, 2018, at 2:30 PM, David Moreau Simard wrote:
>> When I compared ze10 with ze09 today, I noticed that ze09's "hostname"
>> command returned "ze09" while ze10 had "ze10.openstack.org".
>> 
>> However, both nodes had the full fqdn when doing "hostname -f".
>> 
>> I didn't dig deeper since we're the weekend and all that but there might be
>> a clue in my experience above.
>
> I think the reason for this is that ze09 was rebuilt so the launch
> node scripts modified it setting hostname to only ze09 and not
> ze09.openstack.org. ze10 on the other hand was simply rebooted so its
> old hostname, ze10.openstack.org, stuck.

Distilling this conversation and that in IRC today:

The current software should produce consistent results:
  hostname -> ze09
  hostname --fqdn -> ze09.openstack.org

This is what we want on all machines.

Machines launched before October 2017 were subject to a race with
cloud-init which has since been corrected.  Those may have the FQDN for
the hostname.  That explains the discrepancy observed.

The next time we stop all of Zuul, should we rename all the hosts and
then update the grafana dashboards?

-Jim

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Hostnames

[Jeremy Stanley]

On 2018-01-06 14:16:35 -0500 (-0500), Paul Belanger wrote:
[...]
> I know we also taked about building out own DIBs for control plane
> servers, which would move us to glean by default. In the past we
> discussed using nodepool to build the images, but didn't want to
> add passwords for rax into nodepool.o.o. That would mean a 2nd
> instance of nodepool, do people think that would work? Or maybe
> some sort of periodic job and store credentials in zuul secrets?

In the past we've considered the fact that none of our automation
has access to our control plane provider account credentials to be a
feature. There is a bit of additional risk, for example with giving
Zuul jobs access to those, where a failure in security design for
job secret handling could allow a malicious party to take control of
Zuul itself (and far more for that matter).
-- 
Jeremy Stanley


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Hostnames

[David Moreau Simard]

When I compared ze10 with ze09 today, I noticed that ze09's "hostname"
command returned "ze09" while ze10 had "ze10.openstack.org".

However, both nodes had the full fqdn when doing "hostname -f".

I didn't dig deeper since we're the weekend and all that but there might be
a clue in my experience above.

David Moreau Simard
Senior Software Engineer | Openstack RDO

dmsimard = [irc, github, twitter]

On Jan 6, 2018 1:04 PM, "James E. Blair"  wrote:

> Hi,
>
> It seems that every time we boot a new server, it either randomly has a
> hostname of foo, or foo.openstack.org.  And maybe that changes between
> the first boot and second.
>
> The result of this is that our services which require that they know
> their hostname (which is a lot, especially the complicated ones) end up
> randomly working or not.  We waste time repeating the same diagnosis and
> manual fix each time.
>
> What is the cause of this, and how do we fix this correctly?
>
> -Jim
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Hostnames

[Paul Belanger]

On Sat, Jan 06, 2018 at 10:21:12AM -0800, Clark Boylan wrote:
> On Sat, Jan 6, 2018, at 10:03 AM, James E. Blair wrote:
> > Hi,
> > 
> > It seems that every time we boot a new server, it either randomly has a
> > hostname of foo, or foo.openstack.org.  And maybe that changes between
> > the first boot and second.
> > 
> > The result of this is that our services which require that they know
> > their hostname (which is a lot, especially the complicated ones) end up
> > randomly working or not.  We waste time repeating the same diagnosis and
> > manual fix each time.
> > 
> > What is the cause of this, and how do we fix this correctly?
> 
> It seems to be an intentional behavior [0] of part of the launch node build 
> process [1]. We could remove the split entirely there and in the hosts and 
> mailnametemplate to use fqdns as hostname to fix it.
> 
> [0] 
> https://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/roles/set_hostname/tasks/main.yml#n12
> [1] 
> https://git.openstack.org/cgit/openstack-infra/system-config/tree/launch/launch-node.py#n209
> 
> Clark
> 
We also talked about removing cloud-init, which has been known to modify our
hostnames on reboot.  When I last looked (few months ago) that was the reason
for renames, unsure this time.

I know we also taked about building out own DIBs for control plane servers,
which would move us to glean by default. In the past we discussed using nodepool
to build the images, but didn't want to add passwords for rax into nodepool.o.o.
That would mean a 2nd instance of nodepool, do people think that would work? Or
maybe some sort of periodic job and store credentials in zuul secrets?

PB

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Hostnames

[Clark Boylan]

On Sat, Jan 6, 2018, at 10:03 AM, James E. Blair wrote:
> Hi,
> 
> It seems that every time we boot a new server, it either randomly has a
> hostname of foo, or foo.openstack.org.  And maybe that changes between
> the first boot and second.
> 
> The result of this is that our services which require that they know
> their hostname (which is a lot, especially the complicated ones) end up
> randomly working or not.  We waste time repeating the same diagnosis and
> manual fix each time.
> 
> What is the cause of this, and how do we fix this correctly?

It seems to be an intentional behavior [0] of part of the launch node build 
process [1]. We could remove the split entirely there and in the hosts and 
mailnametemplate to use fqdns as hostname to fix it.

[0] 
https://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/roles/set_hostname/tasks/main.yml#n12
[1] 
https://git.openstack.org/cgit/openstack-infra/system-config/tree/launch/launch-node.py#n209

Clark

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

[OpenStack-Infra] Hostnames

[James E. Blair]

Hi,

It seems that every time we boot a new server, it either randomly has a
hostname of foo, or foo.openstack.org.  And maybe that changes between
the first boot and second.

The result of this is that our services which require that they know
their hostname (which is a lot, especially the complicated ones) end up
randomly working or not.  We waste time repeating the same diagnosis and
manual fix each time.

What is the cause of this, and how do we fix this correctly?

-Jim

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra