[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
To add to Bretts comment in #21, the proposed workaround is only effective if you know about this situation beforehand. I.e. if you enabled the override through vendor data before deploying any Focal or newer instances. Another possible workaround is to preform a "cold migration". When Nova stops/starts an instance the domain XML on the hypervisor is re-created. We can take advantage of this behavior, if an instance is stop/started after a network MTU has been lowered the new domain XML will have the new MTU which makes the libvirt driver enforce the MTU in the instance. The instance configuration will still make systemd-networkd attempt to set the hard coded MTU but it will not be allowed to: ubuntu@u:~$ grep mtu /etc/netplan/50-cloud-init.yaml mtu: 1442 ubuntu@u:~$ zgrep MTU /var/log/syslog.* /var/log/syslog.2.gz:Dec 10 13:54:19 u systemd-networkd[287]: /run/systemd/network/10-netplan-ens2.network: MTUBytes= in [Link] section and UseMTU= in [DHCP] section are set. Disabling UseMTU=. /var/log/syslog.2.gz:Dec 10 13:54:19 u systemd-networkd[287]: ens2: Could not set MTU, ignoring: mtu greater than device maximum. Invalid argument ubuntu@u:~$ ip link 2: ens2: mtu 1441 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:30:f3:82 brd ff:ff:ff:ff:ff:ff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
** Changed in: cloud-init Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
Adding upstream OpenStack Nova to the bug to get their perspective on why the OpenStack datasource is exposing MTU when it knows that the network should be configured using DHCP ref Brett's question in #21. ** Also affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
Hi Alexander, Thanks for reopening. Sorry to hear about the instance loss. We have a short term fix, but I think we need to request a fix in Openstack for network_data.json as well. Currently cloud-init passes a configured MTU value to the renderers (netplan/systemd-networkd/etc), which in turn treat a configured MTU as overriding DHCP MTU options. There are reasons one might want to do this so I don't think we want to try to change this behavior for all of cloud-init. You can force cloud-init to configure the network on every boot[1], the downside is that this will increase subsequent boot times. To semantically match other datasources, Openstack would only expose MTU settings if it intended for the MTU to override DHCP's MTU option. This would mean that the mtu would only get configured if a link's network type was not dhcp (change here[2] I think). Could you open a ticket with openstack for this? If that would break other use cases, this behavior could possibly be worked around in the cloud-init openstack datasource by ignoring the MTU metadata field, but I would want to see if Openstack can fix this first resorting to that. [1] Override: # apply network config on every boot updates: network: when: ['boot'] [2] https://github.com/openstack/nova/blob/261de76104ca67bed3ea6cdbcaaab0e44030f1e2/nova/virt/netutils.py#L266 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
I'm going to re-open this bug after working through Openstack networking migration for OVS to OVN. I ran into an issue where the MTU set in netplan caused all my instances to be lost after the migration. While this can, and should, be documented, I also think that reducing places where instances could be lost should be taken whenever possible. ** Changed in: cloud-init Status: Expired => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
[Expired for netplan.io (Ubuntu) because there has been no activity for 60 days.] ** Changed in: netplan.io (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
[Expired for cloud-init because there has been no activity for 60 days.] ** Changed in: cloud-init Status: Incomplete => Expired -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
On Thu, Oct 15, 2020 at 10:45 AM Frode Nordahl <1899...@bugs.launchpad.net> wrote: > Ryan, thanks for those pointers, will check. I also see in #15 that > Bionic uses Fallback while Focal uses an actual ds, don't know why > though. > Bah, I *keep* forgetting, that Bionic does *NOT* read OpenStack metadata service by default https://github.com/canonical/cloud- init/commit/cd1de5f47ab6b82f2c6fd61a5f6681f33b3e5705 That landed right *after* 18.04 was released. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
Ryan, thanks for those pointers, will check. I also see in #15 that Bionic uses Fallback while Focal uses an actual ds, don't know why though. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
Adding some excerpts from cloud-init logs differences between Focal and
Bionic, appears to be quite a bit difference in how config is handled.
I also went back and compared if there were other differences like image
properties or similar stuff that could find their way into qemu config,
but found none.
If you say cloud-init should be equal on Bionic and Focal, where does
this information come from apart from the cloud metadata (which also is
equal for both instances)?
Focal:
# grep fa:16:3e:d6:d0:91 /var/log/cloud-init.log
2020-10-15 14:48:15,982 - stages.py[DEBUG]: applying net config names for
{'version': 1, 'config': [{'type': 'physical', 'mtu': 8942, 'subnets':
[{'type': 'dhcp4'}], 'mac_address': 'fa:16:3e:d6:d0:91', 'name': 'ens2'}]}
2020-10-15 14:48:15,991 - __init__.py[DEBUG]: no work necessary for renaming of
[['fa:16:3e:d6:d0:91', 'ens2', 'virtio_net', '0x0001']]
2020-10-15 14:48:15,991 - stages.py[INFO]: Applying network configuration from
ds bringup=False: {'version': 1, 'config': [{'type': 'physical', 'mtu': 8942,
'subnets': [{'type': 'dhcp4'}], 'mac_address': 'fa:16:3e:d6:d0:91', 'name':
'ens2'}]}
2020-10-15 14:48:18,268 - stages.py[DEBUG]: applying net config names for
{'version': 1, 'config': [{'type': 'physical', 'mtu': 8942, 'subnets':
[{'type': 'dhcp4'}], 'mac_address': 'fa:16:3e:d6:d0:91', 'name': 'ens2'}]}
2020-10-15 14:48:18,277 - __init__.py[DEBUG]: no work necessary for renaming of
[['fa:16:3e:d6:d0:91', 'ens2', 'virtio_net', '0x0001']]
Bionic:
$ grep fa:16:3e:f7:fc:c4 /var/log/cloud-init.log
2020-10-14 19:08:10,670 - stages.py[DEBUG]: applying net config names for
{'ethernets': {'ens2': {'dhcp4': True, 'set-name': 'ens2', 'match':
{'macaddress': 'fa:16:3e:f7:fc:c4'}}}, 'version': 2}
2020-10-14 19:08:10,679 - __init__.py[DEBUG]: no work necessary for renaming of
[['fa:16:3e:f7:fc:c4', 'ens2', 'virtio_net', '0x0001']]
2020-10-14 19:08:10,680 - stages.py[INFO]: Applying network configuration from
fallback bringup=False: {'ethernets': {'ens2': {'dhcp4': True, 'set-name':
'ens2', 'match': {'macaddress': 'fa:16:3e:f7:fc:c4'}}}, 'version': 2}
{'type': 'physical', 'name': 'ens2', 'mac_address': 'fa:16:3e:f7:fc:c4',
'match': {'macaddress': 'fa:16:3e:f7:fc:c4'}, 'subnets': [{'type': 'dhcp4'}]}
{'ens2': {'dhcp4': True, 'set-name': 'ens2', 'match': {'macaddress':
'fa:16:3e:f7:fc:c4'}}}
2020-10-14 19:08:12,659 - stages.py[DEBUG]: applying net config names for
{'ethernets': {'ens2': {'dhcp4': True, 'set-name': 'ens2', 'match':
{'macaddress': 'fa:16:3e:f7:fc:c4'}}}, 'version': 2}
2020-10-14 19:08:12,666 - __init__.py[DEBUG]: no work necessary for renaming of
[['fa:16:3e:f7:fc:c4', 'ens2', 'virtio_net', '0x0001']]
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1899487
Title:
cloud-init hard codes MTU configuration at initial deploy time
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
> I guess it's time for me to ask a question: is it cloud-init that
> renders /etc/netplan/50-cloud-init.yaml? If so where does netplan
> fit in when the difference is how that file is rendered and not
> how it is interpreted. As you can see in #10 the mtu statement is
> not in the file on bionic, while it is on focal.
>
> Since versions appear to be the same my guess would be that there
> is some internal modelling of how bionic vs. focal should be
> configured?
There isn't an internal model; cloud-init SRU's master back to
previous releases. For OpenStack, cloud-init on Xenial does not
render network-data.json by default as that's a behavior change
added in Bionic and newer.
The pipeline looks like:
cloudinit (fetch network-data.json from OpenStack)
`-> cloudinit (convert network-data.json to network-config-v1)
`-> cloudinit (converts network-config-v1 -> netplan on Ubuntu)
`-> cloudinit (calls netplan generate -> systemd-networkd files)
> I can't let go of why the same version of cloud-init renders
> different config with the same data source.
Are you sure the 'mtu' value it was present in the network-data.json
at the time cloud-init fetched it vs. when you curl later?
Ceck the cloud-init.log file; you should see the network
config after it was converted from network-data.json somewhere
in the log.
Give your JSON from [10], bionic and focal render this the same
## BIONIC ##
% lxc launch ubuntu-daily:bionic b1
% lxc exec b1 bash
root@b1:~# lsb_release -rd
Description:Ubuntu 18.04.5 LTS
Release:18.04
root@b1:~# dpkg --list | egrep "(cloud-init|netplan)"
ii cloud-init 20.3-2-g371b392c-0ubuntu1~18.04.1 all
Init scripts for cloud instances
ii cloud-initramfs-copymods 0.40ubuntu1.1 all
copy initramfs modules into root filesystem for later use
ii cloud-initramfs-dyn-netconf0.40ubuntu1.1 all
write a network interface file in /run for BOOTIF
ii libnetplan0:amd64 0.99-0ubuntu3~18.04.3 amd64
YAML network configuration abstraction runtime library
ii netplan.io 0.99-0ubuntu3~18.04.3 amd64
YAML network configuration abstraction for various backends
root@b1:~# cat /etc/cloud/build.info
build_name: server
serial: 20201014
root@b1:~# cat network-data.json
{"links": [{"id": "tapc352887e-0f", "vif_id":
"c352887e-0fff-481b-af47-7df9f7c2ff05", "type": "ovs", "mtu": 8950,
"ethernet_mac_address": "fa:16:3e:a3:34:78"}], "networks": [{"id": "network0",
"type": "ipv4_dhcp", "link": "tapc352887e-0f", "network_id":
"f8123ceb-e29d-4f4a-b200-6fb3bf3984ba"}], "services": []}
root@b1:~# cloud-init devel net-convert --network-data network-data.json -k
network_data.json -m "ens4,fa:16:3e:a3:34:78" -d test -D ubuntu --debug -O
netplan
2020-10-15 15:09:17,796 - util.py[DEBUG]: Reading from /proc/uptime
(quiet=False)
2020-10-15 15:09:17,797 - util.py[DEBUG]: Read 14 bytes from /proc/uptime
2020-10-15 15:09:17,797 - util.py[DEBUG]: Reading from
/sys/class/net/eth0/addr_assign_type (quiet=False)
2020-10-15 15:09:17,797 - util.py[DEBUG]: Read 2 bytes from
/sys/class/net/eth0/addr_assign_type
2020-10-15 15:09:17,797 - util.py[DEBUG]: Reading from
/sys/class/net/eth0/uevent (quiet=False)
2020-10-15 15:09:17,797 - util.py[DEBUG]: Read 26 bytes from
/sys/class/net/eth0/uevent
2020-10-15 15:09:17,797 - util.py[DEBUG]: Reading from
/sys/class/net/eth0/address (quiet=False)
2020-10-15 15:09:17,797 - util.py[DEBUG]: Read 18 bytes from
/sys/class/net/eth0/address
2020-10-15 15:09:17,798 - util.py[DEBUG]: Reading from
/sys/class/net/eth0/device/device (quiet=False)
2020-10-15 15:09:17,798 - util.py[DEBUG]: Reading from
/sys/class/net/lo/addr_assign_type (quiet=False)
2020-10-15 15:09:17,798 - util.py[DEBUG]: Read 2 bytes from
/sys/class/net/lo/addr_assign_type
2020-10-15 15:09:17,798 - util.py[DEBUG]: Reading from /sys/class/net/lo/uevent
(quiet=False)
2020-10-15 15:09:17,798 - util.py[DEBUG]: Read 23 bytes from
/sys/class/net/lo/uevent
2020-10-15 15:09:17,798 - util.py[DEBUG]: Reading from
/sys/class/net/lo/address (quiet=False)
2020-10-15 15:09:17,798 - util.py[DEBUG]: Read 18 bytes from
/sys/class/net/lo/address
2020-10-15 15:09:17,798 - util.py[DEBUG]: Reading from
/sys/class/net/lo/device/device (quiet=False)
2020-10-15 15:09:17,798 - util.py[DEBUG]: Reading from /sys/class/net/eth0/type
(quiet=False)
2020-10-15 15:09:17,798 - util.py[DEBUG]: Read 2 bytes from
/sys/class/net/eth0/type
2020-10-15 15:09:17,798 - util.py[DEBUG]: Reading from /sys/class/net/lo/type
(quiet=False)
2020-10-15 15:09:17,798 - util.py[DEBUG]: Read 4 bytes from
/sys/class/net/lo/type
Internal State
--- !!python/object:cloudinit.net.network_state.NetworkState
_has_default_route: null
_network_state:
config:
- mac_address: fa:16:3e:a3:34:78
mtu: 8950
name: ens4
subnets:
- type: dhcp4
type:
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
I can't let go of why the same version of cloud-init renders different config with the same data source. So a question: are there other data sources in use that we have not yet examined? Information from the hypervisor configuration trickling through the virtio drivers or something like that? I will compare the qemu configuration for the to instances and see if there are any differences. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
> Hmm, however it should be the same version of cloud-init in either Bionic or Focal tests. Is this a netplan change? In the bionic image I have: $ dpkg -l | egrep "(cloud-init|netplan)" ii cloud-init 20.3-2-g371b392c-0ubuntu1~18.04.1 all Init scripts for cloud instances ii cloud-initramfs-copymods 0.40ubuntu1.1 all copy initramfs modules into root filesystem for later use ii cloud-initramfs-dyn-netconf 0.40ubuntu1.1 all write a network interface file in /run for BOOTIF ii libnetplan0:amd640.99-0ubuntu3~18.04.3 amd64YAML network configuration abstraction runtime library ii netplan.io 0.99-0ubuntu3~18.04.3 amd64YAML network configuration abstraction for various backends In the focal image I have: $ dpkg -l | egrep "(cloud-init|netplan)" ii cloud-init 20.3-2-g371b392c-0ubuntu1~20.04.1 all initialization and customization tool for cloud instances ii cloud-initramfs-copymods 0.45ubuntu1 all copy initramfs modules into root filesystem for later use ii cloud-initramfs-dyn-netconf0.45ubuntu1 all write a network interface file in /run for BOOTIF ii libnetplan0:amd64 0.99-0ubuntu3~20.04.2 amd64 YAML network configuration abstraction runtime library ii netplan.io 0.99-0ubuntu3~20.04.2 amd64 YAML network configuration abstraction for various backends I guess it's time for me to ask a question: is it cloud-init that renders /etc/netplan/50-cloud-init.yaml? If so where does netplan fit in when the difference is how that file is rendered and not how it is interpreted. As you can see in #10 the mtu statement is not in the file on bionic, while it is on focal. Since versions appear to be the same my guess would be that there is some internal modelling of how bionic vs. focal should be configured? > The controls in the OpenStack DHCP service are purely a "on/off" switch (`advertise_mtu`) about advertising MTU in the DHCP network details and not the control for what that setting is? The reason I want to make sure is because I'm always leery when there are multiple sources of possible truth that have to be sorted and if a user can bite themselves by changing an MTU value in one place but also another and which do you listen to/respect. Previously you had the `advertise_mtu` "on/off" switch which was removed at OpenStack Ocata (leaving it permanently "on"). In addition to that the operator of the cloud can inject DHCP configration into the DHCP servers. The end user consuming the cloud also have control over their virtual network MTU's through the OpenStack Neutron API. > The actual value for the MTU is a Neutron setting and in theory, should be the same then from DHCP network data or by the provided network_data.json information? The value for the MTU is a per virtual network setting which is exposed to the end user of the cloud. And yes, the setting set on the virtual network should be the one exposed in the network_data.json. But remember that the operator of the cloud has power to inject options directly into the DHCP server which could mean the DHCP server could advertise a different MTU than the user has chosen for their network. If the operator of the cloud has chosen to do so it is most likely for a very good operational reason. If the end user or operator intends to configure instances with DHCP, DHCP should be authoritative source of truth. > The final nail in this coffin is that the setting cloud-init is setting overrides the value for MTU that comes in via DHCP. > Do I have that right? If so, a couple of questions then. Yes. > 1. It seems it would be worthwhile to see if there was some method for a refreshing cloud-init's details on the network. Ideally here, changing the network data in Neutron would be able to trigger an update on instances, though that's a tricky can of worms that could lead to broken networking on existing hosts if it goes wrong. It smells a bit like the work we want to completed next cycle around allowing hotplug of a new nic/device and be able to help drive networking config for it...just without the whole new device thing heh. This does indeed sound interesting, with regards to a cloud operator possibly not having any access or control over the instances end users run on their cloud having levers to control network configuration in such instances for maintenance/migration purposes in some manner would be valuable. The alternative is forklift and endless nagging of end users to do manual intervention and the support load that comes afterwards when everything breaks because they did not pay attention to the operators requests in time. > 2. Shou
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
In our functional test we use Bionic images, and sure enough the netplan.yaml [9] written there does _NOT_ include the MTU! The OpenStack Metadata source remains the same [10]. Hmm, however it should be the same version of cloud-init in either Bionic or Focal tests. Is this a netplan change? That's one side of the coin. In working to understand the "best path forward" I just want to make sure I'm following. The controls in the OpenStack DHCP service are purely a "on/off" switch (`advertise_mtu`) about advertising MTU in the DHCP network details and not the control for what that setting is? The reason I want to make sure is because I'm always leery when there are multiple sources of possible truth that have to be sorted and if a user can bite themselves by changing an MTU value in one place but also another and which do you listen to/respect. The actual value for the MTU is a Neutron setting and in theory, should be the same then from DHCP network data or by the provided network_data.json information? In this case the pain point is that existing instances won't process the DHCP change of MTU properly because cloud-init has written out the netplan.yaml and even though DHCP comes in with a new setting cloud-init isn't triggered in any fashion to update its understanding of the world and write out a new compatible netplan. The final nail in this coffin is that the setting cloud-init is setting overrides the value for MTU that comes in via DHCP. Do I have that right? If so, a couple of questions then. 1. It seems it would be worthwhile to see if there was some method for a refreshing cloud-init's details on the network. Ideally here, changing the network data in Neutron would be able to trigger an update on instances, though that's a tricky can of worms that could lead to broken networking on existing hosts if it goes wrong. It smells a bit like the work we want to completed next cycle around allowing hotplug of a new nic/device and be able to help drive networking config for it...just without the whole new device thing heh. 2. Should the setting that cloud-init writes be an overwriting value in netplan? Could this be a bug that netplan is not allowing the dhcp details to be respected over what cloud-init started with? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
Comment #8 made me think about why this works in our functional test, so
I went to investigate that. In our functional test we use Bionic images,
and sure enough the netplan.yaml [9] written there does _NOT_ include
the MTU! The OpenStack Metadata source remains the same [10].
9: ubuntu@banana-1:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 18.04.5 LTS
Release:18.04
Codename: bionic
ubuntu@banana-1:~$ cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
ethernets:
ens2:
dhcp4: true
match:
macaddress: fa:16:3e:a3:34:78
set-name: ens2
version: 2
10: $ curl http://169.254.169.254/openstack/2018-08-27/network_data.json
{"links": [{"id": "tapc352887e-0f", "vif_id":
"c352887e-0fff-481b-af47-7df9f7c2ff05", "type": "ovs", "mtu": 8950,
"ethernet_mac_address": "fa:16:3e:a3:34:78"}], "networks": [{"id": "network0",
"type": "ipv4_dhcp", "link": "tapc352887e-0f", "network_id":
"f8123ceb-e29d-4f4a-b200-6fb3bf3984ba"}], "services": []}
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1899487
Title:
cloud-init hard codes MTU configuration at initial deploy time
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
The CI artifact referenced in [6] in the previous comment was removed, the source can be viewed here: https://review.opendev.org/#/c/715132/4 /deploy-guide/source/app-ovn.rst -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
Richard, The "MTU controls in OpenStack have been removed" part pertains to the removal of the operator facing configuration option for the OpenStack DHCP server as to whether or not it should provide information about the network MTU in it's response to clients. Where the removal here means that it is permanently ON, as in you could expect a OpenStack cloud to always provide information about MTU in DHCP packets from the time it was removed. This was introduced in OpenStack Ocata back in 2017 [4]. Prior to this the default value of `advertise_mtu` was true. I guess the reason for mentioning this was to paint a picture of what to expect from OpenStack clouds out there in the wild wrt. whether or not the DHCP server provides information about MTU or not. The act of reducing the MTU on networks is done through the OpenStack Neutron API and/or through migration tools [5]. The effect of reducing the MTU of a network construct in OpenStack result in reducing the MTU for the involved router interfaces as well as the associated DHCP server configuration. There also exist levers to inject configuration into the OpenStack DHCP server to prepare for a migration which is what we use in our recommended migration path [6], you can view the functional test code here [7]. The functional test code does not include a step for reducing MTU and waiting X hours for a DHCP renewal which is why this issue was missed, instead it injects the reduced MTU config prior to launching a first test instance [8]. 4: https://github.com/openstack/neutron/commit/832240aaa83d557d87307656cb0712dcbf7aa2c3 5: https://docs.openstack.org/neutron/latest/ovn/migration.html 6: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b37/715132/4/check/build-openstack-deploy-guide/b37f49a/docs/app-ovn.html#migration-from-neutron-ml2-ovs-to-ml2-ovn 7: https://github.com/openstack-charmers/zaza-openstack-tests/blob/d4deb0478a0540cc61c39bae80e35e4335571554/zaza/openstack/charm_tests/ovn/tests.py#L143-L412 8: https://github.com/openstack/charm-neutron-openvswitch/blob/bc97a66b87d33e43ab5b842b000b49a75b1c5797/tests/tests.yaml#L39 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
Frode, can you explain to me the OpenStack operator path here. I'm not familiar with how these adjustments are practically made. You mention "a operator to reduce the MTU available to instances" and "To maximize performance these clouds have configured their instances to use the maximum available MTU without leaving any headroom" but then also say that the MTU controls in OpenStack have been removed? I'd like to understand where the knobs a cloud operator have available to them and then look at how to identify the "source of truth". So far I understand one is coming from the cloud itself, but I'm not sure how, another potential source of truth is a DHCP value provided to the instance. I assume that DHCP knob is in the DHCP server config and not done through a more centralized OpenStack knob that assures common behavior among DHCP and non-DHCP instances? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
> On the flip side the presence of the MTU key in the OpenStack > metadata cannot be used as an indicator for intent from either the > system or the user that the DHCP server should not be providing the > MTU either. > > Looking at the commit that changed the behaviour in OpenStack the > intent of the original code was to always provide the MTU value in > the metadata regardless of network type, the fact that it showed as > `null` was a bug. > > Up until 2017 the default for the OpenStack controlled DHCP server > was to always provide an MTU. In 2017 the ability to control this > behaviour was removed and from that point onward it always provides > an MTU. > > The user has no way of influencing the contents of the OpenStack > network metadata, apart from downgrading to a 5 year old version. > Before we continue suggesting that cloud-init should somehow guess what OpenStack meant to do with network configuration it provided to cloud-init I'd like to make sure we discuss what we're suggesting. If we can guess that an OpenStack which sent an MTU really didn't mean for cloud-init to use this MTU because the DHCP server might also send an MTU, can we not guess that the IP address it sent us wasn't the correct one and instead we should add .1 to the lowest octet? I'm being pendantic here to make a point. OpenStack is the "Oracle" here; just like MAAS, or Ec2 or Azure. The network configuration it provides to the guest is meant to be taken as it is provided. If the configuration is sub-optimal should not the cloud itself resolve this? Has there been any attempt to ask in OpenStack upstream why the MTU plumbing/control was removed? If a specific MTU is needed for a network in OpenStack, should that not be configured in OpenStack such that the MTU value is either statically provided in the network-data.json sent *or* update the DHCP server running on that network to provide the correct value? > I don't see an easy way of overriding cloud-inits default behaviour > by adding additional configuration through vendor data either. This is correct. Network-config cannot be part of user-data or vendor-data; the network needs to be configured prior to cloud-init fetching/reading this data (which may be URLs to remote cloud-config). > > Perhaps adding a cloud-init config stanza for how the OpenStack > source driver should interpret the presence of MTU in the network > metadata could be a path to retain compability with anyone relying > on the current behaviour and at the same time providing a way > forward for everyone else? > > Meanwhile instances are configured to obtain an address dynamically > but stuck with a static value for MTU forever, and not being able to > adjust to changes being made to the environment without manual > intervention to individual instances. If the network-data.json provided by the metadata service is not sufficient and you cannot change this service one can provide a network configuration in a ConfigDrive. ** Changed in: cloud-init Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
On the flip side the presence of the MTU key in the OpenStack metadata cannot be used as an indicator for intent from either the system or the user that the DHCP server should not be providing the MTU either. Looking at the commit that changed the behaviour in OpenStack the intent of the original code was to always provide the MTU value in the metadata regardless of network type, the fact that it showed as `null` was a bug. Up until 2017 the default for the OpenStack controlled DHCP server was to always provide an MTU. In 2017 the ability to control this behaviour was removed and from that point onward it always provides an MTU. The user has no way of influencing the contents of the OpenStack network metadata, apart from downgrading to a 5 year old version. I don't see an easy way of overriding cloud-inits default behaviour by adding additional configuration through vendor data either. Perhaps adding a cloud-init config stanza for how the OpenStack source driver should interpret the presence of MTU in the network metadata could be a path to retain compability with anyone relying on the current behaviour and at the same time providing a way forward for everyone else? Meanwhile instances are configured to obtain an address dynamically but stuck with a static value for MTU forever, and not being able to adjust to changes being made to the environment without manual intervention to individual instances. ** Changed in: cloud-init Status: Incomplete => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
> So I would suggest that whenever OpenStack eludes to dynamic configuration > being in play cloud-init should not write the MTU value into the on-disk > configuration but let it be configured by dynamic network configuration > protocol. > > What do you think? I would argue the opposite. The existing behavior is that the MTU provided by the network-data.json is the source of truth. Cloud-init itself cannot determine whether the DHCP service provides MTU or not, nor whether the MTU provided in the DHCP response is infact the desired MTU. Further, for cloud-init now to start ignoring MTU if DHCP is present would break existing users who's DHCP server either does not provide MTU or provides an incorrect value. If network-data.json MTU value is null, then I think it all of this works the way you want, correct? Looking at the bug which fixed this always null value should it not be enhanced to only set this MTU value if the DHCP service on the network is not providing an MTU? ** Changed in: cloud-init Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
That is an excellent question, I see that the example provided in the Nova documentation [0] provides `null` for the MTU. There are also a Nova bug 1746323 on the lack of actual API documentation for the OpenStack format metadata, so I guess they could expect no less than diverging implementations consuming it in the wild. However, I also see that the reporting of `null` MTU was fixed [1] on the back of Nova bug 1576713 a few years back so that it now provides an actual MTU regardless of network addressing type. The OpenStack format metadata does provide a separate field that distinguishes between the various types of dynamic and static configuration [2] and I see that cloud-init already makes use of it [3]. So I would suggest that whenever OpenStack eludes to dynamic configuration being in play cloud-init should not write the MTU value into the on-disk configuration but let it be configured by dynamic network configuration protocol. What do you think? 0: https://docs.openstack.org/nova/latest/user/metadata.html#openstack-format-metadata 1: https://review.opendev.org/#/c/316395/ 2: https://github.com/openstack/nova/blob/261de76104ca67bed3ea6cdbcaaab0e44030f1e2/nova/virt/netutils.py#L282-L309 3: https://github.com/canonical/cloud-init/blob/07104504ab5b30efd2d1f7a8c36effe18b8e5fe0/cloudinit/sources/helpers/openstack.py#L598-L609 ** Changed in: cloud-init Status: Incomplete => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899487 Title: cloud-init hard codes MTU configuration at initial deploy time To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899487] Re: cloud-init hard codes MTU configuration at initial deploy time
" # curl http://169.254.169.254/openstack/2018-08-27/network_data.json
{"links": [{"id": "tapa035fb68-01", "vif_id":
"a035fb68-010c-42e3-8da7-ea3c36a0d607", "type": "ovs", "mtu": 8942,
"ethernet_mac_address": "fa:16:3e:31:26:f7"}], "networks": [{"id": "network0",
"type": "ipv4_dhcp", "link": "tapa035fb68-01", "network_id":
"b4ef84c0-1235-48a8-aaf7-03fab7ef5367"}], "services": []}"
How is cloud-init to know from this network-config.json that DHCP will
provide an MTU value? How does it know that it should ignore the
provided MTU? If DHCP is providing MTU, should network-config.json then
not provide the MTU value?
** Also affects: netplan.io (Ubuntu)
Importance: Undecided
Status: New
** Changed in: cloud-init
Status: New => Incomplete
** Changed in: netplan.io (Ubuntu)
Status: New => Incomplete
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1899487
Title:
cloud-init hard codes MTU configuration at initial deploy time
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1899487/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
