Re: generic (cloud) image problems
On 2019-12-30 12:47:21 +0100 (+0100), Thomas Goirand wrote: [...] > Thinking that there's only VMs is a dangerous assumption. There's also > bare-metal with Ironic. In such environment, it wouldn't be surprising > to have VLAN and bonding, then the user may need bridging. [...] There are plenty of legitimate reasons to trunk tagged VLAN traffic directly into a virtual machine too (for example, a wide variety of NFV use cases). -- Jeremy Stanley signature.asc Description: PGP signature
Re: generic (cloud) image problems
On Tue, 31. Dez. 2019 at 10:36 Uhr Thomas Goirand wrote: > > I disagree with that. As mentioned several times, this is not about > > constructing the /etc/network/interfaces manually but cloud-init (as > > provided by the cloud image!) behavior, writing the network > > configuration based on supplied configuration data in a generic > > format. > > +1 > > You don't need to re-explain, IMO, proposing a patch (merge request) to > fix the problem would work better, at this point. Merge request #168 has been raised in salsa. For the other problem (50-cloud-init.cfg not included by /etc/network/interfaces), there is already a bug (#931173) which I commented with a suggestion for a patch (backport fix from upstream). Thanks, Christian
Re: generic (cloud) image problems
On 12/30/19 3:53 AM, Noah Meyerhans wrote: > Unfortunately, AFAICT, ifupdown is also pretty much dead. It hasn't > seen any major feature development in quite some time, and hasn't seen > an upload to unstable in nearly a year. There's at least ifupdown2 from Cumulus (also in Debian) that receive a very active development. Also, if ifupdown isn't receiving so much development, it's also because it's just working fine. Do you see any missing feature? > I suspect that the Right Way to do this on a modern Debian system would > be to use systemd-networkd and systemd netdev configuration. I don't > know enough about this right now to provide an example, or to figure out > if/how this is usable with our current interface configuration > management approach, but systemd.netdev(5) provides some basic examples. > > noah It's fine to talk about the future, and I could somehow agree that this can be one of the paths. But let's fix things for now. Also, I don't think there's any OpenVSwitch integration in systemd, is there? I know that OVS works very well with ifupdown (it took so long to me to fix all the issues in the OVS package that I know it works now...). Thomas Goirand (zigo)
Re: generic (cloud) image problems
On 12/30/19 3:26 PM, Christian Tramnitz wrote: > On Mo., 30. Dez. 2019 at 03:53 Uhr Noah Meyerhans wrote: > >> I agree that current support is not perfect, but I don't think there's >> any reason for our users to feel blocked by this lack of support. >> 100% of expected functionality is available. > > I disagree with that. As mentioned several times, this is not about > constructing the /etc/network/interfaces manually but cloud-init (as > provided by the cloud image!) behavior, writing the network > configuration based on supplied configuration data in a generic > format. +1 You don't need to re-explain, IMO, proposing a patch (merge request) to fix the problem would work better, at this point. > ** For Ubuntu a netplan renderer has to be used which requires > a) a v2 config file which cloud-init seems to choke on > b) some modifications to /etc/cloud/cloud.cfg to use a different > network renderer > I'm not sure if the netplan renderer is compatible with Debian's > systemd-networkd Last time I used it, I hated netplan. It's just another indirection that generates things from a yaml instead. It's also very touchy, and quickly fails to work if it doesn't like one of the elements of the yaml file. Definitively, netplan isn't the way to go. Cheers, Thomas Goirand (zigo)
Re: generic (cloud) image problems
On Mo., 30. Dez. 2019 at 03:53 Uhr Noah Meyerhans wrote: > I agree that current support is not perfect, but I don't think there's > any reason for our users to feel blocked by this lack of support. > 100% of expected functionality is available. I disagree with that. As mentioned several times, this is not about constructing the /etc/network/interfaces manually but cloud-init (as provided by the cloud image!) behavior, writing the network configuration based on supplied configuration data in a generic format. But the resulting configuration can (in some cases that are not uncommon*) not be applied, because the presence of those tools is assumed. There is no workaround available or at least it hasn't been mentioned (using pre/post command may work when editing the file manually, but this is not what cloud-init does). I already stated what happens when supplying a v1 or v2 configuration file to cloud-init 18.3. I'm not saying that adding bridge-utils and vlan is the *only* option to solve this problem. I would agree that it would be a better mid- to long-term option to add a renderer to cloud-init that produces a network configuration that can be consumed by systemd-networkd. But afaik** this option currently doesn't exist and even if someone added it upstream soon it would take a while until it trickles down (current cloud-init is at 19.4, the version in the Buster cloud-image is 18.3). So I would like to ask for *some* workaround before it can be done the "right way", even if that means installing bridge-utils and vlan packages in the cloud-image. *You can argue that the combination of bonding, vlan and bridging may be uncommon *within* a VM. However, it already fails if *one* of the components is being used. I'm sure you would agree that using *either* bonding, vlan or bridging is not really uncommon. Especially in the container world creating bridges or vlans for container networks is standard practice. Especially for the generic images (that are not running on the common cloud providers like AWS, GCP and Azure) VLANs and bridges are a typical interconnect between bare-metal or VPS systems. ** For Ubuntu a netplan renderer has to be used which requires a) a v2 config file which cloud-init seems to choke on b) some modifications to /etc/cloud/cloud.cfg to use a different network renderer I'm not sure if the netplan renderer is compatible with Debian's systemd-networkd Thanks, Christian
Re: generic (cloud) image problems
On 12/30/19 1:53 AM, Ross Vandegrift wrote: > On Sun, Dec 29, 2019 at 11:16:20PM +0100, Thomas Goirand wrote: >> It isn't uncommon to have a vlan on top of a bridge, on top of a bond >> (in fact, that's very common for OpenStack), and in this type of setup, >> not having the ifenslave + vlan + bridge-utils packages would be a real >> pain. > > Is it common that this is exposed to the VM though? Or does the VM just see a > virtual ethernet device? > > Ross Thinking that there's only VMs is a dangerous assumption. There's also bare-metal with Ironic. In such environment, it wouldn't be surprising to have VLAN and bonding, then the user may need bridging. For those who want to run untrusted networking for users, there's super nice network cards such as the ConnectX-6 Dx from Mellanox: https://www.mellanox.com/page/products_dyn?product_family=302=connectx_6_dx_ic These cards have a multicore ARM CPU, 16 GBytes of RAM, which contains a full operating system (I've been told, CentOS or Ubuntu), support nice stuff like NVMe over fabric, and much more. They would be ideal to support running bare-metal systems using Ironic. I haven't been able to test them though (they are at about 1600 USD a piece...). But with such cards, I am guessing it should be possible to have any kind of VLAN, bonding and bridging for the bare-metal OS, configured by cloud-init, on-demand for the customer. With the OS, it is possible to filter VLANs to make sure the customer isn't destroying the OpenStack management network (for example, by using the management network VLAN and using the IP address of the router gateway). So yeah, it does make sense to provide VLAN, bridging and bonding, at least for our non-minimal generic cloud image. It'd be IMO fine to provide them only on that one though, if we document it, so the other smaller image can be kept small. Cheers, Thomas Goirand (zigo)
Re: generic (cloud) image problems
On Sun, Dec 29, 2019 at 11:16:20PM +0100, Thomas Goirand wrote: > >> Right, both bridge-utils and vlan are required to setup a bridge or > >> vlan from *within* /etc/network/interfaces. > > > > That's not really true. bridge-utils and vlan add some additional > > syntax, but everything can be accomplished with some basic pre/post-up > > commands in the interfaces entry. > > Yes, indeed. But that's precisely what you don't want to have to do > because it's too complicated. Best is if you can just use the "normal" > syntax, with something like: > > auto vlan80 > iface vlan80 inet static > vlan-raw-device bond0 > address 10.0.0.1/24 I agree that current support is not perfect, but I don't think there's any reason for our users to feel blocked by this lack of support. 100% of expected functionality is available. > Probably, the issue is in ifupdown, which probably should be "fixed" to > use ip correctly, rather than the legacy tools. Whoever's fault, the > direct effect is that it gets less user friendly, and we should fix that. Unfortunately, AFAICT, ifupdown is also pretty much dead. It hasn't seen any major feature development in quite some time, and hasn't seen an upload to unstable in nearly a year. I suspect that the Right Way to do this on a modern Debian system would be to use systemd-networkd and systemd netdev configuration. I don't know enough about this right now to provide an example, or to figure out if/how this is usable with our current interface configuration management approach, but systemd.netdev(5) provides some basic examples. noah
Re: generic (cloud) image problems
On Sun, Dec 29, 2019 at 11:16:20PM +0100, Thomas Goirand wrote: > It isn't uncommon to have a vlan on top of a bridge, on top of a bond > (in fact, that's very common for OpenStack), and in this type of setup, > not having the ifenslave + vlan + bridge-utils packages would be a real > pain. Is it common that this is exposed to the VM though? Or does the VM just see a virtual ethernet device? Ross
Re: generic (cloud) image problems
On 12/28/19 7:27 PM, Noah Meyerhans wrote: > On Fri, Dec 27, 2019 at 10:33:16PM +0100, Christian Tramnitz wrote: >> Right, both bridge-utils and vlan are required to setup a bridge or >> vlan from *within* /etc/network/interfaces. > > That's not really true. bridge-utils and vlan add some additional > syntax, but everything can be accomplished with some basic pre/post-up > commands in the interfaces entry. Yes, indeed. But that's precisely what you don't want to have to do because it's too complicated. Best is if you can just use the "normal" syntax, with something like: auto vlan80 iface vlan80 inet static vlan-raw-device bond0 address 10.0.0.1/24 Using the pre-up/up thing is really not what one does, and certainly not what we want to impose to our users. It isn't uncommon to have a vlan on top of a bridge, on top of a bond (in fact, that's very common for OpenStack), and in this type of setup, not having the ifenslave + vlan + bridge-utils packages would be a real pain. Probably, the issue is in ifupdown, which probably should be "fixed" to use ip correctly, rather than the legacy tools. Whoever's fault, the direct effect is that it gets less user friendly, and we should fix that. Cheers, Thomas Goirand (zigo)
Re: generic (cloud) image problems
On Sa., 28. Dez. 2019 at 19:20 Uhr Noah Meyerhans wrote: > > On Sat, Dec 28, 2019 at 12:23:04PM +0100, Christian Tramnitz wrote: > > There is no workaround for the lack of vlan and bridge-utils though. > > It would be great it we could get those two included into the base > > image. > > Both vlan and bridge-utils are legacy tools that are replaced by ip(8) > on modern GNU/Linux systems. You should use that instead. I do not > believe it would be desirable to include the legacy tools in the default > images. > > If you really aren't willing to use ip(8), then I suggest constructing > your own VM images based on our configuration. This is a fully > supported use case and somethat that we do consider a reasonable option > for people who need customizations on top of what we've build. I > documented the process a while back at > https://noah.meyerhans.us/blog/2017/02/10/using-fai-to-customize-and-build-your-own-cloud-images/ > and since then we've made it easier by eliminating the need to > explicitly call fai-diskimage. If you make you package modifications to > e.g. the package_config/DEBIAN file, then you can build an image using > "make buster-image-generic" It's not about my willingness to use specific tools, it's about what the image (without additional packages installed) does when using cloud-init. I would consider my use case as category 2 (as per your blog post): I would like to directly launch the debian cloud-image and make use of cloud-init and configuration management and may never log into the system. However, the /etc/network/interfaces.d/50-cloud-init.cfg file *generated by cloud-init* uses a syntax that assumes bridge-utils and vlan are installed (when using a bridge / vlan statement in a v1 network config file). And I'm repeating myself here: the cloud-init generated config isn't even included by /etc/network/interfaces due to the wrong include statement. Maybe I'm missing something here that the network config presented by cloud-init has to look different, but I have not found any hints what the *expected* format for a Debian cloud-image would be. There is also no specific renderer in /etc/cloud/cloud.cfg so I'd assume whatever I pass to cloud-init in a supported format is parsed to an output that is understood by the system. Currently I'm using the nocloud provider with v1 network config. The cloud-image bundled cloud-init version (18.3) produces a file that cannot be used with image-provided packages. I already tried with a v2 network config file (converted with cloud-init's tools/net-convert.py) but that is failing with: cloud-init[297]: 2019-12-29 13:04:21,276 - stages.py[WARNING]: Failed to rename devices: Failed to apply network config names. Found bad network config version: None (and not creating a 50-cloud-init.cfg at all). Also, if you call bridge-utils deprecated, I'd suggest to change the documentation here: https://wiki.debian.org/BridgeNetworkConnections#Installing_the_software BR, Christian
Re: generic (cloud) image problems
On Fri, Dec 27, 2019 at 10:33:16PM +0100, Christian Tramnitz wrote: > Right, both bridge-utils and vlan are required to setup a bridge or > vlan from *within* /etc/network/interfaces. That's not really true. bridge-utils and vlan add some additional syntax, but everything can be accomplished with some basic pre/post-up commands in the interfaces entry.
Re: generic (cloud) image problems
On Sat, Dec 28, 2019 at 12:23:04PM +0100, Christian Tramnitz wrote: > There is no workaround for the lack of vlan and bridge-utils though. > It would be great it we could get those two included into the base > image. Both vlan and bridge-utils are legacy tools that are replaced by ip(8) on modern GNU/Linux systems. You should use that instead. I do not believe it would be desirable to include the legacy tools in the default images. If you really aren't willing to use ip(8), then I suggest constructing your own VM images based on our configuration. This is a fully supported use case and somethat that we do consider a reasonable option for people who need customizations on top of what we've build. I documented the process a while back at https://noah.meyerhans.us/blog/2017/02/10/using-fai-to-customize-and-build-your-own-cloud-images/ and since then we've made it easier by eliminating the need to explicitly call fai-diskimage. If you make you package modifications to e.g. the package_config/DEBIAN file, then you can build an image using "make buster-image-generic" noah
Re: generic (cloud) image problems
On Fr., 27. Dez. 2019 at 23:46 Uhr Noah Meyerhans wrote: > Yes, but the workaround I provided completely replaces the missing > functionality. IMO if the docs described the process in the workaround, > rather than the existing behavior based on apt-key, we'd be done. I think the workaround is perfectly fine. Actually I am already passing the PGP key as "key" block to cloud-init, but it is making use of apt-key add anyway. Just placing it in /etc/apt/trusted.gpg.d/foo.gpg via write_files is the better solution imho. I'm doing the very same thing for my CentOS-based cloud-init image bootstrapping process (just to a different path: /etc/pki/rpm-gpg). There is no workaround for the lack of vlan and bridge-utils though. It would be great it we could get those two included into the base image. Repairing /etc/network/interfaces to include cloud-init generated network configs would also be great. I still have to check if a workaround (also using write_files) would work here too, but I think network setup is executed even before that stage too. Best regards, Christian
Re: generic (cloud) image problems
On Fri, Dec 27, 2019 at 11:39:00PM +0100, Thomas Goirand wrote: > >> it is virtually impossible to use the image for cloud-init based > >> deployment: > >> While cloud-init can also install missing packages, the install would > >> happen *after* package installation and network setup. bridge-utils > >> and vlan are required if the network setup has either a vlan or a > >> bridge and gnupg2 is required to add 3rd party repos (otherwise > >> "apt-key add" won't work). Both - network access and 3rd party repos - > >> may be required for further steps of an deployment (i.e. to install > >> automation tools like salt, chef or puppet). > > > > I describe a workaround for the lack of "apt-key add" functionality at > > https://salsa.debian.org/cloud-team/debian-cloud-images/issues/17#note_126311 > > > > We (the cloud team) have basically concluded in recent conversations > > that cloud-init's usage of apt-key is a bug and should be removed. > > > > noah > > Though until that's fixed, Christian is right to complain. Yes, but the workaround I provided completely replaces the missing functionality. IMO if the docs described the process in the workaround, rather than the existing behavior based on apt-key, we'd be done. noah
Re: generic (cloud) image problems
On 12/27/19 10:25 PM, Noah Meyerhans wrote: > On Fri, Dec 27, 2019 at 02:16:05PM +0100, Christian Tramnitz wrote: >> However, I'm running into multiple problems staging a system through >> cloud-init. >> Without the packages >> - bridge-utils >> - vlan >> - gnupg2 >> it is virtually impossible to use the image for cloud-init based deployment: >> While cloud-init can also install missing packages, the install would >> happen *after* package installation and network setup. bridge-utils >> and vlan are required if the network setup has either a vlan or a >> bridge and gnupg2 is required to add 3rd party repos (otherwise >> "apt-key add" won't work). Both - network access and 3rd party repos - >> may be required for further steps of an deployment (i.e. to install >> automation tools like salt, chef or puppet). > > I describe a workaround for the lack of "apt-key add" functionality at > https://salsa.debian.org/cloud-team/debian-cloud-images/issues/17#note_126311 > > We (the cloud team) have basically concluded in recent conversations > that cloud-init's usage of apt-key is a bug and should be removed. > > noah Though until that's fixed, Christian is right to complain. Thomas
Re: generic (cloud) image problems
Am Fr., 27. Dez. 2019 um 22:04 Uhr schrieb Thomas Goirand : > Probably the ip command can be used to setup vlan, though to my > understanding, the vlan package is needed for ifupdown to understand > interfaces of type vlan, and the same way, bridge-utils is needed > for bridges, and ifenslave for bonding. I do believe that Christian > request is very much valid, and that the above 3 packages should be > installed, together with gnupg2. Right, both bridge-utils and vlan are required to setup a bridge or vlan from *within* /etc/network/interfaces. ifupdown2 or iproute2 don't help here. Also, if the network can't be setup initially there will be no network connection to load additional packages (chicken and egg problem) > >> and gnupg2 is required to add 3rd party repos (otherwise > >> "apt-key add" won't work). Both - network access and 3rd party repos - > >> may be required for further steps of an deployment (i.e. to install > >> automation tools like salt, chef or puppet). > > Though I do understand that gnupg2 is better there by default, I have a > few remarks that may help as workarounds. > > I don't see why you'd like to install puppet from upstream. The puppet > package in Debian is very well maintained, and probably better than the > one from upstream. In my case I need to install Saltstack salt-minion form a third party repo. Maybe puppet was a bad example, but at least chef would have a similar issue. > Also, if you're installing stuff through metadata, why can't you first > install gnupg2, then install 3rd party repos? The order in cloud-init is fixed (as per /etc/cloud/cloud.cfg). "apt-configure" (which sets up repositories) is run in the config stage while "package-update-upgrade-install" is invoked in the final stage. There is no back and forth between stages. This is also the reason why core packages need to be in the base image. If they are only installed in the final stage, modules requiring them in earlier phases can't take advantage of the install. I'm not sure if the config stage would be invoked a second time on reboot, but at least the network setup isn't performed again (after installing packages). Best regards, Christian
Re: generic (cloud) image problems
On Fri, Dec 27, 2019 at 02:16:05PM +0100, Christian Tramnitz wrote: > However, I'm running into multiple problems staging a system through > cloud-init. > Without the packages > - bridge-utils > - vlan > - gnupg2 > it is virtually impossible to use the image for cloud-init based deployment: > While cloud-init can also install missing packages, the install would > happen *after* package installation and network setup. bridge-utils > and vlan are required if the network setup has either a vlan or a > bridge and gnupg2 is required to add 3rd party repos (otherwise > "apt-key add" won't work). Both - network access and 3rd party repos - > may be required for further steps of an deployment (i.e. to install > automation tools like salt, chef or puppet). I describe a workaround for the lack of "apt-key add" functionality at https://salsa.debian.org/cloud-team/debian-cloud-images/issues/17#note_126311 We (the cloud team) have basically concluded in recent conversations that cloud-init's usage of apt-key is a bug and should be removed. noah
Re: generic (cloud) image problems
On 12/27/19 9:15 PM, Geert Stappers wrote: > On Fri, Dec 27, 2019 at 02:16:05PM +0100, Christian Tramnitz wrote: >> I'm trying to use the Buster generic image >> (https://cdimage.debian.org/cdimage/cloud/buster/20191117-80/debian-10-generic-amd64-20191117-80.qcow2) >> to automate deployment of KVM-based VMs. The reason that I'm not using >> the genericcloud or OpenStack image is that I'm passing through >> network VFs that require "hardware" drivers not included in the other >> images. >> >> However, I'm running into multiple problems staging a system through >> cloud-init. >> Without the packages >> - bridge-utils >> - vlan >> - gnupg2 >> it is virtually impossible to use the image for cloud-init based deployment: >> While cloud-init can also install missing packages, the install would >> happen *after* package installation and network setup. bridge-utils >> and vlan are required if the network setup has either a vlan or a bridge > > package iproute2 provides what you think you are missing. > > See https://www.baturin.org/docs/iproute2/#Link%20management for > examples of VLAN and bridge configuration. Probably the ip command can be used to setup vlan, though to my understanding, the vlan package is needed for ifupdown to understand interfaces of type vlan, and the same way, bridge-utils is needed for bridges, and ifenslave for bonding. I do believe that Christian request is very much valid, and that the above 3 packages should be installed, together with gnupg2. >> and gnupg2 is required to add 3rd party repos (otherwise >> "apt-key add" won't work). Both - network access and 3rd party repos - >> may be required for further steps of an deployment (i.e. to install >> automation tools like salt, chef or puppet). Though I do understand that gnupg2 is better there by default, I have a few remarks that may help as workarounds. I don't see why you'd like to install puppet from upstream. The puppet package in Debian is very well maintained, and probably better than the one from upstream. Also, if you're installing stuff through metadata, why can't you first install gnupg2, then install 3rd party repos? Cheers, Thomas Goirand (zigo)
Re: generic (cloud) image problems
On Fri, Dec 27, 2019 at 02:16:05PM +0100, Christian Tramnitz wrote: > I'm trying to use the Buster generic image > (https://cdimage.debian.org/cdimage/cloud/buster/20191117-80/debian-10-generic-amd64-20191117-80.qcow2) > to automate deployment of KVM-based VMs. The reason that I'm not using > the genericcloud or OpenStack image is that I'm passing through > network VFs that require "hardware" drivers not included in the other > images. > > However, I'm running into multiple problems staging a system through > cloud-init. > Without the packages > - bridge-utils > - vlan > - gnupg2 > it is virtually impossible to use the image for cloud-init based deployment: > While cloud-init can also install missing packages, the install would > happen *after* package installation and network setup. bridge-utils > and vlan are required if the network setup has either a vlan or a bridge package iproute2 provides what you think you are missing. See https://www.baturin.org/docs/iproute2/#Link%20management for examples of VLAN and bridge configuration. > and gnupg2 is required to add 3rd party repos (otherwise > "apt-key add" won't work). Both - network access and 3rd party repos - > may be required for further steps of an deployment (i.e. to install > automation tools like salt, chef or puppet). > > So it would be great if the above mentioned packages could be included > at least in the generic build (not sure if this is also a typical > requirement for the other images). > > > Also, the file "/etc/network/interfaces" is using an include line that > will *not* process the cloud-init generated > "/etc/network/interfaces.d/50-cloud-init.cfg": > > source-directory /etc/network/interfaces.d > > > Instead, sth like: > > source /etc/network/interfaces.d/*.cfg > > would be needed to process cloud-init generated network config - or - > the cloud-init generated network config should not have the ".cfg" > suffix. > > > > Best regards, >Christian Thanks Groeten Geert Stappers Happy the feedback from Christian, not so happy with several topics in single thread under one Subject -- Leven en laten leven
