Bug#1042842: network interface names wrong in domU (>10 interfaces)
Hello, I'm currently in the process of cleaning up and i noticed this bug i reported is still open. Based on what we know now i think the bug can be closed. To summarize: * the ordering of network interfaces (and therefore the ethn names) was never meant to be stable and changed with Xen 4.17 * domUs starting with bookworm use enXn by default and their order matches the config * bullseye domUs on a bookworm dom0 with more than 10 network interfaces need a workaround: * use udev from bullseye-backports and enXn naming scheme * use custom udev rules to get fixed interface names Thanks for your help with this issue, Valentin
Bug#1042842: network interface names wrong in domU (>10 interfaces)
On 08 Aug 2023 16:59, Hans van Kranenburg wrote: I didn't read the other mailthread on the xen list fully yet. You gave me the idea to post the IRC digest, so the report here is more complete, and people not tracking the xen-devel ML can read it nicely. For those who do, the mail is dated "02 Aug 2023 18:19", and titled "Network interfaces naming changes in domUs with >10 vifs (Debian bug 1042842)". The ML post got no answer yet. [--- IRC ---] - AFAIK, there is no sorting in Xenstored. And you should not expect that even if libxl sorted properly it will be seen in the same order on the other end. - is the ethN number in domU related to vif number in xenstore, or to device detection order? - there's no order to eth names at all. they're allocated first-come-first-serve, so it entirely depends on how parallel the probing of nic drivers are. even if netfront is serialised around xenstore accesses, it probably allocates in the order that XS_DIRECTORY comes back with - from simple tests, it looks like VIFs are created in Xenstore in the order of the config file, but if you "xenstore-ls /[...]/vif", you can see vifs are ordered like vif1,vif10,vif11,vif2,etc - the order is different between Xen 4.14 and 4.17 (ie. the "expected" order works on 4.14, not 4.17) - But really, Debian should have never relied on how the nodes are ordered. This is not something we guarantee in the Xenstored API - the last big batch of XSA content for the xenstoreds did some major rearranging of oxenstored. We dropped a NIH second garbage collector, and a NIH weakref system IIRC. I could entirely believe that the apparent sort order changed as a result - generally, I think Linux world established quite some time ago that ethN names are not stable - It's definitely a complicated issue. Perhaps best to post to xen-devel so we can have a discussion. I expect the answer is not-a-Xen bug, but I don't think we have a clear understanding of the problem yet [--- /IRC ---] I'll report back when having tested the 111 vifs domU ... if my system agrees o_O As it requires a script to populate the cfg, one could also enhance it to try how dynamically adding/removing vifs is handled. (BTW, before this report I thought Xen had a hard limit of 8 vifs per domU. Or was that only on FreeBSD domUs ? Can't remember).
Bug#1042842: network interface names wrong in domU (>10 interfaces)
Hi,
On 8/8/23 15:22, Valentin Kleibel wrote:
>> On [0], you can read "In both cases the device naming is subject to the
>> usual guest or backend domain facilities for renaming network devices".
>> It says "naming/renaming", but you can assume "detecting".
>>
>>> I also checked which net_ids udev knows about and the only things that
>>> pop up are:
>>> ID_NET_NAMING_SCHEME=v247
>>> ID_NET_NAME_MAC=enx00163efd832b
>>> ID_OUI_FROM_DATABASE=Xensource, Inc.
What I do is stuff like this:
-$ cat /etc/udev/rules.d/70-persistent-vifname.rules
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/0",
NAME="vlan2"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/1",
NAME="vlan3"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/2",
NAME="vlan4"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/3",
NAME="vlan6"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/4",
NAME="vlan9"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/5",
NAME="vlan10"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/6",
NAME="vlan11"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/7",
NAME="vlan12"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/8",
NAME="vlan13"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/9",
NAME="vlan14"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/10",
NAME="vlan15"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/11",
NAME="vlan16"
The vif/X always matches the order in which you define the interfaces
inside the guest config file.
After starting to build router VMs (well before the whole interface
naming madness was a thing), it took only the first time when we wanted
to throw away a vlan, to realize that all the ethX numbers would shift 1
up, and from then on, I've always been using this so set my own style
predictable names (whenever there's more than one, otherwise it's just
eth0).
>> Is it from dom0 or domU ?
>> Are you using "net.ifnames=0" on the domU kernel command line ?
>> "v247" looks like systemd "predictive naming scheme" (eth -> enX).
>> From bookworm on, domUs vifs get named enXN (enX0, enX1, ...).
>> Read on :
>> https://www.debian.org/releases/stable/i386/release-notes/ch-information.en.html#xen-network
>
> This is from the domU, running bullseye with a bookworm dom0.
>
>> See how ethN interfaces get messed up, like in your setup, but
>> predictable names would work, as you can see in "altname enXN" :
>> eth1 (:01) -> enX1
>> eth2 (:10) -> enX10
>> eth3 (:02) -> enX2
But yeah, so, even while not depending on whatever order it gets
initialized, and still having it function correctly, this is still just
pretty annoying... If I'm doing stuff around here, and just quickly want
to look up things (e.g. messing around with vlan15 settings), and
quickly type ip a instead of having to spend more time typing ip a show
dev vlan15 jadijadi, I still every time get this short "WTF huh, argh",
raises arms, does table flip, grmbl grbml feeling for a split second.
2: vlan2: I could not get our bullseye domU to show the "predictable names" even
> though i tried installing the bullseye-backports kernel 6.1.
> After you wrote this i installed udev 252.5 from backports and it now
> uses the correct enXn interface names, even with kernel 5.10.
>
>> So, my answer does not tell you if something changed in Xen itself, only
>> in Debian.
>> But I guess it relates to what Xen devs told us : vifs detection order
>> cannot be relied upon, that's why "predictable names" were invented.
>> The vif detection part is related to the domains kernels, not Xen itself
>> (at least that's what I understood).
>>
>> Using eth0 nowadays is a bit like using /dev/sda for hard drives, it's
>> considered legacy as it may create problems in some setups, like yours
>> (ie. for disks, it's recommended to use UUIDs or /dev/disk/by-*).
>>
>> I hope this answers your question.
>
> Thank you, yes it does.
>
> In our case the dom0 was updated to bookworm while the domU is still
> running bullseye.
> -> updated Xen so the vif detection order changed (which we relied on)
I didn't read the other mailthread on the xen list fully yet. But, I
think it's shouldn't be very hard to find the code changes and see if
it's deterministic and can just be fixed. Simply just to decrease the
totally unnecessary amount of silliness.
> -> the predictable network names for Xen don't work with bullseye
>
> So my new resolution for bullseye domUs on a bookworm dom0 is to install
> udev from backports and change the domUs network config to use the new
> enXn naming scheme instead of ethn.
Or the "device/vif/X" way...
So, anyway, did someone already did some test "just because we can" to
see how much network interfaces you can get added for fun, and if the
pattern keeps looking the same, also with enX4 enX40 .. enX49 enX5 etc?
:D enX1 enX10 enX100 .. enX109 enX11 enX110 argh o_O
Have fun,
Hans
Bug#1042842: [Pkg-xen-devel] Bug#1042842: network interface names wrong in domU (>10 interfaces)
On [0], you can read "In both cases the device naming is subject to the usual guest or backend domain facilities for renaming network devices". It says "naming/renaming", but you can assume "detecting". I also checked which net_ids udev knows about and the only things that pop up are: ID_NET_NAMING_SCHEME=v247 ID_NET_NAME_MAC=enx00163efd832b ID_OUI_FROM_DATABASE=Xensource, Inc. Is it from dom0 or domU ? Are you using "net.ifnames=0" on the domU kernel command line ? "v247" looks like systemd "predictive naming scheme" (eth -> enX). From bookworm on, domUs vifs get named enXN (enX0, enX1, ...). Read on : https://www.debian.org/releases/stable/i386/release-notes/ch-information.en.html#xen-network This is from the domU, running bullseye with a bookworm dom0. See how ethN interfaces get messed up, like in your setup, but predictable names would work, as you can see in "altname enXN" : eth1 (:01) -> enX1 eth2 (:10) -> enX10 eth3 (:02) -> enX2 I could not get our bullseye domU to show the "predictable names" even though i tried installing the bullseye-backports kernel 6.1. After you wrote this i installed udev 252.5 from backports and it now uses the correct enXn interface names, even with kernel 5.10. So, my answer does not tell you if something changed in Xen itself, only in Debian. But I guess it relates to what Xen devs told us : vifs detection order cannot be relied upon, that's why "predictable names" were invented. The vif detection part is related to the domains kernels, not Xen itself (at least that's what I understood). Using eth0 nowadays is a bit like using /dev/sda for hard drives, it's considered legacy as it may create problems in some setups, like yours (ie. for disks, it's recommended to use UUIDs or /dev/disk/by-*). I hope this answers your question. Thank you, yes it does. In our case the dom0 was updated to bookworm while the domU is still running bullseye. -> updated Xen so the vif detection order changed (which we relied on) -> the predictable network names for Xen don't work with bullseye So my new resolution for bullseye domUs on a bookworm dom0 is to install udev from backports and change the domUs network config to use the new enXn naming scheme instead of ethn.
Bug#1042842: [Pkg-xen-devel] Bug#1042842: network interface names wrong in domU (>10 interfaces)
On 08 Aug 2023 12:08, Valentin Kleibel wrote: I posted on xen-devel, you can follow from : https://lists.xenproject.org/archives/html/xen-devel/2023-08/msg00244.html (Unfortunately, the formatting is weird via html, split the IRC part on "- "). Thank you for posting upstream. No prob, although if that very answer does not answer your question, I guess you'd be better off replying on xen-devel ML (reply to my post or at least reference it). All documentation i found found on the Xen wiki suggests that interfaces are connected vifX.Y <-> ethY. [0] [1] The only other way i know of for identifying the interfaces are MAC Addresses which can be randomly assigned if you don't configure them. On [0], you can read "In both cases the device naming is subject to the usual guest or backend domain facilities for renaming network devices". It says "naming/renaming", but you can assume "detecting". I also checked which net_ids udev knows about and the only things that pop up are: ID_NET_NAMING_SCHEME=v247 ID_NET_NAME_MAC=enx00163efd832b ID_OUI_FROM_DATABASE=Xensource, Inc. Is it from dom0 or domU ? Are you using "net.ifnames=0" on the domU kernel command line ? "v247" looks like systemd "predictive naming scheme" (eth -> enX). From bookworm on, domUs vifs get named enXN (enX0, enX1, ...). Read on : https://www.debian.org/releases/stable/i386/release-notes/ch-information.en.html#xen-network Either i am missing the way you're supposed to do this, or there is a bug somewhere in the toolchain. Unfortunately i'm not able to pinpoint the source of the issue, any help would be appreciated. I made some tests with a domU using many interfaces, like : [...] vif = [ 'bridge=xbr-tst,mac=00:16:3e:de:bd:00,type=vif,vifname=domu-a' , 'bridge=xbr-tst,mac=00:16:3e:de:bd:01,type=vif,vifname=domu-b' , 'bridge=xbr-tst,mac=00:16:3e:de:bd:02,type=vif,vifname=domu-c' , 'bridge=xbr-tst,mac=00:16:3e:de:bd:03,type=vif,vifname=domu-d' , 'bridge=xbr-tst,mac=00:16:3e:de:bd:04,type=vif,vifname=domu-e' , 'bridge=xbr-tst,mac=00:16:3e:de:bd:05,type=vif,vifname=domu-f' , 'bridge=xbr-tst,mac=00:16:3e:de:bd:06,type=vif,vifname=domu-g' , 'bridge=xbr-tst,mac=00:16:3e:de:bd:07,type=vif,vifname=domu-h' , 'bridge=xbr-tst,mac=00:16:3e:de:bd:08,type=vif,vifname=domu-i' , 'bridge=xbr-tst,mac=00:16:3e:de:bd:09,type=vif,vifname=domu-j' , 'bridge=xbr-tst,mac=00:16:3e:de:bd:10,type=vif,vifname=domu-k' , ] [...] - This is dom0's corresponding dmesg: [...] xbr-tst: port 3(domu-b) entered blocking state xbr-tst: port 3(domu-b) entered disabled state device domu-b entered promiscuous mode xbr-tst: port 4(domu-i) entered blocking state xbr-tst: port 4(domu-i) entered disabled state device domu-i entered promiscuous mode [...] Here you can see : port 3 <-> domu-b port 4 <-> domu-i We learn here that dom0 did not detect vifs serially. - In the domU, "ip link" shows : [...] eth0 link/ether 00:16:3e:de:bd:00 altname enX0 eth1 link/ether 00:16:3e:de:bd:01 altname enX1 eth2 link/ether 00:16:3e:de:bd:10 altname enX10 eth3 link/ether 00:16:3e:de:bd:02 altname enX2 [...] See how ethN interfaces get messed up, like in your setup, but predictable names would work, as you can see in "altname enXN" : eth1 (:01) -> enX1 eth2 (:10) -> enX10 eth3 (:02) -> enX2 So, my answer does not tell you if something changed in Xen itself, only in Debian. But I guess it relates to what Xen devs told us : vifs detection order cannot be relied upon, that's why "predictable names" were invented. The vif detection part is related to the domains kernels, not Xen itself (at least that's what I understood). Using eth0 nowadays is a bit like using /dev/sda for hard drives, it's considered legacy as it may create problems in some setups, like yours (ie. for disks, it's recommended to use UUIDs or /dev/disk/by-*). I hope this answers your question.
Bug#1042842: network interface names wrong in domU (>10 interfaces)
I posted on xen-devel, you can follow from : https://lists.xenproject.org/archives/html/xen-devel/2023-08/msg00244.html (Unfortunately, the formatting is weird via html, split the IRC part on "- "). Thank you for posting upstream. Note that, at first sight, I was told this seems "not-a-Xen" bug (read the IRC excerpts). All documentation i found found on the Xen wiki suggests that interfaces are connected vifX.Y <-> ethY. [0] [1] The only other way i know of for identifying the interfaces are MAC Addresses which can be randomly assigned if you don't configure them. I also checked which net_ids udev knows about and the only things that pop up are: ID_NET_NAMING_SCHEME=v247 ID_NET_NAME_MAC=enx00163efd832b ID_OUI_FROM_DATABASE=Xensource, Inc. Either i am missing the way you're supposed to do this, or there is a bug somewhere in the toolchain. Unfortunately i'm not able to pinpoint the source of the issue, any help would be appreciated. Valentin [0] https://wiki.xenproject.org/wiki/Xen_Networking#Paravirtualised_Network_Devices [1] https://wiki.xenproject.org/wiki/Network_Throughput_and_Performance_Guide#Technical_Overview
Bug#1042842: [Pkg-xen-devel] Bug#1042842: network interface names wrong in domU (>10 interfaces)
On 02 Aug 2023 18:09, Valentin Kleibel wrote: #xen-devel is the IRC Xen channel. I just pinged them, I'll wait. Depending on their answers, I'll post on the xen-devel mailing list. thanks for the clarification, looking forward to an answer. I posted on xen-devel, you can follow from : https://lists.xenproject.org/archives/html/xen-devel/2023-08/msg00244.html (Unfortunately, the formatting is weird via html, split the IRC part on "- "). Note that, at first sight, I was told this seems "not-a-Xen" bug (read the IRC excerpts). Our current workaround is to edit the interface names in the domUs config to match the wrong sorting. And be extra careful that the domUs MACs match the ones we expect on that network. Via udev (MAC matching) or /etc/network/interfaces ? I ask because it may help others, while this gets resolved. We just edited /etc/network/interfaces, as it only affects a few of our domUs. i think udev rules matching the MAC would be a better solution. I just didn't take the time to write them and went for the quick and dirty solution. Till it works, "whatever the bottle, till we have the poison" ;) This link may be useful: https://wiki.debian.org/NetworkInterfaceNames
Bug#1042842: network interface names wrong in domU (>10 interfaces)
#xen-devel is the IRC Xen channel. I just pinged them, I'll wait. Depending on their answers, I'll post on the xen-devel mailing list. thanks for the clarification, looking forward to an answer. Our current workaround is to edit the interface names in the domUs config to match the wrong sorting. And be extra careful that the domUs MACs match the ones we expect on that network. Via udev (MAC matching) or /etc/network/interfaces ? I ask because it may help others, while this gets resolved. We just edited /etc/network/interfaces, as it only affects a few of our domUs. i think udev rules matching the MAC would be a better solution. I just didn't take the time to write them and went for the quick and dirty solution. Valentin
Bug#1042842: network interface names wrong in domU (>10 interfaces)
Hello, the bug has been mentionned on #xen-devel, will keep you posted. Meanwhile, you may try to force the domU vif names with a letter, like : vif = [ 'mac=00:16:3e:fd:83:2f,bridge=lanbr,vifname=domu-a', 'mac=00:16:3e:fd:83:30,bridge=lanbr,vifname=domu-b', 'mac=00:16:3e:fd:83:31,bridge=lanbr,vifname=domu-c', ... ] Note it's just a workaround, and I've not tested it. I only guess letters would be sorted correctly. If you test this, can you report back please ? -- zithro / Cyril
Bug#1042842: network interface names wrong in domU (>10 interfaces)
Package: xen-utils-4.17 Version: 4.17.1+2-gb773c48e36-1 Severity: important Dear Maintainers, On one of our domUs we discovered that the network interface names were wrongly assigned since recreating the domU after an upgrade to bookworm. If over 10 network interfaces are configured the mapping (dom0) vifX.10 <-> eth10 (domU) does not apply anymore. Instead the interfaces on dom0 are sorted primarily by the leftmost digit. so for 11 interfaces we will end up with: vifX.0 <> eth0 vifX.1 <> eth1 vifX.10 <> eth2 vifX.2 <> eth3 vifX.3 <> eth4 This was observed with linux-kernel versions 5.10.179-3 and 6.1.38-2 (all combinations of domU and dom0) and xen 4.17.1+2-gb773c48e36-1. You can find a config snippet and "xl network-list" + "ip a" command output demonstrating the issue below. Booting the host with Xen 4.14.5+94-ge49571868d-1 restored the expected behaviour. Looking for relevant changes i found commit fce6999 [0] which changes the way libxl__device_list works, but i'm not sure that's the cause of this issue. Thanks for your help, Valentin [0] http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=fce6999 Sample vif configuration (ascending MACs): vif = [ 'mac=00:16:3e:fd:83:2f,bridge=lanbr', 'mac=00:16:3e:fd:83:30,bridge=lanbr', 'mac=00:16:3e:fd:83:31,bridge=lanbr', 'mac=00:16:3e:fd:83:32,bridge=lanbr', 'mac=00:16:3e:fd:83:33,bridge=lanbr', 'mac=00:16:3e:fd:83:34,bridge=lanbr', 'mac=00:16:3e:fd:83:35,bridge=lanbr', 'mac=00:16:3e:fd:83:36,bridge=lanbr', 'mac=00:16:3e:fd:83:37,bridge=lanbr', 'mac=00:16:3e:fd:83:38,bridge=lanbr', 'mac=00:16:3e:fd:83:39,bridge=lanbr', ] dom0# xl network-list 3 Idx BE Mac Addr. handle state evt-ch tx-/rx-ring-ref BE-path 0 0 00:16:3e:fd:83:2f 0 4 -1-1/-1 /local/domain/0/backend/vif/3/0 1 0 00:16:3e:fd:83:30 1 4 -1-1/-1 /local/domain/0/backend/vif/3/1 10 0 00:16:3e:fd:83:3910 4 -1-1/-1 /local/domain/0/backend/vif/3/10 2 0 00:16:3e:fd:83:31 2 4 -1-1/-1 /local/domain/0/backend/vif/3/2 3 0 00:16:3e:fd:83:32 3 4 -1-1/-1 /local/domain/0/backend/vif/3/3 4 0 00:16:3e:fd:83:33 4 4 -1-1/-1 /local/domain/0/backend/vif/3/4 5 0 00:16:3e:fd:83:34 5 4 -1-1/-1 /local/domain/0/backend/vif/3/5 6 0 00:16:3e:fd:83:35 6 4 -1-1/-1 /local/domain/0/backend/vif/3/6 7 0 00:16:3e:fd:83:36 7 4 -1-1/-1 /local/domain/0/backend/vif/3/7 8 0 00:16:3e:fd:83:37 8 4 -1-1/-1 /local/domain/0/backend/vif/3/8 9 0 00:16:3e:fd:83:38 9 4 -1-1/-1 /local/domain/0/backend/vif/3/9 domU# ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:16:3e:fd:83:2f brd ff:ff:ff:ff:ff:ff inet X.X.X.X/16 brd X.X.X.X scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::216:3eff:fefd:832f/64 scope link valid_lft forever preferred_lft forever 3: eth1: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:fd:83:30 brd ff:ff:ff:ff:ff:ff 4: eth2: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:fd:83:39 brd ff:ff:ff:ff:ff:ff 5: eth3: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:fd:83:31 brd ff:ff:ff:ff:ff:ff 6: eth4: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:fd:83:32 brd ff:ff:ff:ff:ff:ff 7: eth5: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:fd:83:33 brd ff:ff:ff:ff:ff:ff 8: eth6: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:fd:83:34 brd ff:ff:ff:ff:ff:ff 9: eth7: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:fd:83:35 brd ff:ff:ff:ff:ff:ff 10: eth8: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:fd:83:36 brd ff:ff:ff:ff:ff:ff 11: eth9: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:fd:83:37 brd ff:ff:ff:ff:ff:ff 12: eth10: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:fd:83:38 brd ff:ff:ff:ff:ff:ff
