Some network types use longer or shorter addresses, not all of them try to mimic Ethernet.
For example FireWire uses 64-bit hardware addresses but IP-over-FW extends it to 128-bit addresses in ARP for technical reasons, and I think it's the same for Infiniband and IPoIB. Unfortunately Networkd doesn't understand any of it. On Tue, Jul 30, 2024, 04:42 serenissi <sereni...@inventati.org> wrote: > I can't tell more about the IPoIB going down after networkd restart > without additional debugging info. But from the complains, did you try > removing the problematic keys (ipoib is part of netdev, not network. > network has no knowledge of the device type)? > > Also are you sure 80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11 > is the mac address? It is clearly more than 48 bits. > On 7/29/24 08:43, Chandler Sobel-Sorenson wrote: > > I'm quite frustrated having spent many hours and little success, when > things were perfectly fine before our backup generator decided not to kick > in and power surges ensued, messing up all kinds of things stupid > electrons. We have a server that is a bit more important than the others, > runs LDAP and hosts our home directories, and runs the subnet manager for > an Infiniband network. > > The main uplink is a 2x10gbe bond device, and there's a local 1gbe > 10-net interface/network, and an Infiniband with IPoIB. The main issue is > getting the IPoIB working. The bond and its 2 links work fine after the > machine boots, but whenever I restart systemd-networkd, it becomes > unreachable even though there aren't any problems reported in the system > logs and networkctl reports it's still routable. The only thing I've > figured to get it reachable again is to reboot the system, so I'd like to > figure out the problem there because that shouldn't be happening. > > The IPoIB is driving me nuts, I read through all of systemd.network > and systemd.netdev docs, and it's somehow become rogue and unmanaged when > it used to be configured. systemd-networkd keeps telling me about unknown > keys and unknown sections, even though they were all added in versions > previous to the current, which is: > > systemd 252 (252.26-1~deb12u2) running on Debian 11 Linux 5.10.0-20-amd64 > > Below are the configs and messages from systemd. Hope you can help. > # networkctl > *IDX LINK TYPE OPERATIONAL SETUP * > 1 lo loopback carrier unmanaged > 2 ens1f0 ether enslaved configured <--10gbe link 1 > 3 enp4s0f0 ether off unmanaged <--not connected > 4 ens1f1 ether enslaved configured <--10gbe link 2 > 5 enp4s0f1 ether routable configured <--1gbe local 10-net > 6 ibs3 infiniband off unmanaged <--lazy, rogue > 7 ibs3d1 infiniband off unmanaged <--not connected > 8 bond007 bond routable configured > > *Infiniband, IPoIB, ibs3 * # cat 10-ibs3.netdev > [Match] > > [NetDev] > Name=ibs3 > Kind=ipoib > MTUBytes=65520 > MACAddress=80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11 > > [IPoIB] > Mode=connected > > > # cat ibs3-10_10_11_203.network > [Match] > Name=ibs3 > > PermanentMACAddress=80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11 > Path=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/ibs3 > Driver=ib_ipoib > Type=infiniband > Kind=ipoib > Property=ID_NET_MANAGED_BY=io.systemd.Network > > [Link] > MTUBytes=65520 > > [Network] > Kind=ipoib > Address=10.10.11.203/24 > Gateway=10.10.11.203 > LinkLocalAddressing=no > IPv4AcceptLocal=yes > KeepConfiguration=static > > [IPoIB] > Mode=connected > systemd-networkd complaints > /etc/systemd/network/10-ibs3.netdev:8: Not a valid hardware address, > ignoring assignment: > 80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11 > /etc/systemd/network/10-ibs3.netdev:8: Not a valid hardware address, > ignoring assignment: > 80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11 > /etc/systemd/network/ibs3-10_10_11_203.network:15: Unknown key 'Kind' in > section [Network], ignoring. > /etc/systemd/network/ibs3-10_10_11_203.network:22: Unknown section > 'IPoIB'. Ignoring. > > *bond* > # cat 10-bond007.netdev > [NetDev] > Name=bond007 > Kind=bond > MTUBytes=9000 > > [Bond] > Mode=802.3ad > MIIMonitorSec=1000 > UpDelaySec=1000 > DownDelaySec=2000 > > > # cat Intel_X710_DA2-bond007.network > [Match] > Path=pci-0000:05:00.0 > Path=pci-0000:05:00.1 > > [Network] > Bond=bond007 > > > # cat bond007-10_140_78_70.network > [Match] > Name=bond007 > > [Network] > Address=10.140.78.70/28 > Gateway=10.140.78.65 > DNS=128.196.11.233 > DNS=128.196.11.234 > DNS=128.196.11.235 > LinkLocalAddressing=no > IPv6AcceptRA=no > > Best Regards, > Chandler > [image: The University of Arizona block 'A' logo.] > *Chandler Sobel-Sorenson* > Sr. Systems Administrator > Arizona Genomics Institute > School of Plant Sciences—Research > THE UNIVERSITY OF ARIZONA > > Thomas W. Keating Bioresearch Bldg. | Rm. 200A24 > 1657 E. Helen St. | Tucson, AZ 85721 > <https://www.google.com/maps/search/1657+E.+Helen+St.+%7C+Tucson,+AZ+85721+%0D%0A++++++++++++++++++++++Office:++520?entry=gmail&source=g> > Office: > <https://www.google.com/maps/search/1657+E.+Helen+St.+%7C+Tucson,+AZ+85721+%0D%0A++++++++++++++++++++++Office:++520?entry=gmail&source=g> > 520-626-9589 | Cell: 520-907-4352 > > chand...@genome.arizona.edu <+chand...@genome.arizona.edu> > Pronouns: he/him/his > *www.genome.arizona.edu* <https://www.genome.arizona.edu/> > > [image: Integrity, Compassion, Exploration, Adaptation, Inclusion, > Determination] <https://brand.arizona.edu/signature> > >
