Hi Andrei, On Thu, Nov 18, 2021, 12:13 AM Andrei Borzenkov <arvidj...@gmail.com> wrote:
> On 18.11.2021 03:20, Brian Hutchinson wrote: > > Yet another update, I was able to get it working .. but feel like it is a > > hack so comments welcome ... see below: > > > > On Wed, Nov 17, 2021 at 12:26 AM Brian Hutchinson <b.hutch...@gmail.com> > > wrote: > > > >> Update below > >> > >> On Tue, Nov 16, 2021 at 2:27 PM Brian Hutchinson <b.hutch...@gmail.com> > >> wrote: > >> > >>> Hi Mikulėnas, > >>> > >>> On Tue, Nov 16, 2021, 3:12 AM Mantas Mikulėnas <graw...@gmail.com> > wrote: > >>> > >>>> Most of this looks like it could be done with systemd-networkd to > create > >>>> a bond .netdev, with a small oneshot service for i2c. (What's the > exact > >>>> criteria for when it should be run? Does it depend on bond0 being > there, > >>>> does it need to be last, etc?) > >>>> > >>> > >>> It can be last in the startup chain I guess, don't know what other > >>> systemd things that might need the network to be up before the last > unit > >>> file runs. > >>> > >>> I start linuxptp too so I would have the unit file that starts ptp4l > >>> start after the bond was created etc. > >>> > >>> Same thing for the i2c command to enable the switch. > >>> > >>> Regards, > >>> > >>> Brian > >>> > >>> > >>>> On Tue, Nov 16, 2021, 02:58 Brian Hutchinson <b.hutch...@gmail.com> > >>>> wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> I'm on a IMX8 platform and have a Microchip KSZ9567 Ethernet > switch. I > >>>>> can use IP commands to manually bring lan1 and lan2 interfaces up > and then > >>>>> create a redundant/failover bond ... but I'm having difficulty > figuring out > >>>>> how to do this the "systemd" way. > >>>>> > >>>>> My first attempt was to just have systemd run a script of all the > >>>>> commands I do manually but during system startup there appears to be > race > >>>>> conditions so I have to set my service type to "Idle" and sometimes > even > >>>>> that doesn't work. So I want to exploit any systemd support for DSA > and > >>>>> bonding. > >>>>> > >>>>> Here is script my manual steps which is what I want systemd to > >>>>> ultimately do: > >>>>> > >>>>> #!/bin/bash > >>>>> > >>>>> # Create a redundant bond between ksz9567 DSA lan1 and lan2 > interfaces > >>>>> > >>>>> # Load bonding kernel module > >>>>> modprobe bonding > >>>>> > >>>>> # Bring up CPU interface (cpu to switch port 7 - the RGMII link) > >>>>> ip link set eth0 up > >>>>> > >>>>> # Create a bond > >>>>> echo +bond0 > /sys/class/net/bonding_masters > >>>>> > >>>>> # Set mode to active-backup (redundancy failover) > >>>>> echo active-backup > /sys/class/net/bond0/bonding/mode > >>>>> > >>>>> # Set time it takes (in ms) for slave to move when a link goes down > >>>>> echo 1000 > /sys/class/net/bond0/bonding/miimon > >>>>> > >>>>> # Add slaves to bond > >>>>> > >>>>> echo +lan1 > /sys/class/net/bond0/bonding/slaves > >>>>> echo +lan2 > /sys/class/net/bond0/bonding/slaves > >>>>> > >>>>> # Set IP and netmask of the bond > >>>>> ip addr add 192.168.0.4/24 dev bond0 > >>>>> > >>>>> # And bring bond up. Pings and network connectivity should work now > >>>>> ip link set bond0 up > >>>>> > >>>>> # For a board that doesn't have Ethernet switch hardware strapped to > >>>>> enable at boot .. enable it now > >>>>> i2cset -f -y 0 0x5f 0x03 0x00 0x01 i > >>>>> > >>>>> Thanks for any information, pointers etc. > >>>>> > >>>>> Regards, > >>>>> > >>>>> Brian > >>>>> > >>>> > >> So not sure I'm doing this right. eth0 needs to be up before lan1 and > >> lan2 can be added to the bond. Not quite sure how to do that with > systemd > >> but I made the following files and see some progress but ping doesn't > work > >> so appears I have no network connectivity: > >> > >> root@imx8mmevk:/etc/systemd/network# cat 10-bond1.netdev > >> [NetDev] > >> Name=bond1 > >> Kind=bond > >> > >> [Bond] > >> Mode=active-backup > >> PrimaryReselectPolicy=failure > >> MIIMonitorSec=2s > >> > >> root@imx8mmevk:/etc/systemd/network# cat 10-bond1.network > >> [Match] > >> Name=bond1 > >> > >> [Network] > >> Address=192.168.0.4/24 > >> > >> root@imx8mmevk:/etc/systemd/network# cat 20-lan1.network > >> [Match] > >> Name=lan1 > >> > >> [Network] > >> Bond=bond1 > >> PrimarySlave=true > >> > >> root@imx8mmevk:/etc/systemd/network# cat 30-lan2.network > >> > >> [Match] > >> Name=lan2 > >> > >> [Network] > >> Bond=bond > >> > >> ip link list > >> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode > >> DEFAULT group default qlen 1000 > >> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > >> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1506 qdisc mq state UP > mode > >> DEFAULT group default qlen 1000 > >> link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff > >> 3: lan1@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode > >> DEFAULT group default qlen 1000 > >> link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff > >> 4: lan2@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode > >> DEFAULT group default qlen 1000 > >> link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff > >> 5: bond1: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc > >> noqueue state DOWN mode DEFAULT group default qlen 1000 > >> link/ether be:87:0a:9b:13:03 brd ff:ff:ff:ff:ff:ff > >> > >> cat /proc/net/bonding/bond1 > >> Ethernet Channel Bonding Driver: v5.10.69 > >> > >> Bonding Mode: fault-tolerance (active-backup) > >> Primary Slave: None > >> Currently Active Slave: None > >> MII Status: down > >> MII Polling Interval (ms): 2000 > >> Up Delay (ms): 0 > >> Down Delay (ms): 0 > >> Peer Notification Delay (ms): 0 > >> > >> At this point there should be info on lan1 and lan2 status but don't see > >> it. > >> > >> ... so of course I can't ping. > >> > >> Next I did systemctl restart systemd-networkd and saw the following: > >> > >> systemctl restart systemd-networkd > >> root@imx8mmevk:~# [ 33.816313] device eth0 entered promiscuous mode > >> [ 33.821026] audit: type=1700 audit(1636550966.339:2): dev=eth0 > prom=256 > >> old_prom=0 auid=4294967295 uid=995 gid=994 ses=4294967295 > >> [ 33.867164] ksz9477-switch 0-005f lan2: configuring for phy/gmii link > >> mode > >> [ 33.875066] bond1: (slave lan2): Enslaving as a backup interface > with a > >> down link > >> [ 33.919055] ksz9477-switch 0-005f lan1: configuring for phy/gmii link > >> mode > >> [ 33.926683] bond1: (slave lan1): Enslaving as a backup interface > with a > >> down link > >> [ 38.066148] ksz9477-switch 0-005f lan1: Link is Up - 1Gbps/Full - > flow > >> control rx/tx > >> [ 39.472022] bond1: (slave lan1): link status definitely up, 1000 Mbps > >> full duplex > >> [ 39.479537] bond1: (slave lan1): making interface the new active one > >> [ 39.486154] bond1: active interface up! > >> [ 39.490071] IPv6: ADDRCONF(NETDEV_CHANGE): bond1: link becomes ready > >> > >> At which point I can ping. Feels like there still might be some kind of > >> race condition as things won't work unless I manually issue systemctl > >> restart systemd-networkd command after logging in. > >> > >> In kernel dmesg logs I see: > >> [ 4.165940] bond1: (slave lan2): Opening slave failed > >> [ 4.196834] bond1: (slave lan1): Opening slave failed > >> [ 4.315588] Generic PHY fixed-0:00: attached PHY driver [Generic PHY] > >> (mii_bus:phy_addr=fixed-0:00, irq=POLL) > >> [ 4.326181] fec 30be0000.ethernet eth0: Link is Up - 1Gbps/Full - > flow > >> control off > >> [ 4.561000] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > >> > >> Loaded: loaded (/lib/systemd/system/systemd-networkd.service; > enabled; > >> vendor preset: enabled) > >> Active: [[0;1;32mactive (running)[[0m since Sun 2020-09-20 10:43:59 > >> UTC; 1 years 1 months ago > >> TriggeredBy: [[0;1;32m*[[0m systemd-networkd.socket > >> Docs: man:systemd-networkd.service(8) > >> Main PID: 251 (systemd-network) > >> Status: "Processing requests..." > >> Tasks: 1 (limit: 1574) > >> Memory: 1.5M > >> CGroup: /system.slice/systemd-networkd.service > >> `-251 /lib/systemd/systemd-networkd > >> > >> Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: > [[0;1;31m[[0;1;39m[[0;1;31mlan2: > >> Could not join netdev: Network is down[[0m > >> Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: > [[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mlan2: > >> Failed[[0m > >> Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: > [[0;1;31m[[0;1;39m[[0;1;31mlan1: > >> Could not join netdev: Network is down[[0m > >> Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: > [[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mlan1: > >> Failed[[0m > >> Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: eth0: IPv6 successfully > >> enabled > >> Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: eth0: Link UP > >> Nov 10 13:28:57 imx8mmevk systemd-networkd[251]: eth0: Gained carrier > >> Nov 10 13:28:57 imx8mmevk systemd-networkd[251]: bond1: IPv6 > successfully > >> enabled > >> Nov 10 13:28:57 imx8mmevk systemd-networkd[251]: bond1: Link UP > >> Nov 10 13:28:58 imx8mmevk systemd-networkd[251]: eth0: Gained IPv6LL > >> > >> ... so looks like the bond stuff is trying to happen before eth0 (my DSA > >> HOST/CPU interface to switch) is up. How can I make eth0 up with > systemd? > >> eth0 just needs to be up ... no IP etc., and the bond1 gets the IP etc. > >> > >> For now I'm doing the i2c command to enable my switch in u-boot but > still > >> need to incorporate that into systemd somehow. > >> > >> Any ideas as to what I'm doing wrong? I think at a minimum I need to > >> bring eth0 up before the bonding stuff happens but not quite sure what > that > >> would look like using systemd. > >> > >> Regards, > >> > >> Brian > >> > >> > > I tried and tried to get eth0 to come up before the bond was brought > up. I > > had everything named in lexical order but didn't appear to matter. I > added > > a eth0.network file and in it specified ActivationPolicy=always-up and > > other things but could not get eth0 to come up. > > > > It was obvious the bond was trying to be established before eth0 was up > and > > since this is using DSA that just won't work. Dmesg logs would show > slaves > > being added before eth0 was up as in copy/paste from previous email. > > > > So I added a service to bring eth0 up: > > > > cat eth0-up.service > > [Unit] > > Description=Bring eth0 up before bonding > > Before=network-pre.target > > Wants=network-pre.target > > > > [Service] > > ExecStart=/usr/local/bin/eth0-up.sh > > RemainAfterExit=yes > > > > [Install] > > WantedBy=multi-user.target > > > > cat /usr/local/bin/eth0-up.sh > > #!/bin/bash > > ip link set eth0 up > > > > ... I feel like this is a hack, that systemd can probably do this but > > either I can't figure out how or there is a problem with the code. > > > > How lan1 and lan2 are related to eth0? Your script never creates or sets > up them. > ... its a DSA (Distributed Switch Architecture) thing. Port 7 of my switch is a RGMII CPU/host link to my IMX8 which is eth0. When using DSA driver to bust switch up into individual ports (like individual NIC cards), eth0 cannot be used for anything (like assigned an IP address etc.) but the interface has to be up for DSA driver and also to add slaves to bond. Regards, Brian >
