Re: [systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?

[Alvin Šipraga]

On 11/19/21 12:52, Paul Menzel wrote:
> 
>> Hmm, looks like I might just have to wait a bit until we can step up to
>> 5.11.  We just stepped up to 5.10.69.  IMX8 Freescale yocto lags a bit.
>> And too bleeding edge hurts.
> 
> Well, I know opinions differ on that, but in this case, it obviously 
> hurt that you use an old Linux kernel, as the problem you have might be 
> already fixed it current Linux kernels.
> 
> It’d be great if you verified, that the problem is fixed in Linux 5.15, 
> which is also an LTS series, and then maybe get the commits in questions 
> into the 5.10.x stable series.

I don't think the commits in question will be suitable for stable, since 
they are not bugfixes as such. DSA still works just fine without them.

But yes, staying up-to-date on kernel releases is always salient advice. :-)

Kind regards,

Alvin

Re: [systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?

[Paul Menzel]

Dear Brian,


Am 19.11.21 um 00:34 schrieb Brian Hutchinson:


On Thu, Nov 18, 2021 at 5:48 PM Alvin Šipraga wrote:


On 11/18/21 23:25, Brian Hutchinson wrote:



On Thu, Nov 18, 2021 at 4:20 PM Alvin Šipraga wrote:



 On 11/18/21 01:20, Brian Hutchinson wrote:


[…]


I don't think that will work either.  eth0 has to be up and stay up or
DSA driver won't work at all.  eth0 has to be up or the slaves can't be
added to the bond.


This use-case is also addressed in the newer kernels (>=5.11), see the
below commit. It is not only when a user port is brought up (as I
summarized it), but rather when it is opened.

commit 9d5ef190e5615a7b63af89f88c4106a5bc127974
Author: Vladimir Oltean 
Date:   Fri Feb 5 15:37:10 2021 +0200

  net: dsa: automatically bring up DSA master when opening user port


[…]


Hmm, looks like I might just have to wait a bit until we can step up to
5.11.  We just stepped up to 5.10.69.  IMX8 Freescale yocto lags a bit.
And too bleeding edge hurts.


Well, I know opinions differ on that, but in this case, it obviously 
hurt that you use an old Linux kernel, as the problem you have might be 
already fixed it current Linux kernels.


It’d be great if you verified, that the problem is fixed in Linux 5.15, 
which is also an LTS series, and then maybe get the commits in questions 
into the 5.10.x stable series.


[…]


Kind regards,

Paul

Re: [systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?

[Brian Hutchinson]

Hi Alvin,

On Thu, Nov 18, 2021 at 5:48 PM Alvin Šipraga  wrote:

> On 11/18/21 23:25, Brian Hutchinson wrote:
> > Hi Alvin,
> >
> >
> > On Thu, Nov 18, 2021 at 4:20 PM Alvin Šipraga  > > wrote:
> >
> > Hi Brian,
> >
> > On 11/18/21 01:20, Brian Hutchinson wrote:
> >   > Yet another update, I was able to get it working .. but feel
> > like it is
> >   > a hack so comments welcome ... see below:
> >   >
> >
> > 
> >
> >   >
> >   > I tried and tried to get eth0 to come up before the bond was
> brought
> >   > up.  I had everything named in lexical order but didn't appear to
> >   > matter.  I added a eth0.network file and in it specified
> >   > |ActivationPolicy=|always-up and other things but could not get
> > eth0 to
> >   > come up.
> >
> > What kernel version are you using? Since Linux 5.11 we have the
> >
> >
> > I'm using linux-fslc-imx 5.10.69
> >
> > following two changes ([1] and [2]) which should automatically bring
> > up/down the master (eth0) whenever user ports (lan1, lan2) are
> brought
> > up/down. Please confirm whether or not you are using 5.11 or later.
> >
> >
> > I don't think that will work either.  eth0 has to be up and stay up or
> > DSA driver won't work at all.  eth0 has to be up or the slaves can't be
> > added to the bond.
>
> This use-case is also addressed in the newer kernels (>=5.11), see the
> below commit. It is not only when a user port is brought up (as I
> summarized it), but rather when it is opened.
>
> commit 9d5ef190e5615a7b63af89f88c4106a5bc127974
> Author: Vladimir Oltean 
> Date:   Fri Feb 5 15:37:10 2021 +0200
>
>  net: dsa: automatically bring up DSA master when opening user port
>
>  DSA wants the master interface to be open before the user port is
> due to
>  historical reasons. The promiscuity of interfaces that are down used
> to
>  have issues, as referenced Lennert Buytenhek in commit df02c6ff2e39
>  ("dsa: fix master interface allmulti/promisc handling").
>
>  The bugfix mentioned there, commit b6c40d68ff64 ("net: only invoke
>  dev->change_rx_flags when device is UP"), was basically a "don't do
>  that" approach to working around the promiscuity while down issue.
>
>  Further work done by Vlad Yasevich in commit d2615bf45069 ("net: core:
>  Always propagate flag changes to interfaces") has resolved the
>  underlying issue, and it is strictly up to the DSA and 8021q drivers
>  now, it is no longer mandated by the networking core that the master
>  interface must be up when changing its promiscuity.
>
>  From DSA's point of view, deciding to error out in dsa_slave_open
>  because the master isn't up is
>  (a) a bad user experience and
>  (b) knocking at an open door.
>  Even if there still was an issue with promiscuity while down, DSA
> could
>  still just open the master and avoid it.
>
>  Doing it this way has the additional benefit that user space can now
>  remove DSA-specific workarounds, like systemd-networkd with
> BindCarrier:
>  https://github.com/systemd/systemd/issues/7478
>
>  And we can finally remove one of the 2 bullets in the "Common pitfalls
>  using DSA setups" chapter.
>
>  Tested with two cascaded DSA switches:
>
>  $ ip link set sw0p2 up
>  fsl_enetc :00:00.2 eno2: configuring for fixed/internal link mode
>  fsl_enetc :00:00.2 eno2: Link is Up - 1Gbps/Full - flow control
> rx/tx
>  mscc_felix :00:00.5 swp0: configuring for fixed/sgmii link mode
>  mscc_felix :00:00.5 swp0: Link is Up - 1Gbps/Full - flow
> control off
>  8021q: adding VLAN 0 to HW filter on device swp0
>  sja1105 spi2.0 sw0p2: configuring for phy/rgmii-id link mode
>  IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready
>  IPv6: ADDRCONF(NETDEV_CHANGE): swp0: link becomes ready
>
>  Signed-off-by: Vladimir Oltean 
>  Reviewed-by: Andrew Lunn 
>  Reviewed-by: Florian Fainelli 
>  Signed-off-by: Jakub Kicinski 
>
>
Hmm, looks like I might just have to wait a bit until we can step up to
5.11.  We just stepped up to 5.10.69.  IMX8 Freescale yocto lags a bit.
And too bleeding edge hurts.


> >
> > If you look back up the thread where I'm doing the commands manually
> > (I'll re-copy below) ... the first thing is to bring eth0 up.  If that
> > doesn't happen, nothing else works.
>
> I forgot about the bond in my reply. Does it work if you do
> BindCarrier=eth0 from the bond's .network file? You might be out of luck
>

No.  Trying it in my 10-bond1.network doesn't work either :(

depending on how networkd implements BindCarrier=, but hopefully
> something like that works. Remember also that you need a .network file
> covering eth0, even though it's just a conduit and you don't want to
>

So I tried having a pretty bare 05-eth0.network file and tried everything I
could think of to get eth0 up with systemd and 

Re: [systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?

[Alvin Šipraga]

On 11/18/21 23:25, Brian Hutchinson wrote:
> Hi Alvin,
> 
> 
> On Thu, Nov 18, 2021 at 4:20 PM Alvin Šipraga  > wrote:
> 
> Hi Brian,
> 
> On 11/18/21 01:20, Brian Hutchinson wrote:
>   > Yet another update, I was able to get it working .. but feel
> like it is
>   > a hack so comments welcome ... see below:
>   >
> 
> 
> 
>   >
>   > I tried and tried to get eth0 to come up before the bond was brought
>   > up.  I had everything named in lexical order but didn't appear to
>   > matter.  I added a eth0.network file and in it specified
>   > |ActivationPolicy=|always-up and other things but could not get
> eth0 to
>   > come up.
> 
> What kernel version are you using? Since Linux 5.11 we have the
> 
> 
> I'm using linux-fslc-imx 5.10.69
> 
> following two changes ([1] and [2]) which should automatically bring
> up/down the master (eth0) whenever user ports (lan1, lan2) are brought
> up/down. Please confirm whether or not you are using 5.11 or later.
> 
> 
> I don't think that will work either.  eth0 has to be up and stay up or 
> DSA driver won't work at all.  eth0 has to be up or the slaves can't be 
> added to the bond.

This use-case is also addressed in the newer kernels (>=5.11), see the 
below commit. It is not only when a user port is brought up (as I 
summarized it), but rather when it is opened.

commit 9d5ef190e5615a7b63af89f88c4106a5bc127974
Author: Vladimir Oltean 
Date:   Fri Feb 5 15:37:10 2021 +0200

 net: dsa: automatically bring up DSA master when opening user port

 DSA wants the master interface to be open before the user port is 
due to
 historical reasons. The promiscuity of interfaces that are down used to
 have issues, as referenced Lennert Buytenhek in commit df02c6ff2e39
 ("dsa: fix master interface allmulti/promisc handling").

 The bugfix mentioned there, commit b6c40d68ff64 ("net: only invoke
 dev->change_rx_flags when device is UP"), was basically a "don't do
 that" approach to working around the promiscuity while down issue.

 Further work done by Vlad Yasevich in commit d2615bf45069 ("net: core:
 Always propagate flag changes to interfaces") has resolved the
 underlying issue, and it is strictly up to the DSA and 8021q drivers
 now, it is no longer mandated by the networking core that the master
 interface must be up when changing its promiscuity.

 From DSA's point of view, deciding to error out in dsa_slave_open
 because the master isn't up is
 (a) a bad user experience and
 (b) knocking at an open door.
 Even if there still was an issue with promiscuity while down, DSA could
 still just open the master and avoid it.

 Doing it this way has the additional benefit that user space can now
 remove DSA-specific workarounds, like systemd-networkd with 
BindCarrier:
 https://github.com/systemd/systemd/issues/7478

 And we can finally remove one of the 2 bullets in the "Common pitfalls
 using DSA setups" chapter.

 Tested with two cascaded DSA switches:

 $ ip link set sw0p2 up
 fsl_enetc :00:00.2 eno2: configuring for fixed/internal link mode
 fsl_enetc :00:00.2 eno2: Link is Up - 1Gbps/Full - flow control 
rx/tx
 mscc_felix :00:00.5 swp0: configuring for fixed/sgmii link mode
 mscc_felix :00:00.5 swp0: Link is Up - 1Gbps/Full - flow 
control off
 8021q: adding VLAN 0 to HW filter on device swp0
 sja1105 spi2.0 sw0p2: configuring for phy/rgmii-id link mode
 IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready
 IPv6: ADDRCONF(NETDEV_CHANGE): swp0: link becomes ready

 Signed-off-by: Vladimir Oltean 
 Reviewed-by: Andrew Lunn 
 Reviewed-by: Florian Fainelli 
 Signed-off-by: Jakub Kicinski 


> 
> If you look back up the thread where I'm doing the commands manually 
> (I'll re-copy below) ... the first thing is to bring eth0 up.  If that 
> doesn't happen, nothing else works.

I forgot about the bond in my reply. Does it work if you do 
BindCarrier=eth0 from the bond's .network file? You might be out of luck 
depending on how networkd implements BindCarrier=, but hopefully 
something like that works. Remember also that you need a .network file 
covering eth0, even though it's just a conduit and you don't want to 
give it an IP. You can do that by omitting the [Network] section and 
just having a [Match] on eth0.

> 
> These are the manual steps that work.  This is what I'm trying to 
> automate the "systemd way".  I read that "bond0" is somehow reserved so 
> my systemd.netdev and .network files use bond1.
> 
> #!/bin/bash
> 
> # Create a redundant bond between ksz9567 DSA lan1 and lan2 interfaces
> 
> # Load bonding kernel module
> modprobe bonding
> 
> # Bring up CPU interface (cpu to switch port 7 - the RGMII link)
> ip link set eth0 up
> 
> # Create a bond
> echo +bond0 > /sys/class/net/bonding_masters
> 
> # 

Re: [systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?

[Brian Hutchinson]

Hi Alvin,


On Thu, Nov 18, 2021 at 4:20 PM Alvin Šipraga  wrote:

> Hi Brian,
>
> On 11/18/21 01:20, Brian Hutchinson wrote:
>  > Yet another update, I was able to get it working .. but feel like it is
>  > a hack so comments welcome ... see below:
>  >
>
> 
>
>  >
>  > I tried and tried to get eth0 to come up before the bond was brought
>  > up.  I had everything named in lexical order but didn't appear to
>  > matter.  I added a eth0.network file and in it specified
>  > |ActivationPolicy=|always-up and other things but could not get eth0 to
>  > come up.
>
> What kernel version are you using? Since Linux 5.11 we have the
>

I'm using linux-fslc-imx 5.10.69

following two changes ([1] and [2]) which should automatically bring
> up/down the master (eth0) whenever user ports (lan1, lan2) are brought
> up/down. Please confirm whether or not you are using 5.11 or later.
>

I don't think that will work either.  eth0 has to be up and stay up or DSA
driver won't work at all.  eth0 has to be up or the slaves can't be added
to the bond.

If you look back up the thread where I'm doing the commands manually (I'll
re-copy below) ... the first thing is to bring eth0 up.  If that doesn't
happen, nothing else works.

These are the manual steps that work.  This is what I'm trying to automate
the "systemd way".  I read that "bond0" is somehow reserved so my
systemd.netdev and .network files use bond1.

#!/bin/bash

# Create a redundant bond between ksz9567 DSA lan1 and lan2 interfaces

# Load bonding kernel module
modprobe bonding

# Bring up CPU interface (cpu to switch port 7 - the RGMII link)
ip link set eth0 up

# Create a bond
echo +bond0 > /sys/class/net/bonding_masters

# Set mode to active-backup (redundancy failover)
echo active-backup > /sys/class/net/bond0/bonding/mode

# Set time it takes (in ms) for slave to move when a link goes down
echo 1000 > /sys/class/net/bond0/bonding/miimon

# Add slaves to bond

echo +lan1 > /sys/class/net/bond0/bonding/slaves
echo +lan2 > /sys/class/net/bond0/bonding/slaves

# Set IP and netmask of the bond
ip addr add 192.168.0.4/24 dev bond0

# And bring bond up.  Pings and network connectivity should work now
ip link set bond0 up

# For a board that doesn't have Ethernet switch hardware strapped to enable
at boot .. enable it now
i2cset -f -y 0 0x5f 0x03 0x00 0x01 i


> If you are using an older kernel, and systemd is version 243 or later,
>

I checked and I'm using 244.5

you can use the BindCarrier= setting (see [1]) in the [Network] section
> of your .network file(s) for lan1 and lan2. Something like this:
>
> [Match]
> Name=lan*
>
> [Network]
> BindCarrier=eth0
>

I added the BindCarrier=eth0 in both of my DSA lan1 and lan2 .network
files.  I got the same result as before ... dmesg reports lan1 and lan2
slaves are brought up before eth0 ... and so they fail.  The only thing so
far that works is to bring eth0 up with a network-pre.target service.  And
I'm doing the i2c command to enable my switch in a
After=network-online.target service.

[4.367802] bond1: (slave lan2): Opening slave failed
[4.467561] bond1: (slave lan1): Opening slave failed
[4.640914] fec 30be.ethernet eth0: Link is Up - 1Gbps/Full - flow
control off
[5.104983] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Regards,

Brian

Re: [systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?

[Alvin Šipraga]

Hi Brian,

On 11/18/21 01:20, Brian Hutchinson wrote:
 > Yet another update, I was able to get it working .. but feel like it is
 > a hack so comments welcome ... see below:
 >



 >
 > I tried and tried to get eth0 to come up before the bond was brought
 > up.  I had everything named in lexical order but didn't appear to
 > matter.  I added a eth0.network file and in it specified
 > |ActivationPolicy=|always-up and other things but could not get eth0 to
 > come up.

What kernel version are you using? Since Linux 5.11 we have the 
following two changes ([1] and [2]) which should automatically bring 
up/down the master (eth0) whenever user ports (lan1, lan2) are brought 
up/down. Please confirm whether or not you are using 5.11 or later.

If you are using an older kernel, and systemd is version 243 or later, 
you can use the BindCarrier= setting (see [1]) in the [Network] section 
of your .network file(s) for lan1 and lan2. Something like this:

[Match]
Name=lan*

[Network]
BindCarrier=eth0

See also [3].

[1] 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9d5ef190e5615a7b63af89f88c4106a5bc127974
[2] 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c0a8a9c274936543e436aef691499304ce3127dc
[3] 
https://www.freedesktop.org/software/systemd/man/systemd.network.html#BindCarrier=
[4] https://github.com/systemd/systemd/issues/7478#issuecomment-514562190

If both your kernel and systemd are old, I think you'll have to use 
something like your hack.

One more thing:

 > [Service]
 > ExecStart=/usr/local/bin/eth0-up.sh
 > RemainAfterExit=yes
 >
 > [Install]
 > WantedBy=multi-user.target
 >
 > cat /usr/local/bin/eth0-up.sh
 > #!/bin/bash
 > ip link set eth0 up

You don't need a separate shell script, you can just invoke the ip 
command directly:

[Service]
ExecStart=/usr/bin/ip link set eth0 up
...


Kind regards,

Alvin

Re: [systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?

[Brian Hutchinson]

Hi Andrei,

On Thu, Nov 18, 2021, 12:13 AM Andrei Borzenkov  wrote:

> On 18.11.2021 03:20, Brian Hutchinson wrote:
> > Yet another update, I was able to get it working .. but feel like it is a
> > hack so comments welcome ... see below:
> >
> > On Wed, Nov 17, 2021 at 12:26 AM Brian Hutchinson 
> > wrote:
> >
> >> Update below
> >>
> >> On Tue, Nov 16, 2021 at 2:27 PM Brian Hutchinson 
> >> wrote:
> >>
> >>> Hi Mikulėnas,
> >>>
> >>> On Tue, Nov 16, 2021, 3:12 AM Mantas Mikulėnas 
> wrote:
> >>>
>  Most of this looks like it could be done with systemd-networkd to
> create
>  a bond .netdev, with a small oneshot service for i2c. (What's the
> exact
>  criteria for when it should be run? Does it depend on bond0 being
> there,
>  does it need to be last, etc?)
> 
> >>>
> >>> It can be last in the startup chain I guess, don't know what other
> >>> systemd things that might need the network to be up before the last
> unit
> >>> file runs.
> >>>
> >>> I start linuxptp too so I would have the unit file that starts ptp4l
> >>> start after the bond was created etc.
> >>>
> >>> Same thing for the i2c command to enable the switch.
> >>>
> >>> Regards,
> >>>
> >>> Brian
> >>>
> >>>
>  On Tue, Nov 16, 2021, 02:58 Brian Hutchinson 
>  wrote:
> 
> > Hi,
> >
> > I'm on a IMX8 platform and have a Microchip KSZ9567 Ethernet
> switch.  I
> > can use IP commands to manually bring lan1 and lan2 interfaces up
> and then
> > create a redundant/failover bond ... but I'm having difficulty
> figuring out
> > how to do this the "systemd" way.
> >
> > My first attempt was to just have systemd run a script of all the
> > commands I do manually but during system startup there appears to be
> race
> > conditions so I have to set my service type to "Idle" and sometimes
> even
> > that doesn't work. So I want to exploit any systemd support for DSA
> and
> > bonding.
> >
> > Here is script my manual steps which is what I want systemd to
> > ultimately do:
> >
> > #!/bin/bash
> >
> > # Create a redundant bond between ksz9567 DSA lan1 and lan2
> interfaces
> >
> > # Load bonding kernel module
> > modprobe bonding
> >
> > # Bring up CPU interface (cpu to switch port 7 - the RGMII link)
> > ip link set eth0 up
> >
> > # Create a bond
> > echo +bond0 > /sys/class/net/bonding_masters
> >
> > # Set mode to active-backup (redundancy failover)
> > echo active-backup > /sys/class/net/bond0/bonding/mode
> >
> > # Set time it takes (in ms) for slave to move when a link goes down
> > echo 1000 > /sys/class/net/bond0/bonding/miimon
> >
> > # Add slaves to bond
> >
> > echo +lan1 > /sys/class/net/bond0/bonding/slaves
> > echo +lan2 > /sys/class/net/bond0/bonding/slaves
> >
> > # Set IP and netmask of the bond
> > ip addr add 192.168.0.4/24 dev bond0
> >
> > # And bring bond up.  Pings and network connectivity should work now
> > ip link set bond0 up
> >
> > # For a board that doesn't have Ethernet switch hardware strapped to
> > enable at boot .. enable it now
> > i2cset -f -y 0 0x5f 0x03 0x00 0x01 i
> >
> > Thanks for any information, pointers etc.
> >
> > Regards,
> >
> > Brian
> >
> 
> >> So not sure I'm doing this right.  eth0 needs to be up before lan1 and
> >> lan2 can be added to the bond.  Not quite sure how to do that with
> systemd
> >> but I made the following files and see some progress but ping doesn't
> work
> >> so appears I have no network connectivity:
> >>
> >> root@imx8mmevk:/etc/systemd/network# cat 10-bond1.netdev
> >> [NetDev]
> >> Name=bond1
> >> Kind=bond
> >>
> >> [Bond]
> >> Mode=active-backup
> >> PrimaryReselectPolicy=failure
> >> MIIMonitorSec=2s
> >>
> >> root@imx8mmevk:/etc/systemd/network# cat 10-bond1.network
> >> [Match]
> >> Name=bond1
> >>
> >> [Network]
> >> Address=192.168.0.4/24
> >>
> >> root@imx8mmevk:/etc/systemd/network# cat 20-lan1.network
> >> [Match]
> >> Name=lan1
> >>
> >> [Network]
> >> Bond=bond1
> >> PrimarySlave=true
> >>
> >> root@imx8mmevk:/etc/systemd/network# cat 30-lan2.network
> >>
> >> [Match]
> >> Name=lan2
> >>
> >> [Network]
> >> Bond=bond
> >>
> >> ip link list
> >> 1: lo:  mtu 65536 qdisc noqueue state UNKNOWN mode
> >> DEFAULT group default qlen 1000
> >>link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> >> 2: eth0:  mtu 1506 qdisc mq state UP
> mode
> >> DEFAULT group default qlen 1000
> >>link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
> >> 3: lan1@eth0:  mtu 1500 qdisc noop state DOWN mode
> >> DEFAULT group default qlen 1000
> >>link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
> >> 4: lan2@eth0:  mtu 1500 qdisc noop state DOWN mode
> >> DEFAULT group default qlen 1000
> >>link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
> >> 5: bond1:  mtu 1500 qdisc
> >> noqueue state DOWN mode DEFAULT group default 

Re: [systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?

[Andrei Borzenkov]

On 18.11.2021 03:20, Brian Hutchinson wrote:
> Yet another update, I was able to get it working .. but feel like it is a
> hack so comments welcome ... see below:
> 
> On Wed, Nov 17, 2021 at 12:26 AM Brian Hutchinson 
> wrote:
> 
>> Update below
>>
>> On Tue, Nov 16, 2021 at 2:27 PM Brian Hutchinson 
>> wrote:
>>
>>> Hi Mikulėnas,
>>>
>>> On Tue, Nov 16, 2021, 3:12 AM Mantas Mikulėnas  wrote:
>>>
 Most of this looks like it could be done with systemd-networkd to create
 a bond .netdev, with a small oneshot service for i2c. (What's the exact
 criteria for when it should be run? Does it depend on bond0 being there,
 does it need to be last, etc?)

>>>
>>> It can be last in the startup chain I guess, don't know what other
>>> systemd things that might need the network to be up before the last unit
>>> file runs.
>>>
>>> I start linuxptp too so I would have the unit file that starts ptp4l
>>> start after the bond was created etc.
>>>
>>> Same thing for the i2c command to enable the switch.
>>>
>>> Regards,
>>>
>>> Brian
>>>
>>>
 On Tue, Nov 16, 2021, 02:58 Brian Hutchinson 
 wrote:

> Hi,
>
> I'm on a IMX8 platform and have a Microchip KSZ9567 Ethernet switch.  I
> can use IP commands to manually bring lan1 and lan2 interfaces up and then
> create a redundant/failover bond ... but I'm having difficulty figuring 
> out
> how to do this the "systemd" way.
>
> My first attempt was to just have systemd run a script of all the
> commands I do manually but during system startup there appears to be race
> conditions so I have to set my service type to "Idle" and sometimes even
> that doesn't work. So I want to exploit any systemd support for DSA and
> bonding.
>
> Here is script my manual steps which is what I want systemd to
> ultimately do:
>
> #!/bin/bash
>
> # Create a redundant bond between ksz9567 DSA lan1 and lan2 interfaces
>
> # Load bonding kernel module
> modprobe bonding
>
> # Bring up CPU interface (cpu to switch port 7 - the RGMII link)
> ip link set eth0 up
>
> # Create a bond
> echo +bond0 > /sys/class/net/bonding_masters
>
> # Set mode to active-backup (redundancy failover)
> echo active-backup > /sys/class/net/bond0/bonding/mode
>
> # Set time it takes (in ms) for slave to move when a link goes down
> echo 1000 > /sys/class/net/bond0/bonding/miimon
>
> # Add slaves to bond
>
> echo +lan1 > /sys/class/net/bond0/bonding/slaves
> echo +lan2 > /sys/class/net/bond0/bonding/slaves
>
> # Set IP and netmask of the bond
> ip addr add 192.168.0.4/24 dev bond0
>
> # And bring bond up.  Pings and network connectivity should work now
> ip link set bond0 up
>
> # For a board that doesn't have Ethernet switch hardware strapped to
> enable at boot .. enable it now
> i2cset -f -y 0 0x5f 0x03 0x00 0x01 i
>
> Thanks for any information, pointers etc.
>
> Regards,
>
> Brian
>

>> So not sure I'm doing this right.  eth0 needs to be up before lan1 and
>> lan2 can be added to the bond.  Not quite sure how to do that with systemd
>> but I made the following files and see some progress but ping doesn't work
>> so appears I have no network connectivity:
>>
>> root@imx8mmevk:/etc/systemd/network# cat 10-bond1.netdev
>> [NetDev]
>> Name=bond1
>> Kind=bond
>>
>> [Bond]
>> Mode=active-backup
>> PrimaryReselectPolicy=failure
>> MIIMonitorSec=2s
>>
>> root@imx8mmevk:/etc/systemd/network# cat 10-bond1.network
>> [Match]
>> Name=bond1
>>
>> [Network]
>> Address=192.168.0.4/24
>>
>> root@imx8mmevk:/etc/systemd/network# cat 20-lan1.network
>> [Match]
>> Name=lan1
>>
>> [Network]
>> Bond=bond1
>> PrimarySlave=true
>>
>> root@imx8mmevk:/etc/systemd/network# cat 30-lan2.network
>>
>> [Match]
>> Name=lan2
>>
>> [Network]
>> Bond=bond
>>
>> ip link list
>> 1: lo:  mtu 65536 qdisc noqueue state UNKNOWN mode
>> DEFAULT group default qlen 1000
>>link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>> 2: eth0:  mtu 1506 qdisc mq state UP mode
>> DEFAULT group default qlen 1000
>>link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
>> 3: lan1@eth0:  mtu 1500 qdisc noop state DOWN mode
>> DEFAULT group default qlen 1000
>>link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
>> 4: lan2@eth0:  mtu 1500 qdisc noop state DOWN mode
>> DEFAULT group default qlen 1000
>>link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
>> 5: bond1:  mtu 1500 qdisc
>> noqueue state DOWN mode DEFAULT group default qlen 1000
>>link/ether be:87:0a:9b:13:03 brd ff:ff:ff:ff:ff:ff
>>
>> cat /proc/net/bonding/bond1
>> Ethernet Channel Bonding Driver: v5.10.69
>>
>> Bonding Mode: fault-tolerance (active-backup)
>> Primary Slave: None
>> Currently Active Slave: None
>> MII Status: down
>> MII Polling Interval (ms): 2000
>> Up Delay (ms): 0
>> Down Delay (ms): 0
>> Peer Notification Delay (ms): 0
>>

Re: [systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?

[Brian Hutchinson]

Yet another update, I was able to get it working .. but feel like it is a
hack so comments welcome ... see below:

On Wed, Nov 17, 2021 at 12:26 AM Brian Hutchinson 
wrote:

> Update below
>
> On Tue, Nov 16, 2021 at 2:27 PM Brian Hutchinson 
> wrote:
>
>> Hi Mikulėnas,
>>
>> On Tue, Nov 16, 2021, 3:12 AM Mantas Mikulėnas  wrote:
>>
>>> Most of this looks like it could be done with systemd-networkd to create
>>> a bond .netdev, with a small oneshot service for i2c. (What's the exact
>>> criteria for when it should be run? Does it depend on bond0 being there,
>>> does it need to be last, etc?)
>>>
>>
>> It can be last in the startup chain I guess, don't know what other
>> systemd things that might need the network to be up before the last unit
>> file runs.
>>
>> I start linuxptp too so I would have the unit file that starts ptp4l
>> start after the bond was created etc.
>>
>> Same thing for the i2c command to enable the switch.
>>
>> Regards,
>>
>> Brian
>>
>>
>>> On Tue, Nov 16, 2021, 02:58 Brian Hutchinson 
>>> wrote:
>>>
 Hi,

 I'm on a IMX8 platform and have a Microchip KSZ9567 Ethernet switch.  I
 can use IP commands to manually bring lan1 and lan2 interfaces up and then
 create a redundant/failover bond ... but I'm having difficulty figuring out
 how to do this the "systemd" way.

 My first attempt was to just have systemd run a script of all the
 commands I do manually but during system startup there appears to be race
 conditions so I have to set my service type to "Idle" and sometimes even
 that doesn't work. So I want to exploit any systemd support for DSA and
 bonding.

 Here is script my manual steps which is what I want systemd to
 ultimately do:

 #!/bin/bash

 # Create a redundant bond between ksz9567 DSA lan1 and lan2 interfaces

 # Load bonding kernel module
 modprobe bonding

 # Bring up CPU interface (cpu to switch port 7 - the RGMII link)
 ip link set eth0 up

 # Create a bond
 echo +bond0 > /sys/class/net/bonding_masters

 # Set mode to active-backup (redundancy failover)
 echo active-backup > /sys/class/net/bond0/bonding/mode

 # Set time it takes (in ms) for slave to move when a link goes down
 echo 1000 > /sys/class/net/bond0/bonding/miimon

 # Add slaves to bond

 echo +lan1 > /sys/class/net/bond0/bonding/slaves
 echo +lan2 > /sys/class/net/bond0/bonding/slaves

 # Set IP and netmask of the bond
 ip addr add 192.168.0.4/24 dev bond0

 # And bring bond up.  Pings and network connectivity should work now
 ip link set bond0 up

 # For a board that doesn't have Ethernet switch hardware strapped to
 enable at boot .. enable it now
 i2cset -f -y 0 0x5f 0x03 0x00 0x01 i

 Thanks for any information, pointers etc.

 Regards,

 Brian

>>>
> So not sure I'm doing this right.  eth0 needs to be up before lan1 and
> lan2 can be added to the bond.  Not quite sure how to do that with systemd
> but I made the following files and see some progress but ping doesn't work
> so appears I have no network connectivity:
>
> root@imx8mmevk:/etc/systemd/network# cat 10-bond1.netdev
> [NetDev]
> Name=bond1
> Kind=bond
>
> [Bond]
> Mode=active-backup
> PrimaryReselectPolicy=failure
> MIIMonitorSec=2s
>
> root@imx8mmevk:/etc/systemd/network# cat 10-bond1.network
> [Match]
> Name=bond1
>
> [Network]
> Address=192.168.0.4/24
>
> root@imx8mmevk:/etc/systemd/network# cat 20-lan1.network
> [Match]
> Name=lan1
>
> [Network]
> Bond=bond1
> PrimarySlave=true
>
> root@imx8mmevk:/etc/systemd/network# cat 30-lan2.network
>
> [Match]
> Name=lan2
>
> [Network]
> Bond=bond
>
> ip link list
> 1: lo:  mtu 65536 qdisc noqueue state UNKNOWN mode
> DEFAULT group default qlen 1000
>link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> 2: eth0:  mtu 1506 qdisc mq state UP mode
> DEFAULT group default qlen 1000
>link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
> 3: lan1@eth0:  mtu 1500 qdisc noop state DOWN mode
> DEFAULT group default qlen 1000
>link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
> 4: lan2@eth0:  mtu 1500 qdisc noop state DOWN mode
> DEFAULT group default qlen 1000
>link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
> 5: bond1:  mtu 1500 qdisc
> noqueue state DOWN mode DEFAULT group default qlen 1000
>link/ether be:87:0a:9b:13:03 brd ff:ff:ff:ff:ff:ff
>
> cat /proc/net/bonding/bond1
> Ethernet Channel Bonding Driver: v5.10.69
>
> Bonding Mode: fault-tolerance (active-backup)
> Primary Slave: None
> Currently Active Slave: None
> MII Status: down
> MII Polling Interval (ms): 2000
> Up Delay (ms): 0
> Down Delay (ms): 0
> Peer Notification Delay (ms): 0
>
> At this point there should be info on lan1 and lan2 status but don't see
> it.
>
> ... so of course I can't ping.
>
> Next I did systemctl restart systemd-networkd and saw the following:
>
> systemctl restart 

Re: [systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?

[Brian Hutchinson]

Update below

On Tue, Nov 16, 2021 at 2:27 PM Brian Hutchinson 
wrote:

> Hi Mikulėnas,
>
> On Tue, Nov 16, 2021, 3:12 AM Mantas Mikulėnas  wrote:
>
>> Most of this looks like it could be done with systemd-networkd to create
>> a bond .netdev, with a small oneshot service for i2c. (What's the exact
>> criteria for when it should be run? Does it depend on bond0 being there,
>> does it need to be last, etc?)
>>
>
> It can be last in the startup chain I guess, don't know what other systemd
> things that might need the network to be up before the last unit file runs.
>
> I start linuxptp too so I would have the unit file that starts ptp4l start
> after the bond was created etc.
>
> Same thing for the i2c command to enable the switch.
>
> Regards,
>
> Brian
>
>
>> On Tue, Nov 16, 2021, 02:58 Brian Hutchinson 
>> wrote:
>>
>>> Hi,
>>>
>>> I'm on a IMX8 platform and have a Microchip KSZ9567 Ethernet switch.  I
>>> can use IP commands to manually bring lan1 and lan2 interfaces up and then
>>> create a redundant/failover bond ... but I'm having difficulty figuring out
>>> how to do this the "systemd" way.
>>>
>>> My first attempt was to just have systemd run a script of all the
>>> commands I do manually but during system startup there appears to be race
>>> conditions so I have to set my service type to "Idle" and sometimes even
>>> that doesn't work. So I want to exploit any systemd support for DSA and
>>> bonding.
>>>
>>> Here is script my manual steps which is what I want systemd to
>>> ultimately do:
>>>
>>> #!/bin/bash
>>>
>>> # Create a redundant bond between ksz9567 DSA lan1 and lan2 interfaces
>>>
>>> # Load bonding kernel module
>>> modprobe bonding
>>>
>>> # Bring up CPU interface (cpu to switch port 7 - the RGMII link)
>>> ip link set eth0 up
>>>
>>> # Create a bond
>>> echo +bond0 > /sys/class/net/bonding_masters
>>>
>>> # Set mode to active-backup (redundancy failover)
>>> echo active-backup > /sys/class/net/bond0/bonding/mode
>>>
>>> # Set time it takes (in ms) for slave to move when a link goes down
>>> echo 1000 > /sys/class/net/bond0/bonding/miimon
>>>
>>> # Add slaves to bond
>>>
>>> echo +lan1 > /sys/class/net/bond0/bonding/slaves
>>> echo +lan2 > /sys/class/net/bond0/bonding/slaves
>>>
>>> # Set IP and netmask of the bond
>>> ip addr add 192.168.0.4/24 dev bond0
>>>
>>> # And bring bond up.  Pings and network connectivity should work now
>>> ip link set bond0 up
>>>
>>> # For a board that doesn't have Ethernet switch hardware strapped to
>>> enable at boot .. enable it now
>>> i2cset -f -y 0 0x5f 0x03 0x00 0x01 i
>>>
>>> Thanks for any information, pointers etc.
>>>
>>> Regards,
>>>
>>> Brian
>>>
>>
So not sure I'm doing this right.  eth0 needs to be up before lan1 and lan2
can be added to the bond.  Not quite sure how to do that with systemd but I
made the following files and see some progress but ping doesn't work so
appears I have no network connectivity:

root@imx8mmevk:/etc/systemd/network# cat 10-bond1.netdev
[NetDev]
Name=bond1
Kind=bond

[Bond]
Mode=active-backup
PrimaryReselectPolicy=failure
MIIMonitorSec=2s

root@imx8mmevk:/etc/systemd/network# cat 10-bond1.network
[Match]
Name=bond1

[Network]
Address=192.168.0.4/24

root@imx8mmevk:/etc/systemd/network# cat 20-lan1.network
[Match]
Name=lan1

[Network]
Bond=bond1
PrimarySlave=true

root@imx8mmevk:/etc/systemd/network# cat 30-lan2.network

[Match]
Name=lan2

[Network]
Bond=bond

ip link list
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN mode
DEFAULT group default qlen 1000
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0:  mtu 1506 qdisc mq state UP mode
DEFAULT group default qlen 1000
   link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
3: lan1@eth0:  mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
   link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
4: lan2@eth0:  mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
   link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
5: bond1:  mtu 1500 qdisc noqueue
state DOWN mode DEFAULT group default qlen 1000
   link/ether be:87:0a:9b:13:03 brd ff:ff:ff:ff:ff:ff

cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v5.10.69

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: None
MII Status: down
MII Polling Interval (ms): 2000
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

At this point there should be info on lan1 and lan2 status but don't see it.

... so of course I can't ping.

Next I did systemctl restart systemd-networkd and saw the following:

systemctl restart systemd-networkd
root@imx8mmevk:~# [   33.816313] device eth0 entered promiscuous mode
[   33.821026] audit: type=1700 audit(1636550966.339:2): dev=eth0 prom=256
old_prom=0 auid=4294967295 uid=995 gid=994 ses=4294967295
[   33.867164] ksz9477-switch 0-005f lan2: configuring for phy/gmii link
mode
[   33.875066] bond1: (slave lan2): Enslaving as a backup interface with a
down link
[   33.919055] ksz9477-switch 

Re: [systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?

[Brian Hutchinson]

Hi Mikulėnas,

On Tue, Nov 16, 2021, 3:12 AM Mantas Mikulėnas  wrote:

> Most of this looks like it could be done with systemd-networkd to create a
> bond .netdev, with a small oneshot service for i2c. (What's the exact
> criteria for when it should be run? Does it depend on bond0 being there,
> does it need to be last, etc?)
>

It can be last in the startup chain I guess, don't know what other systemd
things that might need the network to be up before the last unit file runs.

I start linuxptp too so I would have the unit file that starts ptp4l start
after the bond was created etc.

Same thing for the i2c command to enable the switch.

Regards,

Brian


> On Tue, Nov 16, 2021, 02:58 Brian Hutchinson  wrote:
>
>> Hi,
>>
>> I'm on a IMX8 platform and have a Microchip KSZ9567 Ethernet switch.  I
>> can use IP commands to manually bring lan1 and lan2 interfaces up and then
>> create a redundant/failover bond ... but I'm having difficulty figuring out
>> how to do this the "systemd" way.
>>
>> My first attempt was to just have systemd run a script of all the
>> commands I do manually but during system startup there appears to be race
>> conditions so I have to set my service type to "Idle" and sometimes even
>> that doesn't work. So I want to exploit any systemd support for DSA and
>> bonding.
>>
>> Here is script my manual steps which is what I want systemd to ultimately
>> do:
>>
>> #!/bin/bash
>>
>> # Create a redundant bond between ksz9567 DSA lan1 and lan2 interfaces
>>
>> # Load bonding kernel module
>> modprobe bonding
>>
>> # Bring up CPU interface (cpu to switch port 7 - the RGMII link)
>> ip link set eth0 up
>>
>> # Create a bond
>> echo +bond0 > /sys/class/net/bonding_masters
>>
>> # Set mode to active-backup (redundancy failover)
>> echo active-backup > /sys/class/net/bond0/bonding/mode
>>
>> # Set time it takes (in ms) for slave to move when a link goes down
>> echo 1000 > /sys/class/net/bond0/bonding/miimon
>>
>> # Add slaves to bond
>>
>> echo +lan1 > /sys/class/net/bond0/bonding/slaves
>> echo +lan2 > /sys/class/net/bond0/bonding/slaves
>>
>> # Set IP and netmask of the bond
>> ip addr add 192.168.0.4/24 dev bond0
>>
>> # And bring bond up.  Pings and network connectivity should work now
>> ip link set bond0 up
>>
>> # For a board that doesn't have Ethernet switch hardware strapped to
>> enable at boot .. enable it now
>> i2cset -f -y 0 0x5f 0x03 0x00 0x01 i
>>
>> Thanks for any information, pointers etc.
>>
>> Regards,
>>
>> Brian
>>
>

Re: [systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?

[Mantas Mikulėnas]

Most of this looks like it could be done with systemd-networkd to create a
bond .netdev, with a small oneshot service for i2c. (What's the exact
criteria for when it should be run? Does it depend on bond0 being there,
does it need to be last, etc?)

On Tue, Nov 16, 2021, 02:58 Brian Hutchinson  wrote:

> Hi,
>
> I'm on a IMX8 platform and have a Microchip KSZ9567 Ethernet switch.  I
> can use IP commands to manually bring lan1 and lan2 interfaces up and then
> create a redundant/failover bond ... but I'm having difficulty figuring out
> how to do this the "systemd" way.
>
> My first attempt was to just have systemd run a script of all the commands
> I do manually but during system startup there appears to be race conditions
> so I have to set my service type to "Idle" and sometimes even that doesn't
> work. So I want to exploit any systemd support for DSA and bonding.
>
> Here is script my manual steps which is what I want systemd to ultimately
> do:
>
> #!/bin/bash
>
> # Create a redundant bond between ksz9567 DSA lan1 and lan2 interfaces
>
> # Load bonding kernel module
> modprobe bonding
>
> # Bring up CPU interface (cpu to switch port 7 - the RGMII link)
> ip link set eth0 up
>
> # Create a bond
> echo +bond0 > /sys/class/net/bonding_masters
>
> # Set mode to active-backup (redundancy failover)
> echo active-backup > /sys/class/net/bond0/bonding/mode
>
> # Set time it takes (in ms) for slave to move when a link goes down
> echo 1000 > /sys/class/net/bond0/bonding/miimon
>
> # Add slaves to bond
>
> echo +lan1 > /sys/class/net/bond0/bonding/slaves
> echo +lan2 > /sys/class/net/bond0/bonding/slaves
>
> # Set IP and netmask of the bond
> ip addr add 192.168.0.4/24 dev bond0
>
> # And bring bond up.  Pings and network connectivity should work now
> ip link set bond0 up
>
> # For a board that doesn't have Ethernet switch hardware strapped to
> enable at boot .. enable it now
> i2cset -f -y 0 0x5f 0x03 0x00 0x01 i
>
> Thanks for any information, pointers etc.
>
> Regards,
>
> Brian
>