Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On Thu, 2022-05-12 at 18:41 +0200, Lennart Poettering wrote: > On Mo, 09.05.22 19:27, Zbigniew Jędrzejewski-Szmek > (zbys...@in.waw.pl) wrote: > > > FWIW, I still think it's a better _default_. The patch that finally > > introduced this was my patch [1], so I'm obviously biased… Some > > more > > considerations: > > I agree with this. > > Finding good defaults is always difficult, but I must say stability > and predictability is a property I like above a lot of other stuff. I > understand that in plenty environments it's important not to add new > MAC addresses to the mix, but it's impossible to know in which > environment we are. Most devices are created by tools that intentionally choose the MAC address. So this primarily matters for simpler tools. The predictability comes at the expense of introducing races. Races that theoretically already exist when you don't wait for udev to complete initialization. But which in practice are not be harmful, because udev does not do much of relevance for software devices (MacAddressPolicy= aside). I am surprised that this race doesn't bother more people. It seems more severe to me than the bond/bridge problem that Dusty is about. Because that is just an undesired configuration, that can be changed. > > So, either way, some people will always be unhappy with the defaults > we pick. Changing defaults makes sense if it's highly likely that the > vast majority of users would benefit from the new default. But I > don't > see that here... And changing defaults comes at a price, because it > will break people's setup. We made a change once here, but I wouldn't > use that as excuse to change it again... Maybe we can just accept that RHEL8/9/10 won't have this behavior. After all, not every default may be best for very distro. In case of CoreOS, maybe that applies to some Fedora spins as well. > > So, I am not convinced. > > What makes me wonder about all of this: we are talking about > synthetic > devices, which means some tool is used to create them. If those tools > prefer a different MAC policy, why don't they drop in a .link file > that matches against the devices that specific software creates? NetworkManager doesn't mind. NetworkManager sets a MAC address (or doesn't) and udev too. Everything happens according to configuration. NetworkManager won't fight udev's configuration, or try to overrule it. It's the user's choice... Except, this choice here comes from the distro and seems to cause problems for some users. If we really wanted to overule this, we would install a 98-nm- default.conf, and don't bother with ID_NM_OWNED_AND_OPERATED=1. That essentially happens on RHEL already, by patching 99-default.conf. > > i.e. let's say NM prefers to use a different MAC policy: they could > drop in a udev rules file that adds some udev property onto the > network devices they manage (i.e. invoke a callout binary, or do a > TEST check or so which checks the iface against some NM state, and > then set ID_NM_OWNED_AND_OPERATED=1 or so). Then, ship a .link file > (or multiple) with a Property= match in the [Match] section, that > sets > the desired policy. ID_NM_OWNED_AND_OPERATED=1 would be nice and have general uses, outside this dicussion. If we had such a property we could tell the user to drop their own .link file to adjust the udev configuration to what they want, specially targeting NetworkManager devices. But it's not clear they really need this. Link files already have plenty of matching capabilities, so they probably could sufficiently just match on the name. Nice feature in theory, but the actual usecases so far were not strong enough to implement it. Patch welcome! > > With such a logic, NM could make its own choices on MAC policy, but > the default systemd policy wouldn't have to change. > > Also, afaik OSes that run in clouds all have some tool like cloud- > init > or ignition or so, which generate .network files in /run with the > right > configuration. Why not generate .link files in /run the same way with > a MAC policy appropriate for the cloud provider? best, Thomas
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On Thu, 12 May 2022 14:36:44 -0400 Dan Streetman wrote: > On Thu, May 12, 2022 at 2:27 PM Dan Streetman wrote: > > > > On Thu, May 12, 2022 at 1:50 PM Dusty Mabe wrote: > > > > > > > > > > > > On 5/12/22 13:36, Dan Streetman wrote: > > > > On Thu, May 12, 2022 at 11:11 AM Thomas Haller > > > > wrote: > > > >> > > > >> Hi Zbyszek, > > > >> > > > >> > > > >> > > > >> I must say, I personally don't care too much. NetworkManager is fine > > > >> either way. > > > >> > > > >> There is however the problem about RHEL8/9, which patches this > > > >> downstream. I don't like that deviation, but I'd also be uncomfortable > > > >> to push that change for RHEL(10) users. > > > >> > > > >> But let me play devil's advocate here... > > > >> > > > >> > > > >> On Mon, 2022-05-09 at 19:27 +0200, Zbigniew Jędrzejewski-Szmek wrote: > > > >>> > > > >>> FWIW, I still think it's a better _default_. > > > >> > > > >> I don't agree that it's clearly better. Your arguments don't seem > > > >> strong, arguably, neither are mine. Except, that it's not clear that > > > >> this solves an actual problem, while it clearly causes problems for > > > >> some people. Just look at the referened issues from !3374. > > > >> > > > >> > > > >> Either > > > >> > > > >> - a user doesn't care about the MAC address, > > > > > > > > note that it's possible for a user not to care about the *specific* > > > > mac address, only that they want the mac to remain consistent. > > > > > > > > for example, on my system i have br0 bridge with multiple interfaces > > > > attached, and my local DHCP server is configured to provide a static > > > > addr to br0's mac. If I replace an interface card (e.g. change from 1g > > > > to 10g, or just replace a failing nic) then the br0 mac *does not* > > > > change, if using systemd's default. If I had br0 inheriting its mac > > > > from one of the attached interfaces, it would change, and i'd have to > > > > update my dhcp server config. > > > > > > > > I think the argument really comes down to, should the default be to > > > > have (without user configuration) a mac that is predictable or a mac > > > > that is consistent. Your argument is for predictability, which makes > > > > the field engineer's work deploying systems easier, but if you lose > > > > consistency then the life of the maintenance engineer gets harder. > > > > > > Interested discussion. Let's take it a bit further. > > > > > > On your system how did your DHCP server get configured to provide an > > > addr to br0's MAC? You had to install the OS, and create br0 first > > > before you even knew the MAC to tell the network admin what the MAC > > > was. Now you're golden, the MAC will never change for the lifetime > > > of that OS install even if someone replaces a NIC, but it wasn't a great > > > first experience really. > > > > No, but the alternative is for me to manually find the mac address > > labels on my nics and use those to pre-configure my DHCP server. I > > understand for large deployments those are typically provided in a > > spreadsheet, but that isn't the case for me, and my approach (i.e. > > simply check the mac once the system is installed) was *much* easier > > and more reliable. > > > > And this doesn't even get into the nuances of figuring out *which* nic > > mac the bridge/bond will get, which won't be obvious to everyone. Note > > that the answer here *is not* that the bridge gets the mac of the > > 'first' interface added to it; the bridge gets the mac of the *lowest* > > mac that is currently attached to it. And yes, this does mean that if > > you use this behavior (of a bridge getting its mac from the attached > > interfaces), the bridge's mac *can change while it is up and running > > with an address*. Try it out yourself ;-) > > And to clarify - bond interfaces do 'keep' the mac of the 'first' > interface added - technically, the bond sets all interfaces added > later to have the same mac. So this is even less 'predictable' than a > bridge, where you can predict which of the 2 or more macs are > 'lowest', a bond mac would just be whatever the 'first' added > interface is. I remember that behavior was documented in one of the 802 standards related to spanning tree. The bridge advertises based on its MAC address.
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On Thu, May 12, 2022 at 2:27 PM Dan Streetman wrote: > > On Thu, May 12, 2022 at 1:50 PM Dusty Mabe wrote: > > > > > > > > On 5/12/22 13:36, Dan Streetman wrote: > > > On Thu, May 12, 2022 at 11:11 AM Thomas Haller wrote: > > >> > > >> Hi Zbyszek, > > >> > > >> > > >> > > >> I must say, I personally don't care too much. NetworkManager is fine > > >> either way. > > >> > > >> There is however the problem about RHEL8/9, which patches this > > >> downstream. I don't like that deviation, but I'd also be uncomfortable > > >> to push that change for RHEL(10) users. > > >> > > >> But let me play devil's advocate here... > > >> > > >> > > >> On Mon, 2022-05-09 at 19:27 +0200, Zbigniew Jędrzejewski-Szmek wrote: > > >>> > > >>> FWIW, I still think it's a better _default_. > > >> > > >> I don't agree that it's clearly better. Your arguments don't seem > > >> strong, arguably, neither are mine. Except, that it's not clear that > > >> this solves an actual problem, while it clearly causes problems for > > >> some people. Just look at the referened issues from !3374. > > >> > > >> > > >> Either > > >> > > >> - a user doesn't care about the MAC address, > > > > > > note that it's possible for a user not to care about the *specific* > > > mac address, only that they want the mac to remain consistent. > > > > > > for example, on my system i have br0 bridge with multiple interfaces > > > attached, and my local DHCP server is configured to provide a static > > > addr to br0's mac. If I replace an interface card (e.g. change from 1g > > > to 10g, or just replace a failing nic) then the br0 mac *does not* > > > change, if using systemd's default. If I had br0 inheriting its mac > > > from one of the attached interfaces, it would change, and i'd have to > > > update my dhcp server config. > > > > > > I think the argument really comes down to, should the default be to > > > have (without user configuration) a mac that is predictable or a mac > > > that is consistent. Your argument is for predictability, which makes > > > the field engineer's work deploying systems easier, but if you lose > > > consistency then the life of the maintenance engineer gets harder. > > > > Interested discussion. Let's take it a bit further. > > > > On your system how did your DHCP server get configured to provide an > > addr to br0's MAC? You had to install the OS, and create br0 first > > before you even knew the MAC to tell the network admin what the MAC > > was. Now you're golden, the MAC will never change for the lifetime > > of that OS install even if someone replaces a NIC, but it wasn't a great > > first experience really. > > No, but the alternative is for me to manually find the mac address > labels on my nics and use those to pre-configure my DHCP server. I > understand for large deployments those are typically provided in a > spreadsheet, but that isn't the case for me, and my approach (i.e. > simply check the mac once the system is installed) was *much* easier > and more reliable. > > And this doesn't even get into the nuances of figuring out *which* nic > mac the bridge/bond will get, which won't be obvious to everyone. Note > that the answer here *is not* that the bridge gets the mac of the > 'first' interface added to it; the bridge gets the mac of the *lowest* > mac that is currently attached to it. And yes, this does mean that if > you use this behavior (of a bridge getting its mac from the attached > interfaces), the bridge's mac *can change while it is up and running > with an address*. Try it out yourself ;-) And to clarify - bond interfaces do 'keep' the mac of the 'first' interface added - technically, the bond sets all interfaces added later to have the same mac. So this is even less 'predictable' than a bridge, where you can predict which of the 2 or more macs are 'lowest', a bond mac would just be whatever the 'first' added interface is. > > > > > > On the other hand, what if you re-provision that server often (new > > machine-id) > > you get a new MAC and you get to dance with your network admin again. OR > > what if you have disk failure? You most likely backed up your critical data, > > but did you backup your machine-id that hashes your new MAC? Probably not > > and even if you did would you want to duplicate that machine-id to the new > > install you would do? > > > > Barring other reasons, if we simplify it down to just the consistency > > argument, > > one approach seems better for if you replace NIC cards often and one of them > > seems better if you re-install your OS often. > > Yes, 'consistency' has to be based on something, so 'consistency' > based on hardware is one approach; consistency based on software is > another. As you said some users may have software churn and so would > want to base their 'consistency' on their (presumably unchanging) > hardware. However, software churn is pretty rare at the bare metal > level, isn't it? Not unheard of, for sure, but I'd hardly say most > datacenters redeploy
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On Thu, 2022-05-12 at 13:36 -0400, Dan Streetman wrote: > On Thu, May 12, 2022 at 11:11 AM Thomas Haller > wrote: > > > > Either > > > > - a user doesn't care about the MAC address, > > note that it's possible for a user not to care about the *specific* > mac address, only that they want the mac to remain consistent. > > for example, on my system i have br0 bridge with multiple interfaces > attached, and my local DHCP server is configured to provide a static > addr to br0's mac. If I replace an interface card (e.g. change from > 1g > to 10g, or just replace a failing nic) then the br0 mac *does not* > change, if using systemd's default. If I had br0 inheriting its mac > from one of the attached interfaces, it would change, and i'd have to > update my dhcp server config. > > I think the argument really comes down to, should the default be to > have (without user configuration) a mac that is predictable or a mac > that is consistent. Your argument is for predictability, which makes > the field engineer's work deploying systems easier, but if you lose > consistency then the life of the maintenance engineer gets harder. > > > > - a user sets the desired MAC address themselves (possibly by > > setting MacAddressPolicy=) > > - or, the user intentionally does not set a MAC address (e.g. for > > bridge/bond). > > > > In no case there is a strong argument why udev should do it, and > > it's > > harmful in some cases. Hi, I considered that as case 2, the one where you care. You have all the means to set a MAC address also to something not specific. According to `man systemd.netdev`, a permanent MAC address is already the default. And I think systemd-networkd doesn't rely on udev's MACAddressPolicy for that (or does it?). NetworkManager also sets the MAC address that was configured, it does not expect udev to handle this. NetworkManager also supports a stable policy. If you use another tool which cannot set the MAC address, you can even use udev's MACAddressPolicy=. But that is then your configuration, not the distro default. Thomas
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On Thu, May 12, 2022 at 1:50 PM Dusty Mabe wrote: > > > > On 5/12/22 13:36, Dan Streetman wrote: > > On Thu, May 12, 2022 at 11:11 AM Thomas Haller wrote: > >> > >> Hi Zbyszek, > >> > >> > >> > >> I must say, I personally don't care too much. NetworkManager is fine > >> either way. > >> > >> There is however the problem about RHEL8/9, which patches this > >> downstream. I don't like that deviation, but I'd also be uncomfortable > >> to push that change for RHEL(10) users. > >> > >> But let me play devil's advocate here... > >> > >> > >> On Mon, 2022-05-09 at 19:27 +0200, Zbigniew Jędrzejewski-Szmek wrote: > >>> > >>> FWIW, I still think it's a better _default_. > >> > >> I don't agree that it's clearly better. Your arguments don't seem > >> strong, arguably, neither are mine. Except, that it's not clear that > >> this solves an actual problem, while it clearly causes problems for > >> some people. Just look at the referened issues from !3374. > >> > >> > >> Either > >> > >> - a user doesn't care about the MAC address, > > > > note that it's possible for a user not to care about the *specific* > > mac address, only that they want the mac to remain consistent. > > > > for example, on my system i have br0 bridge with multiple interfaces > > attached, and my local DHCP server is configured to provide a static > > addr to br0's mac. If I replace an interface card (e.g. change from 1g > > to 10g, or just replace a failing nic) then the br0 mac *does not* > > change, if using systemd's default. If I had br0 inheriting its mac > > from one of the attached interfaces, it would change, and i'd have to > > update my dhcp server config. > > > > I think the argument really comes down to, should the default be to > > have (without user configuration) a mac that is predictable or a mac > > that is consistent. Your argument is for predictability, which makes > > the field engineer's work deploying systems easier, but if you lose > > consistency then the life of the maintenance engineer gets harder. > > Interested discussion. Let's take it a bit further. > > On your system how did your DHCP server get configured to provide an > addr to br0's MAC? You had to install the OS, and create br0 first > before you even knew the MAC to tell the network admin what the MAC > was. Now you're golden, the MAC will never change for the lifetime > of that OS install even if someone replaces a NIC, but it wasn't a great > first experience really. No, but the alternative is for me to manually find the mac address labels on my nics and use those to pre-configure my DHCP server. I understand for large deployments those are typically provided in a spreadsheet, but that isn't the case for me, and my approach (i.e. simply check the mac once the system is installed) was *much* easier and more reliable. And this doesn't even get into the nuances of figuring out *which* nic mac the bridge/bond will get, which won't be obvious to everyone. Note that the answer here *is not* that the bridge gets the mac of the 'first' interface added to it; the bridge gets the mac of the *lowest* mac that is currently attached to it. And yes, this does mean that if you use this behavior (of a bridge getting its mac from the attached interfaces), the bridge's mac *can change while it is up and running with an address*. Try it out yourself ;-) > > On the other hand, what if you re-provision that server often (new machine-id) > you get a new MAC and you get to dance with your network admin again. OR > what if you have disk failure? You most likely backed up your critical data, > but did you backup your machine-id that hashes your new MAC? Probably not > and even if you did would you want to duplicate that machine-id to the new > install you would do? > > Barring other reasons, if we simplify it down to just the consistency > argument, > one approach seems better for if you replace NIC cards often and one of them > seems better if you re-install your OS often. Yes, 'consistency' has to be based on something, so 'consistency' based on hardware is one approach; consistency based on software is another. As you said some users may have software churn and so would want to base their 'consistency' on their (presumably unchanging) hardware. However, software churn is pretty rare at the bare metal level, isn't it? Not unheard of, for sure, but I'd hardly say most datacenters redeploy their bare metal servers regularly (though I admit I have no real knowledge of how often that happens). If we're talking about virtual software churn, which I think is much more common (e.g. redeploying an os into a VM), then - assuming the actual VM definition doesn't change (because if it did, then there is no hw consistency to base the mac consistency on), the machine-id will re-use the VM's uuid (provided via DMI as previously mentioned) and so the systemd-generated mac won't change. > > Dusty > >
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On 5/12/22 13:36, Dan Streetman wrote: > On Thu, May 12, 2022 at 11:11 AM Thomas Haller wrote: >> >> Hi Zbyszek, >> >> >> >> I must say, I personally don't care too much. NetworkManager is fine >> either way. >> >> There is however the problem about RHEL8/9, which patches this >> downstream. I don't like that deviation, but I'd also be uncomfortable >> to push that change for RHEL(10) users. >> >> But let me play devil's advocate here... >> >> >> On Mon, 2022-05-09 at 19:27 +0200, Zbigniew Jędrzejewski-Szmek wrote: >>> >>> FWIW, I still think it's a better _default_. >> >> I don't agree that it's clearly better. Your arguments don't seem >> strong, arguably, neither are mine. Except, that it's not clear that >> this solves an actual problem, while it clearly causes problems for >> some people. Just look at the referened issues from !3374. >> >> >> Either >> >> - a user doesn't care about the MAC address, > > note that it's possible for a user not to care about the *specific* > mac address, only that they want the mac to remain consistent. > > for example, on my system i have br0 bridge with multiple interfaces > attached, and my local DHCP server is configured to provide a static > addr to br0's mac. If I replace an interface card (e.g. change from 1g > to 10g, or just replace a failing nic) then the br0 mac *does not* > change, if using systemd's default. If I had br0 inheriting its mac > from one of the attached interfaces, it would change, and i'd have to > update my dhcp server config. > > I think the argument really comes down to, should the default be to > have (without user configuration) a mac that is predictable or a mac > that is consistent. Your argument is for predictability, which makes > the field engineer's work deploying systems easier, but if you lose > consistency then the life of the maintenance engineer gets harder. Interested discussion. Let's take it a bit further. On your system how did your DHCP server get configured to provide an addr to br0's MAC? You had to install the OS, and create br0 first before you even knew the MAC to tell the network admin what the MAC was. Now you're golden, the MAC will never change for the lifetime of that OS install even if someone replaces a NIC, but it wasn't a great first experience really. On the other hand, what if you re-provision that server often (new machine-id) you get a new MAC and you get to dance with your network admin again. OR what if you have disk failure? You most likely backed up your critical data, but did you backup your machine-id that hashes your new MAC? Probably not and even if you did would you want to duplicate that machine-id to the new install you would do? Barring other reasons, if we simplify it down to just the consistency argument, one approach seems better for if you replace NIC cards often and one of them seems better if you re-install your OS often. Dusty
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On Thu, May 12, 2022 at 11:11 AM Thomas Haller wrote: > > Hi Zbyszek, > > > > I must say, I personally don't care too much. NetworkManager is fine > either way. > > There is however the problem about RHEL8/9, which patches this > downstream. I don't like that deviation, but I'd also be uncomfortable > to push that change for RHEL(10) users. > > But let me play devil's advocate here... > > > On Mon, 2022-05-09 at 19:27 +0200, Zbigniew Jędrzejewski-Szmek wrote: > > > > FWIW, I still think it's a better _default_. > > I don't agree that it's clearly better. Your arguments don't seem > strong, arguably, neither are mine. Except, that it's not clear that > this solves an actual problem, while it clearly causes problems for > some people. Just look at the referened issues from !3374. > > > Either > > - a user doesn't care about the MAC address, note that it's possible for a user not to care about the *specific* mac address, only that they want the mac to remain consistent. for example, on my system i have br0 bridge with multiple interfaces attached, and my local DHCP server is configured to provide a static addr to br0's mac. If I replace an interface card (e.g. change from 1g to 10g, or just replace a failing nic) then the br0 mac *does not* change, if using systemd's default. If I had br0 inheriting its mac from one of the attached interfaces, it would change, and i'd have to update my dhcp server config. I think the argument really comes down to, should the default be to have (without user configuration) a mac that is predictable or a mac that is consistent. Your argument is for predictability, which makes the field engineer's work deploying systems easier, but if you lose consistency then the life of the maintenance engineer gets harder. > - a user sets the desired MAC address themselves (possibly by > setting MacAddressPolicy=) > - or, the user intentionally does not set a MAC address (e.g. for > bridge/bond). > > In no case there is a strong argument why udev should do it, and it's > harmful in some cases. > > > > > > The patch that finally > > introduced this was my patch [1], so I'm obviously biased… Some more > > considerations: > > > > 1. this allows bridge devices to be created without attached > > interfaces, and have a stable MAC address. > > yes. Does somebody care? In particular, kernel will assign a certain > MAC address when the first port gets attached. Some people care about > that. > > > > 2. the idea that all interfaces are always available and always in > > the > > same order is something that isn't necessarilly true for modern > > systems > > where we want to react to hardware being detected dynamically. > > If we wait until late userspace to configure networking, then yeah, > > all > > devices will most likely have been detected by that time. But if we > > want > > to bring networking in the initrd, not all hardware must be detected > > by > > the time we start configuration. > > It's either way a well-known MAC address from one of the interfaces. > > > > > 3. one of the reasons to use bridge/bond and similar devices it that > > the agreggate device can function when some of the child devices die. > > By requiring the presence of one of the devices, we're partially > > defeating this. > > In practice, this race is not happening for many users. > > It's not clear why it's a problem to get the MAC address of an > "unexpected" port (due to the race). It does not negate the uses of the > bride/bond. > > > > > [1] https://github.com/systemd/systemd/commit/6d36464065 > > > > > 1) for bridge/bond interfaces, there is a special meaning of > > > leaving > > > the MAC address unassigned. It causes kernel to automatically set > > > the > > > MAC address when the first port gets attached. By setting a > > > persistent > > > MAC address, that automatism is not longer possible. > > > > > > The MAC address can matter, for example, if you configure the DHCP > > > server to hand out IP addresses based on the MAC address (or based > > > on > > > the client-id, which in turn might be based on the MAC address). If > > > you > > > boot many machines (e.g. in data center or cloud), you might know > > > the > > > MAC address of the machines, and thereby can also determine the > > > assigned IP addresses. With MACAddressPolicy=persistent the MAC > > > address > > > is not predictable. > > > > This is true. But one can just as well argument that with > > MACAddressPolicy=persistent the address is even more predictable. If > > you know the machine-id and device name, you can calculate the > > address > > in advance, even before deciding if the device will e.g. have this or > > that card attached. > > It's unpredictable, because you need to know the machine-id. If you > boot a machine the first, you may not know that before hand. > > Unless you explicitly specifying the machine-id via kernel command > line, or similar. At which point, you need individual boot options (or > images) for each
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On 5/12/22 12:43, Lennart Poettering wrote: > On Mo, 09.05.22 22:37, Dusty Mabe (du...@dustymabe.com) wrote: > >>> This is true. But one can just as well argument that with >>> MACAddressPolicy=persistent the address is even more predictable. If >>> you know the machine-id and device name, you can calculate the address >>> in advance, even before deciding if the device will e.g. have this or >>> that card attached. >> >> Regarding machine-id, isn't that unique and set on first boot? > > Not necessarily. We will initialize it from the ID passed in through > DMI if we detect execution in a VM and the ID is not set yet. This > means cloud providers can control the machine ID a system will use > ahead of time. > OK. But in practice, how often is that used versus machine-id just being randomly generated on first boot? Dusty
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On Mo, 09.05.22 22:37, Dusty Mabe (du...@dustymabe.com) wrote: > > This is true. But one can just as well argument that with > > MACAddressPolicy=persistent the address is even more predictable. If > > you know the machine-id and device name, you can calculate the address > > in advance, even before deciding if the device will e.g. have this or > > that card attached. > > Regarding machine-id, isn't that unique and set on first boot? Not necessarily. We will initialize it from the ID passed in through DMI if we detect execution in a VM and the ID is not set yet. This means cloud providers can control the machine ID a system will use ahead of time. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On Mo, 09.05.22 19:27, Zbigniew Jędrzejewski-Szmek (zbys...@in.waw.pl) wrote: > FWIW, I still think it's a better _default_. The patch that finally > introduced this was my patch [1], so I'm obviously biased… Some more > considerations: I agree with this. Finding good defaults is always difficult, but I must say stability and predictability is a property I like above a lot of other stuff. I understand that in plenty environments it's important not to add new MAC addresses to the mix, but it's impossible to know in which environment we are. So, either way, some people will always be unhappy with the defaults we pick. Changing defaults makes sense if it's highly likely that the vast majority of users would benefit from the new default. But I don't see that here... And changing defaults comes at a price, because it will break people's setup. We made a change once here, but I wouldn't use that as excuse to change it again... So, I am not convinced. What makes me wonder about all of this: we are talking about synthetic devices, which means some tool is used to create them. If those tools prefer a different MAC policy, why don't they drop in a .link file that matches against the devices that specific software creates? i.e. let's say NM prefers to use a different MAC policy: they could drop in a udev rules file that adds some udev property onto the network devices they manage (i.e. invoke a callout binary, or do a TEST check or so which checks the iface against some NM state, and then set ID_NM_OWNED_AND_OPERATED=1 or so). Then, ship a .link file (or multiple) with a Property= match in the [Match] section, that sets the desired policy. With such a logic, NM could make its own choices on MAC policy, but the default systemd policy wouldn't have to change. Also, afaik OSes that run in clouds all have some tool like cloud-init or ignition or so, which generate .network files in /run with the right configuration. Why not generate .link files in /run the same way with a MAC policy appropriate for the cloud provider? Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On Thu, 2022-05-12 at 17:11 +0200, Thomas Haller wrote: > We are talking here about software device which are always created by > some tool/software/user. Presumably that creator has plans for this > interface, and it's not clear why udev is supposed to change such a > fundamental setting. I think that's the important point here. This interface is created by somebody (NetworkManager, networkd, docker, etc.). In all cases, they did not leave the MAC address unspecified by accident. They create (or could create) the interface with the right parameters already. If networkd wishes to default to MACAddressPolicy=persistent (a sensible choice!), it may fully do so. Why is udev doing this? To "help" unknowing users who use `ip link add`? When it just introduces a race for them. best, Thomas
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
Hi Zbyszek, I must say, I personally don't care too much. NetworkManager is fine either way. There is however the problem about RHEL8/9, which patches this downstream. I don't like that deviation, but I'd also be uncomfortable to push that change for RHEL(10) users. But let me play devil's advocate here... On Mon, 2022-05-09 at 19:27 +0200, Zbigniew Jędrzejewski-Szmek wrote: > > FWIW, I still think it's a better _default_. I don't agree that it's clearly better. Your arguments don't seem strong, arguably, neither are mine. Except, that it's not clear that this solves an actual problem, while it clearly causes problems for some people. Just look at the referened issues from !3374. Either - a user doesn't care about the MAC address, - a user sets the desired MAC address themselves (possibly by setting MacAddressPolicy=) - or, the user intentionally does not set a MAC address (e.g. for bridge/bond). In no case there is a strong argument why udev should do it, and it's harmful in some cases. > The patch that finally > introduced this was my patch [1], so I'm obviously biased… Some more > considerations: > > 1. this allows bridge devices to be created without attached > interfaces, and have a stable MAC address. yes. Does somebody care? In particular, kernel will assign a certain MAC address when the first port gets attached. Some people care about that. > 2. the idea that all interfaces are always available and always in > the > same order is something that isn't necessarilly true for modern > systems > where we want to react to hardware being detected dynamically. > If we wait until late userspace to configure networking, then yeah, > all > devices will most likely have been detected by that time. But if we > want > to bring networking in the initrd, not all hardware must be detected > by > the time we start configuration. It's either way a well-known MAC address from one of the interfaces. > > 3. one of the reasons to use bridge/bond and similar devices it that > the agreggate device can function when some of the child devices die. > By requiring the presence of one of the devices, we're partially > defeating this. In practice, this race is not happening for many users. It's not clear why it's a problem to get the MAC address of an "unexpected" port (due to the race). It does not negate the uses of the bride/bond. > > [1] https://github.com/systemd/systemd/commit/6d36464065 > > > 1) for bridge/bond interfaces, there is a special meaning of > > leaving > > the MAC address unassigned. It causes kernel to automatically set > > the > > MAC address when the first port gets attached. By setting a > > persistent > > MAC address, that automatism is not longer possible. > > > > The MAC address can matter, for example, if you configure the DHCP > > server to hand out IP addresses based on the MAC address (or based > > on > > the client-id, which in turn might be based on the MAC address). If > > you > > boot many machines (e.g. in data center or cloud), you might know > > the > > MAC address of the machines, and thereby can also determine the > > assigned IP addresses. With MACAddressPolicy=persistent the MAC > > address > > is not predictable. > > This is true. But one can just as well argument that with > MACAddressPolicy=persistent the address is even more predictable. If > you know the machine-id and device name, you can calculate the > address > in advance, even before deciding if the device will e.g. have this or > that card attached. It's unpredictable, because you need to know the machine-id. If you boot a machine the first, you may not know that before hand. Unless you explicitly specifying the machine-id via kernel command line, or similar. At which point, you need individual boot options (or images) for each machine. > > I'm not sure if we expose this conveniently anywhere… Would it help > if > we document how to do this calculation with python one-liner or some > helper? > > > 2) udev changing the MAC address causes races with naive > > scripts/tools. > > For example: > > > > ip monitor link & > > while : ; do > > ip link del xxx > > ip link add name xxx type dummy \ > > && ip link set xxx addr aa:00:00:00:00:00 \ > > && ip link show xxx | grep -q aa:00:00:00:00:00 \ > > || break > > done > > Again, this is a question of expecations. With MACAP=p, 'dummy' gets > the same address every time, and maybe there's no reason to set it > manually. > > It would be great if we could have 'ip link add' wait for udev to > process the device. Maybe even by default? > > We also discussed adding a kernel command-line switch similar to > net.ifnames= to allow this to be configured globally. Would that > help? That might help. Granted, it's pretty easy already. RHEL ships a one- line patch downstream, and CoreOS probably could deploy some .link file that overrules this. And any user can write their own .link file. > > The cases
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
> 1) for bridge/bond interfaces, there is a special meaning of leaving > the MAC address unassigned. It causes kernel to automatically set the > MAC address when the first port gets attached. By setting a persistent > MAC address, that automatism is not longer possible. This is incredibly important when using bonded interfaces at cloud providers that expect only to see known MAC addresses on the network. For example, at Equinix Metal (formerly packet.net), the switch upstream from the server expects to see MAC addresses from either of the NICs attached to the server. If a bond interface MAC doesn't match either of those two, the network traffic is blocked. This is a common practice for many cloud providers. This issue has caused me plenty of headaches until I realized the switch was eating my traffic coming from an unknown MAC. -- Major Hayden
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On 5/9/22 13:27, Zbigniew Jędrzejewski-Szmek wrote: > On Mon, May 09, 2022 at 03:57:21PM +0200, Lennart Poettering wrote: >> On Mo, 09.05.22 11:23, Thomas Haller (thal...@redhat.com) wrote: >> >>> Hi everybody, >>> >>> this email is for discussing MACAddressPolicy=persistent in >>> /data/src/systemd/network/99-default.link >> >> I think this would be better discussed on a new github issue, as >> suggested here: > > I suggested systemd-devel for this… It's not a question of a bug, > but instead whether the default MACAddressPolicy makes sense. So I think > it's better to discuss this on the mailing list. > > FWIW, I still think it's a better _default_. The patch that finally > introduced this was my patch [1], so I'm obviously biased… Some more > considerations: > > 1. this allows bridge devices to be created without attached > interfaces, and have a stable MAC address. > > 2. the idea that all interfaces are always available and always in the > same order is something that isn't necessarilly true for modern systems > where we want to react to hardware being detected dynamically. > If we wait until late userspace to configure networking, then yeah, all > devices will most likely have been detected by that time. But if we want > to bring networking in the initrd, not all hardware must be detected by > the time we start configuration. > > 3. one of the reasons to use bridge/bond and similar devices it that > the agreggate device can function when some of the child devices die. > By requiring the presence of one of the devices, we're partially > defeating this. > > [1] https://github.com/systemd/systemd/commit/6d36464065 > >> 1) for bridge/bond interfaces, there is a special meaning of leaving >> the MAC address unassigned. It causes kernel to automatically set the >> MAC address when the first port gets attached. By setting a persistent >> MAC address, that automatism is not longer possible. >> >> The MAC address can matter, for example, if you configure the DHCP >> server to hand out IP addresses based on the MAC address (or based on >> the client-id, which in turn might be based on the MAC address). If you >> boot many machines (e.g. in data center or cloud), you might know the >> MAC address of the machines, and thereby can also determine the >> assigned IP addresses. With MACAddressPolicy=persistent the MAC address >> is not predictable. > > This is true. But one can just as well argument that with > MACAddressPolicy=persistent the address is even more predictable. If > you know the machine-id and device name, you can calculate the address > in advance, even before deciding if the device will e.g. have this or > that card attached. Regarding machine-id, isn't that unique and set on first boot? So you don't really even know your "new mac address" until your system is up and running (and now can't get DHCP because your net is locked down). For me it's more about expectations and the user being surprised, which isn't necessarily a bug. If I'm controlling a datacenter (or even my 10s of devices on my home network) I generally know what hardware exists in the environment and don't want to be surprised to see activity from "unexpected hardware". I lock down my DHCP via MAC address of the known hardware NICs in the lab/DC. Often times hardware gets racked and configured in the datacenter environment before it can even be powered on. You get the MAC address from the documentation (labels, stickers, etc) and have your network admins plumb through the necessary bits. I understand your arguments for the new behavior, but my preference would be to keep the old behavior. > > I'm not sure if we expose this conveniently anywhere… Would it help if > we document how to do this calculation with python one-liner or some > helper? > >> 2) udev changing the MAC address causes races with naive scripts/tools. >> For example: >> >> ip monitor link & >> while : ; do >> ip link del xxx >> ip link add name xxx type dummy \ >> && ip link set xxx addr aa:00:00:00:00:00 \ >> && ip link show xxx | grep -q aa:00:00:00:00:00 \ >> || break >> done > > Again, this is a question of expecations. With MACAP=p, 'dummy' gets > the same address every time, and maybe there's no reason to set it > manually. > > It would be great if we could have 'ip link add' wait for udev to > process the device. Maybe even by default? > > We also discussed adding a kernel command-line switch similar to > net.ifnames= to allow this to be configured globally. Would that > help? Right. I opened that request here: https://github.com/systemd/systemd/issues/23294 I think it helps. I would want to make sure it applied in the initrd as well. > > The cases where the old behaviour is relied on seems to be cases like > the data center described above. But in that case you're creating > local configuration anyway, so dropping in an override should be > acceptable. At least for bonds I'd argue pretty hard that
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On Mon, May 09, 2022 at 03:57:21PM +0200, Lennart Poettering wrote: > On Mo, 09.05.22 11:23, Thomas Haller (thal...@redhat.com) wrote: > > > Hi everybody, > > > > this email is for discussing MACAddressPolicy=persistent in > > /data/src/systemd/network/99-default.link > > I think this would be better discussed on a new github issue, as > suggested here: I suggested systemd-devel for this… It's not a question of a bug, but instead whether the default MACAddressPolicy makes sense. So I think it's better to discuss this on the mailing list. FWIW, I still think it's a better _default_. The patch that finally introduced this was my patch [1], so I'm obviously biased… Some more considerations: 1. this allows bridge devices to be created without attached interfaces, and have a stable MAC address. 2. the idea that all interfaces are always available and always in the same order is something that isn't necessarilly true for modern systems where we want to react to hardware being detected dynamically. If we wait until late userspace to configure networking, then yeah, all devices will most likely have been detected by that time. But if we want to bring networking in the initrd, not all hardware must be detected by the time we start configuration. 3. one of the reasons to use bridge/bond and similar devices it that the agreggate device can function when some of the child devices die. By requiring the presence of one of the devices, we're partially defeating this. [1] https://github.com/systemd/systemd/commit/6d36464065 > 1) for bridge/bond interfaces, there is a special meaning of leaving > the MAC address unassigned. It causes kernel to automatically set the > MAC address when the first port gets attached. By setting a persistent > MAC address, that automatism is not longer possible. > > The MAC address can matter, for example, if you configure the DHCP > server to hand out IP addresses based on the MAC address (or based on > the client-id, which in turn might be based on the MAC address). If you > boot many machines (e.g. in data center or cloud), you might know the > MAC address of the machines, and thereby can also determine the > assigned IP addresses. With MACAddressPolicy=persistent the MAC address > is not predictable. This is true. But one can just as well argument that with MACAddressPolicy=persistent the address is even more predictable. If you know the machine-id and device name, you can calculate the address in advance, even before deciding if the device will e.g. have this or that card attached. I'm not sure if we expose this conveniently anywhere… Would it help if we document how to do this calculation with python one-liner or some helper? > 2) udev changing the MAC address causes races with naive scripts/tools. > For example: > > ip monitor link & > while : ; do > ip link del xxx > ip link add name xxx type dummy \ > && ip link set xxx addr aa:00:00:00:00:00 \ > && ip link show xxx | grep -q aa:00:00:00:00:00 \ > || break > done Again, this is a question of expecations. With MACAP=p, 'dummy' gets the same address every time, and maybe there's no reason to set it manually. It would be great if we could have 'ip link add' wait for udev to process the device. Maybe even by default? We also discussed adding a kernel command-line switch similar to net.ifnames= to allow this to be configured globally. Would that help? The cases where the old behaviour is relied on seems to be cases like the data center described above. But in that case you're creating local configuration anyway, so dropping in an override should be acceptable. Zbyszek > https://github.com/systemd/systemd/issues/3374#issuecomment-1031072530 > > or here: > > https://github.com/systemd/systemd/issues/3374#issuecomment-601240730
Re: [systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
On Mo, 09.05.22 11:23, Thomas Haller (thal...@redhat.com) wrote: > Hi everybody, > > this email is for discussing MACAddressPolicy=persistent in > /data/src/systemd/network/99-default.link I think this would be better discussed on a new github issue, as suggested here: https://github.com/systemd/systemd/issues/3374#issuecomment-1031072530 or here: https://github.com/systemd/systemd/issues/3374#issuecomment-601240730 I don't think that new issue was ever filed? Lennart -- Lennart Poettering, Berlin
[systemd-devel] Should `MACAddressPolicy=persistent` for bridges/bonds/all-software-devices be reconsidered?
Hi everybody, this email is for discussing MACAddressPolicy=persistent in /data/src/systemd/network/99-default.link there is a Fedora CoreOS issue about this: [1] https://github.com/coreos/fedora-coreos-tracker/issues/919 Since systemd 242 (Apr 2019), this policy applies to more device types ([2]). [2] https://github.com/systemd/systemd/blob/v247/NEWS#L2332-L2358 This topic was already discussed. For example at [3]. I also brought this up before at [4], but I'd like to reopen this discussion with the recent Fedora CoreOS issue. [3] https://github.com/systemd/systemd/issues/3374 [4] https://github.com/systemd/systemd/issues/3374#issuecomment-522528484 To reiterate, this causes two problems: 1) for bridge/bond interfaces, there is a special meaning of leaving the MAC address unassigned. It causes kernel to automatically set the MAC address when the first port gets attached. By setting a persistent MAC address, that automatism is not longer possible. The MAC address can matter, for example, if you configure the DHCP server to hand out IP addresses based on the MAC address (or based on the client-id, which in turn might be based on the MAC address). If you boot many machines (e.g. in data center or cloud), you might know the MAC address of the machines, and thereby can also determine the assigned IP addresses. With MACAddressPolicy=persistent the MAC address is not predictable. 2) udev changing the MAC address causes races with naive scripts/tools. For example: ip monitor link & while : ; do ip link del xxx ip link add name xxx type dummy \ && ip link set xxx addr aa:00:00:00:00:00 \ && ip link show xxx | grep -q aa:00:00:00:00:00 \ || break done granted, a script should either set the MAC address from the start, or it should wait for udev to settle. Still, how many tools out there are (still) broken? On RHEL, 99-default.link is patched to set MACAddressPolicy=none. On Fedora, despite this behavior being there for a while now, I think it's harmful and should be dropped as well... at least for bridges/bonds (problem 1) or even all software devices (problem 2). In my opionion, this approach should be reconsidered for udev in general. best, Thomas signature.asc Description: This is a digitally signed message part
