Re: [systemd-devel] Antw: [EXT] Re: [systemd‑devel] version bump of minimal kernel version supported by systemd?
On Fr, 01.04.22 13:54, Greg Kroah-Hartman (gre...@linuxfoundation.org) wrote: > > While it is true that the syscall interface is kept reasonably stable, > > almost everything else gets monkeyed with a lot, because a lot of > > kernel developers only consider the syscall interface a program > > interface. This is a problem because a *lot* of things are only > > accessible through other means (procfs, sysfs, uevents, etc.). > > > > Unfortunately, that means that in practice, the kernel interfaces that > > userspace *must* depend on break far more than anyone likes. > > The above example is an interesting case. A new feature was added, was > around for a while, and a few _years later_ it was found out that some That's not quite true. The breakages were actually reported pretty quickly to the kernel people who added the offending patches, and they even changed some things around (an incomplete patch for udev was posted, which we merged), but the issue was still not properly addressed. It died down then, nothing much happened, and udev maintainers didn't bring this up again for a while, as they had other stuff to do. The issue became more and more visible though as more subsystems in the kernel started generating these uevents, to a point where ignoring the issue wasn't sustainable. At that point kernel people were pretty dismissive though (not that they were particularly helpful in the beginning either), partly because the change was in now for so long. So we reworked how udev worked. > people had userspace "scripts" that broke because the feature was > added. nah, this broke C code all over the place, too. Not just "scripts". I am not even disagreeing though that bind/unbind uevents made sense to add. I just want to correct how things happened here. There was a general disinterest from the kernel people who broke things to fix things, and in particular major disinterest in understanding how udev actually works and how udev rules are used IRL. (I mean, that early patch we got and merged literally just changed udev to drop messages with bind/unbind entirely, thus not fixing anything, just hiding the problem with no prospect of actually making it useful for userspace. I figure the kernel devs involved actually cared about Android, and not classic Linux userspace, i.e. udev.) I know the kernel people like to carry that mantra of not breaking userspace quite like a monstrance, but IRL it's broken all the time. Often for good reasons, quite often also for no reason but lack of testing. Things like that will happen. But I also think that Windows for example is probably better at not breaking their interfaces than Linux is. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] Antw: [EXT] Re: [systemd‑devel] version bump of minimal kernel version supported by systemd?
On Fri, Apr 01, 2022 at 07:39:25AM -0400, Neal Gompa wrote: > On Thu, Mar 24, 2022 at 8:15 AM Greg Kroah-Hartman > wrote: > > > > On Thu, Mar 24, 2022 at 10:23:21AM +, Luca Boccassi wrote: > > > On Thu, 2022-03-24 at 09:45 +0100, Greg Kroah-Hartman wrote: > > > > On Thu, Mar 24, 2022 at 09:33:50AM +0100, Ulrich Windl wrote: > > > > > > > > Greg KH schrieb am 24.03.2022 um > > > > > > > > 08:12 in > > > > > Nachricht : > > > > > > On Wed, Mar 23, 2022 at 10:34:00PM +, Dave Howorth wrote: > > > > > > > FWIW, I think Greg was a bit too outspoken calling long > > > > > > > maintenance > > > > > > > attempts 'crazy'; that may have intimidated some. I'm thinking of > > > > > > > moving distro to one that provides longer term maintenance than my > > > > > > > present one. Although CIP is a completely different ball game; I > > > > > > > hope > > > > > > > they succeed. > > > > > > > > > > > > It is not "crazy" it is "well documented". As someone who has been > > > > > > doing this work for 20+ years now and sees all of the stable kernel > > > > > > patches flow by, it's obvious that a distro that does not keep up > > > > > > with > > > > > > them is insecure by design. > > > > > > > > > > If "newer is better" I'd agree. Sometimes "newer is actually worse". > > > > > Some new features intended to improve things sometimes actually make > > > > > things worse. > > > > > > > > That's not the issue here. > > > > > > > > Do you want to run a kernel with known security problems, or one with > > > > "unknown potential problems." The latter is always the case, so please > > > > don't pick the known-insecure one, that's just foolish. > > > > > > "security problems" are a dime a dozen, as they say. Speaking as a > > > (thankfully former) downstream integrator, you'd have much more success > > > if you stopped breaking backward compatibility with userspace all the > > > damn time. Upgrading major kernel version is like rolling a dice, you > > > never know what kind of extremely expensive and time consuming rabbit > > > hole you'll be dragged into because the kernel plays fast and loose > > > with its userspace interfaces, and each and every time there's a chance > > > one might end up having to do major reworks to deal with it. > > > > We should never be breaking working userspace programs when upgrading > > the kernel. If so, please report it to the regressions mailing list. > > > > Of course there's always some corner cases, but for the most part, this > > should never happen. > > > > > So really it shouldn't be that surprising that users are averse to > > > following the "latest is greatest" mantra from kernel.org, given how > > > risky and expensive it is, and how little one gains in return. Rather > > > than changing the world, what about changing your own processes first? > > > A great starting point would be reverting backward incompatible changes > > > regardless of who's affected, instead of doing that only if they affect > > > the personal computer of a handful of maintainers (mainly Linus'), and > > > shrugging reports away with "deal with it" in other cases. > > > > We should never be "shrugging" away reports like this. If you have > > specific incidents that you wish to discuss, I will be glad to do so on > > the regressions kernel mailing list. Otherwise this is way off-topic > > for systemd-devel. > > > > >From a systemd-relevant angle, this has happened before. Recently > even: > https://github.com/systemd/systemd/blob/8c70e8024ba8ff42c23f1a35b9e8fafddd5caa8d/NEWS#L2355-L2437 (for those that don't want to look this up, it's the BIND/UNBIND uevent issue.) > That means that from a reasonable systemd user perspective, we need to > depend on Linux kernel 4.14 or higher to cross over that breakage. Great! :) > While it is true that the syscall interface is kept reasonably stable, > almost everything else gets monkeyed with a lot, because a lot of > kernel developers only consider the syscall interface a program > interface. This is a problem because a *lot* of things are only > accessible through other means (procfs, sysfs, uevents, etc.). > > Unfortunately, that means that in practice, the kernel interfaces that > userspace *must* depend on break far more than anyone likes. The above example is an interesting case. A new feature was added, was around for a while, and a few _years later_ it was found out that some people had userspace "scripts" that broke because the feature was added. Yet people had already started depending on this feature, and really, the "broken" scripts had always been broken, they just worked accidentally by virtue that userspace was assuming the type of uevent that was happening. So what would you do if you were in my position? We can't break existing userspace tools that relied on the new events, and yet upgrading the kernel broke older tools if they were not changed. Anyway, fixing the scripts worked for everyone, and I think systemd did the right thing
Re: [systemd-devel] Antw: [EXT] Re: [systemd‑devel] version bump of minimal kernel version supported by systemd?
On Thu, Mar 24, 2022 at 8:15 AM Greg Kroah-Hartman wrote: > > On Thu, Mar 24, 2022 at 10:23:21AM +, Luca Boccassi wrote: > > On Thu, 2022-03-24 at 09:45 +0100, Greg Kroah-Hartman wrote: > > > On Thu, Mar 24, 2022 at 09:33:50AM +0100, Ulrich Windl wrote: > > > > > > > Greg KH schrieb am 24.03.2022 um > > > > > > > 08:12 in > > > > Nachricht : > > > > > On Wed, Mar 23, 2022 at 10:34:00PM +, Dave Howorth wrote: > > > > > > FWIW, I think Greg was a bit too outspoken calling long maintenance > > > > > > attempts 'crazy'; that may have intimidated some. I'm thinking of > > > > > > moving distro to one that provides longer term maintenance than my > > > > > > present one. Although CIP is a completely different ball game; I > > > > > > hope > > > > > > they succeed. > > > > > > > > > > It is not "crazy" it is "well documented". As someone who has been > > > > > doing this work for 20+ years now and sees all of the stable kernel > > > > > patches flow by, it's obvious that a distro that does not keep up with > > > > > them is insecure by design. > > > > > > > > If "newer is better" I'd agree. Sometimes "newer is actually worse". > > > > Some new features intended to improve things sometimes actually make > > > > things worse. > > > > > > That's not the issue here. > > > > > > Do you want to run a kernel with known security problems, or one with > > > "unknown potential problems." The latter is always the case, so please > > > don't pick the known-insecure one, that's just foolish. > > > > "security problems" are a dime a dozen, as they say. Speaking as a > > (thankfully former) downstream integrator, you'd have much more success > > if you stopped breaking backward compatibility with userspace all the > > damn time. Upgrading major kernel version is like rolling a dice, you > > never know what kind of extremely expensive and time consuming rabbit > > hole you'll be dragged into because the kernel plays fast and loose > > with its userspace interfaces, and each and every time there's a chance > > one might end up having to do major reworks to deal with it. > > We should never be breaking working userspace programs when upgrading > the kernel. If so, please report it to the regressions mailing list. > > Of course there's always some corner cases, but for the most part, this > should never happen. > > > So really it shouldn't be that surprising that users are averse to > > following the "latest is greatest" mantra from kernel.org, given how > > risky and expensive it is, and how little one gains in return. Rather > > than changing the world, what about changing your own processes first? > > A great starting point would be reverting backward incompatible changes > > regardless of who's affected, instead of doing that only if they affect > > the personal computer of a handful of maintainers (mainly Linus'), and > > shrugging reports away with "deal with it" in other cases. > > We should never be "shrugging" away reports like this. If you have > specific incidents that you wish to discuss, I will be glad to do so on > the regressions kernel mailing list. Otherwise this is way off-topic > for systemd-devel. > >From a systemd-relevant angle, this has happened before. Recently even: https://github.com/systemd/systemd/blob/8c70e8024ba8ff42c23f1a35b9e8fafddd5caa8d/NEWS#L2355-L2437 That means that from a reasonable systemd user perspective, we need to depend on Linux kernel 4.14 or higher to cross over that breakage. While it is true that the syscall interface is kept reasonably stable, almost everything else gets monkeyed with a lot, because a lot of kernel developers only consider the syscall interface a program interface. This is a problem because a *lot* of things are only accessible through other means (procfs, sysfs, uevents, etc.). Unfortunately, that means that in practice, the kernel interfaces that userspace *must* depend on break far more than anyone likes. -- 真実はいつも一つ！/ Always, there's only one truth!
Re: [systemd-devel] Antw: [EXT] Re: [systemd‑devel] version bump of minimal kernel version supported by systemd?
On Thu, Mar 24, 2022 at 10:23:21AM +, Luca Boccassi wrote: > On Thu, 2022-03-24 at 09:45 +0100, Greg Kroah-Hartman wrote: > > On Thu, Mar 24, 2022 at 09:33:50AM +0100, Ulrich Windl wrote: > > > > > > Greg KH schrieb am 24.03.2022 um 08:12 > > > > > > in > > > Nachricht : > > > > On Wed, Mar 23, 2022 at 10:34:00PM +, Dave Howorth wrote: > > > > > FWIW, I think Greg was a bit too outspoken calling long maintenance > > > > > attempts 'crazy'; that may have intimidated some. I'm thinking of > > > > > moving distro to one that provides longer term maintenance than my > > > > > present one. Although CIP is a completely different ball game; I hope > > > > > they succeed. > > > > > > > > It is not "crazy" it is "well documented". As someone who has been > > > > doing this work for 20+ years now and sees all of the stable kernel > > > > patches flow by, it's obvious that a distro that does not keep up with > > > > them is insecure by design. > > > > > > If "newer is better" I'd agree. Sometimes "newer is actually worse". > > > Some new features intended to improve things sometimes actually make > > > things worse. > > > > That's not the issue here. > > > > Do you want to run a kernel with known security problems, or one with > > "unknown potential problems." The latter is always the case, so please > > don't pick the known-insecure one, that's just foolish. > > "security problems" are a dime a dozen, as they say. Speaking as a > (thankfully former) downstream integrator, you'd have much more success > if you stopped breaking backward compatibility with userspace all the > damn time. Upgrading major kernel version is like rolling a dice, you > never know what kind of extremely expensive and time consuming rabbit > hole you'll be dragged into because the kernel plays fast and loose > with its userspace interfaces, and each and every time there's a chance > one might end up having to do major reworks to deal with it. We should never be breaking working userspace programs when upgrading the kernel. If so, please report it to the regressions mailing list. Of course there's always some corner cases, but for the most part, this should never happen. > So really it shouldn't be that surprising that users are averse to > following the "latest is greatest" mantra from kernel.org, given how > risky and expensive it is, and how little one gains in return. Rather > than changing the world, what about changing your own processes first? > A great starting point would be reverting backward incompatible changes > regardless of who's affected, instead of doing that only if they affect > the personal computer of a handful of maintainers (mainly Linus'), and > shrugging reports away with "deal with it" in other cases. We should never be "shrugging" away reports like this. If you have specific incidents that you wish to discuss, I will be glad to do so on the regressions kernel mailing list. Otherwise this is way off-topic for systemd-devel. thanks, greg k-h
Re: [systemd-devel] Antw: [EXT] Re: [systemd‑devel] version bump of minimal kernel version supported by systemd?
On Thu, 2022-03-24 at 09:45 +0100, Greg Kroah-Hartman wrote: > On Thu, Mar 24, 2022 at 09:33:50AM +0100, Ulrich Windl wrote: > > > > > Greg KH schrieb am 24.03.2022 um 08:12 in > > Nachricht : > > > On Wed, Mar 23, 2022 at 10:34:00PM +, Dave Howorth wrote: > > > > FWIW, I think Greg was a bit too outspoken calling long maintenance > > > > attempts 'crazy'; that may have intimidated some. I'm thinking of > > > > moving distro to one that provides longer term maintenance than my > > > > present one. Although CIP is a completely different ball game; I hope > > > > they succeed. > > > > > > It is not "crazy" it is "well documented". As someone who has been > > > doing this work for 20+ years now and sees all of the stable kernel > > > patches flow by, it's obvious that a distro that does not keep up with > > > them is insecure by design. > > > > If "newer is better" I'd agree. Sometimes "newer is actually worse". > > Some new features intended to improve things sometimes actually make things > > worse. > > That's not the issue here. > > Do you want to run a kernel with known security problems, or one with > "unknown potential problems." The latter is always the case, so please > don't pick the known-insecure one, that's just foolish. "security problems" are a dime a dozen, as they say. Speaking as a (thankfully former) downstream integrator, you'd have much more success if you stopped breaking backward compatibility with userspace all the damn time. Upgrading major kernel version is like rolling a dice, you never know what kind of extremely expensive and time consuming rabbit hole you'll be dragged into because the kernel plays fast and loose with its userspace interfaces, and each and every time there's a chance one might end up having to do major reworks to deal with it. So really it shouldn't be that surprising that users are averse to following the "latest is greatest" mantra from kernel.org, given how risky and expensive it is, and how little one gains in return. Rather than changing the world, what about changing your own processes first? A great starting point would be reverting backward incompatible changes regardless of who's affected, instead of doing that only if they affect the personal computer of a handful of maintainers (mainly Linus'), and shrugging reports away with "deal with it" in other cases. Just my 2c. -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Re: [systemd-devel] Antw: [EXT] Re: [systemd‑devel] version bump of minimal kernel version supported by systemd?
On Thu, Mar 24, 2022 at 09:33:50AM +0100, Ulrich Windl wrote: > >>> Greg KH schrieb am 24.03.2022 um 08:12 in > Nachricht : > > On Wed, Mar 23, 2022 at 10:34:00PM +, Dave Howorth wrote: > >> FWIW, I think Greg was a bit too outspoken calling long maintenance > >> attempts 'crazy'; that may have intimidated some. I'm thinking of > >> moving distro to one that provides longer term maintenance than my > >> present one. Although CIP is a completely different ball game; I hope > >> they succeed. > > > > It is not "crazy" it is "well documented". As someone who has been > > doing this work for 20+ years now and sees all of the stable kernel > > patches flow by, it's obvious that a distro that does not keep up with > > them is insecure by design. > > If "newer is better" I'd agree. Sometimes "newer is actually worse". > Some new features intended to improve things sometimes actually make things > worse. That's not the issue here. Do you want to run a kernel with known security problems, or one with "unknown potential problems." The latter is always the case, so please don't pick the known-insecure one, that's just foolish. greg k-h
