On 09/12/2020 03:26, Brendan Conoboy wrote:
On Tue, Dec 8, 2020 at 4:19 PM Pete Biggs <p...@biggs.org.uk> wrote:
On Tue, 2020-12-08 at 17:54 -0500, Matthew Miller wrote:
On Tue, Dec 08, 2020 at 03:15:17PM +0000, Pete Biggs wrote:
"CentOS will become the developer playground"
This one is categorically not the case. Even Fedora isn't a developer
playground. Everything landing in CentOS Stream is actually *planned*
(with
emphasis intentional) to go in a future RHEL release.
It's all the talk of SIGs and developing and testing and that Stream
will be the centerpiece of that. That's what I meant.
I don't know if I'd call SIGs a playground, but they're certainly an
important venue for communities to explore variations.
Previously, all the development around RHEL releases was done in
secret,
in
the Red Hat black box. Now it's out of the box and can be watched.
There
may
be some launch pains, but I expect the average quality of an update
hitting
CentOS Stream to be very high.
I don't get that from the documents released today. If Stream is *not*
a test-bed, then surely the code that appears in Stream must be fully
formed in secret behind the scenes first. Yes, it will appear piecemeal
rather than in one big chunk, but it has been categorically denied that
Stream is not a RHEL 8.n+1 beta and is more a RHEL 8.n+1 RC/rolling
release.
I think maybe some of the nervousness about CentOS Stream comes from RHEL
seeming opacity on its development model. As one of the architects of
our
development process I'd be happy to field questions. I'll start by
making
a two points in case they're in any way unclear:
1. Everything that goes into RHEL lands upstream first, then the patches
are backported into the RHEL releases.
2. Most of the work we do or plan on doing is in bugzilla.redhat.com.
If
you go into the RHEL8 product queue today and file a bug you'll see
"CentOS
Stream" as a "Version" where an issue is encountered.
I think what a lot of people are concerned about is the rolling-release
aspect of this. There will be no definitive versioning of CentOS in the
future - all you will be able to say is "fully updated" and it won't be
possible to slot a CentOS system in to exactly match a RHEL version.
Will third party RPMs built against RHEL 8.x be installable on a CentOS
8 Stream system? The answer is surely "it depends", but there are a lot
of hardware vendors that target drivers to RHEL releases, which may
well make CentOS non-viable for hardware that doesn't have drivers
built in to the kernel.
Generally if they follow the ABI guidelines I would expect it to work.
Those are here:
https://access.redhat.com/articles/rhel8-abi-compatibility
For loadable kernel modules there's a kernel ABI.
Hi Brendan,
This point is *critical*, so I apologise in advance for the lengthy
post, *you* are breaking the kernel ABI between RHEL8 and Stream.
One of the main unique selling points of RHEL is the stability of it's
kernel ABI. No other distro provides this. The very nature of rolling
kernel updates in Stream breaks the kernel ABI and breaks compatibility
between RHEL8 and Stream. What works on RHEL8 may not work on Stream. At
the kernel level, the two products diverge in fundamental compatibility
and are not compatible, are no longer the same.
How bad is this divergence / breakage? Well, we know the kernel ABI will
change from time to time, almost exclusively at new point releases due
to the massive backporting work that goes into the RHEL kernel. And this
is fine, we know it's coming, we know when it's coming, and we can plan
for it's impact. It's a price well worth paying.
To put this in context, at elrepo I currently help maintain around 50
3rd party kernel driver packages for RHEL8. When RH released RHEL8.3,
every single package in our repository broke due to changes in the
kernel ABI in the 6 month period between RHEL8.2 and RHEL8.3. It's not
ideal, but we accept it as a price we pay for the otherwise excellent
stability of the kernel ABI during the proceeding 6 months. As I said
above, we know it's coming, we know when it's coming, and we can plan
for it.
Now contrast that with Stream. Every kernel update in Stream has the
potential to break the kernel ABI causing packages built for RHEL to
break. We don't know when that may happen (only that it will), we don't
know how often it will happen, we have no idea which packages it will
break. and we have no way to fix it. Consequently, elrepo has been
unable to support Stream kernels.
It is not just elrepo's users that the Stream kernels will affect. All
OEM hardware manufacturers releasing 3rd party driver rpms as part of
Red Hat's Driver Update Programme or otherwise will be similarly
affected, and their driver updates will not be applicable to or
compatible with CentOS Stream. In fact, RHEL's own driver update
packages will likely need rebuilding against each Stream kernel update,
although presumably you are in the unique position of being able to
integrate any changes directly into the Stream kernel negating the need
for driver updates and to mitigating the impact to yourselves.
Luckily the solution is simple - do not include rolling beta kernel
updates as part of Stream - rather just ship regular RHEL8 kernel
updates that retain kernel ABI compatibility rather than breaking
compatibility of the Stream product. I see no compelling reason to ship
rolling kernel updates to Stream - it's not like I will ever be able to
commit a patch and have you accept/merge it, and it's totally useless
for me as a developer as I can't develop anything against a constantly
moving target, and anyone who does require access has always had that
available through their Red Hat OEM Partner/Developer accounts. Shipping
rolling kernel updates in Stream adds no value and simply breaks
compatibility between RHEL8 and Stream at the most fundamental level. If
you are able to retain kernel ABI compatibility between RHEL8 and Stream
kernels, then we (and other OEMs) will be able to continue to support
Stream users, otherwise Stream users will have to look to alternative
solutions.
