Hi Sean, On Tue, Aug 18 2026, Sean Christopherson wrote:
> On Tue, Aug 18, 2026, Pratyush Yadav wrote: >> Hi Sean, >> >> On Mon, Aug 17 2026, Sean Christopherson wrote: >> >> > On Sat, Aug 15, 2026, Pratyush Yadav wrote: >> >> On Wed, Aug 12 2026, Sean Christopherson wrote: >> >> > On Wed, Aug 12, 2026, Pratyush Yadav wrote: >> >> So with live update, we don't need to keep backwards compatibility in >> >> the ABI forever. Of course, it is good to minimize changes, but we have >> >> more freedom to change it. >> > >> > Ah. So this is the heart of the disconnect. I very strongly disagree >> > with the >> > statement that live update doesn't need to support backwards >> > compatibility. I >> > can totally believe that the folks working on live update are ok breaking >> > backwards >> > compatibility because their use cases are "fine" with such breakage. And >> > I can >> > also believe live update as an upstream kernel feature being developed by >> > those >> > same folks is also ok with breaking backwards compatibility. >> > >> > But with my upstream KVM maintainer hat on, I am not ok with that. I did >> > not agree >> > to support a world where KVM is allowed to break backwards compatibility, >> > so long >> > as it's done "carefully" or whatever. If y'all want to deal with the >> > resulting >> > complexity, that's fine by me, but you'll be doing it without KVM. >> >> Let's step back a bit. I don't think backwards compatibility in the >> _ABI_ is all that important in live update's context. What is important >> is that our users are able to live update from kernel version X to Y. >> The line format the kernel uses to describe its state is an >> implementation detail. Our users never see it. > > The rule is thou shalt not break userspace. Whether or not the breakage is > the > result of an explicit ABI change is irrelevant. > >> The high level idea is that when you need to make a change to the ABI so >> the kernel can better describe the objects/resources/files it is >> passing, you create a new version of the ABI and allow transitions from >> older versions. >> >> Say you make some changes and need an ABI version v5. You don't get rid >> of v4. You keep it around. LUO core will facilitate picking the right >> version for the next kernel. So it will be possible to seamlessly >> upgrade your kernel that speaks v4 to a new kernel that speaks v4 and >> v5. Now this kernel can start speaking v5 if its successor speaks v5 >> too. And so on for going to v6 and v7, etc. >> >> After a "reasonable" time given for upgrades, you deprecate v4. v4 has >> been around long enough and our users have had a chance to go to kernels >> speaking newer versions. That's when you remove the code for v4 from the >> kernel. >> >> We can argue what "reasonable" means, but the core idea stays. >> >> It will still be possible to go back to v4 or earlier, but you'd need an >> extra stop along the way. And of course, vendors can keep a wider >> support matrix downstream if they see the need for it. >> >> Does this idea of "backwards compatibility" sound acceptable to you, at >> least at a high level? > > No. > > It's probably fine for Google and other large companies that tightly control > their > kernels and use cases, and have the resources to juggle the resulting > complexity, > e.g. have kernel engineers on staff to track feature and dependencies, > coordinate > and plan kernel upgrades, etc. > > It's not acceptable for upstream, where downstream consumers often run a > distro > kernel, have much more varied use cases, and don't always have a horde of > kernel > engineers on staff to help them thread the needle you describe above. And if > supporting live update as a general feature for all users of the kernel isn't > being factored into design considerations, then that needs to change, > otherwise > this is all dead in the water. > > I also don't see the point. Maintaining a rigid save/restore ABI is annoying, > but it's not _hard_ (or at least, not _that_ hard), especially if there's a > set > of well-documented best known practices that subsystems can follow, e.g. so > that > individual subsystems don't need to learn painful lessons first-hand. I > genuinely > believe that maintaining the version hell you describe above would be more > costly > in the long run than simply committing to full backwards compatibility within > a > given subsystem. I can imagine that enumerating what subsystems' information > is > in the payload will require a different scheme, but for a given subsystem, I > don't > see any reason to aim for anything less than full backwards compatibility. Let's say for argument's sake that we commit for a fully stable backwards compatible ABI. Even then, you have to deal with multiple ABI versions. Live update's ABI is more complex compared to KVM's save/restore ABI. For the KVM save/restore uAPI, you are largely describing architectural state like CPU registers, etc. These things don't evolve as fast and more or less stay the same. Live update needs to describe the state of kernel objects. These are more complex and evolve faster. For example, say you merge guest_memfd preservation today. Some time later, someone comes up with a more efficient data structure to track the folios in the file. You _have_ to make a backwards-incompatible ABI change to use this data structure. Or say you add a new memory backend (like the HugeTLB patches in flight). That likely will need a different ABI to describe the state of the guest_memfd. So you will end up with multiple ABI versions that aren't always backwards compatible. If you refuse that idea too, then KVM live update will be dead in the water for a different reason. It will be damn near useless because it can't keep up with an evolving subsystem. Now once you get multiple ABI versions and you can seamlessly go from old to new one, say you have a version that was superseded 5 years ago. It would be completely reasonable to say that this version is old enough and no one should be going from a 5 year old kernel to a modern one. So you deprecate this ABI. Deprecating old unused uAPIs is not unprecedented. I think we are better off formalizing this deprecation period from the get go. A somewhat tangential example is BPF kfuncs. My BPF program that works in kernel X might not work in kernel Y because the kfunc has changed or been removed. The argument they make in kfuncs.rst is that kfuncs "provide a kernel <-> kernel API, and thus are not bound by any of the strict stability restrictions associated with kernel <-> user UAPIs". For LUO as well, this is a kernel -> kernel API. Users can also still do a regular kexec or reboot. They just won't get the performance optimization of LUO. Regardless of if you agree with the last bit about deprecating old versions, ABIs evolving with the subsystem is a ground reality of live update and it would be foolish to think we can do with only backwards-compatible ABI changes forever. -- Regards, Pratyush Yadav
