On 4/26/19 12:04 PM, Mario Smarduch wrote:
On 04/26/2019 09:01 AM, Mario Smarduch wrote:
On 04/26/2019 06:32 AM, Joe Lawrence wrote:
On 4/26/19 3:51 AM, Mario Smarduch wrote:
Hello,
I noticed that with livepatch the old and new functions can
run in parallel, but with kpatch the kpatch module load fails on
"activeness safety check".
Hi Mario,
I think you are referring to kpatch's use of stop_machine() to quiesce
the system before patching vs. livepatch's consistency model[1] and its
convergence on a patched state?
Am I missing something with livepatch? Is there another way to
build and run it?
AFAIK there is no way around the consistency model with livepatch. What
you might be able to do is hack the kpatch-build to utilize the kpatch
support module instead of the livepatch API. (Look for the
$KPATCH_MODULE code in the kpatch-build script.) I don't think this use
case is very well tested and in fact, the kpatch.ko will be deprecated
eventually.
Do you have a particular case in mind which the livepatch consistency
model does not support?
I'm also wondering if this still holds today
https://groups.google.com/forum/#!topic/fa.linux.kernel/FhGaKm0Vhm0
Sorry about that link is bad.
Hi,
Since Jiri posted the kGraft patches [1], I wanted to share an
alternative live patching solution called kpatch, which is something
we've been working on at Red Hat for quite a while.
The kernel piece of it ("kpatch core module") is completely
self-contained in a GPL module. It compiles and works without needing
to change any kernel code, and in fact we already have it working fine
with Fedora 20 [2] without any distro kernel patches needed. We'd
definitely like to see it (or some combination of it and kGraft) merged
into Linux.
This patch set is for the core module, which provides the kernel
infrastructure for kpatch. It has a kpatch_register() interface which
allows kernel modules ("patch modules") to replace old functions with
new functions which are loaded with the modules.
This is all still true, but remember that kpatch-build will create
livepatch style patches if it detects that the target kernel supports
livepatch.
There are also some user space tools [2] which aren't included in this
patch set, which magically generate binary patch modules from source
diffs, and manage the loading and unloading of these modules. I didn't
include them here because I think we should agree on what the kernel
parts should look like before trying to discuss the user space tools
(and whether they should be in-tree).
kpatch vs kGraft
----------------
I think the biggest difference between kpatch and kGraft is how they
ensure that the patch is applied atomically and safely.
Since both are (now) using the livepatch kernel API, patch application
is basically the same. I would say that the differences now reside in
how the livepatch kernel modules are constructed: kpatch-build does a
binary diff on ELF objects while kGraft constructs a kernel module
entirely? from source code.
kpatch checks the backtraces of all tasks in stop_machine() to ensure
that no instances of the old function are running when the new function
is applied. I think the biggest downside of this approach is that
stop_machine() has to idle all other CPUs during the patching process,
so it inserts a small amount of latency (a few ms on an idle system).
Instead, kGraft uses per-task consistency: each task either sees the old
version or the new version of the function. This gives a consistent
view with respect to functions, but _not_ data, because the old and new
functions are allowed to run simultaneously and share data. This could
be dangerous if a patch changes how a function uses a data structure.
The new function could make a data change that the old function wasn't
expecting.
I'll reference the "3. Consistency model" section of
Documentation/livepatch/livepatch.txt again for to current state of the
kernel livepatch API. I would describe it as the kGraft model, but with
elements of the kpatch (stack checking for idle process) added to create
a better hybrid approach.
With kpatch, that's not an issue because all the functions are patched
at the same time. So kpatch is safer with respect to data interactions.
Other advantages of the kpatch stop_machine() approach:
- IMO, the kpatch code is much simpler than kGraft. The implementation
is very straightforward and is completely self-contained. It requires
zero changes to the kernel.
I would agree that kpatch is/was conceptually simpler to understand.
(However a new TAINT_KPATCH flag would be a good idea, and we do
anticipate some minor changes to kprobes and ftrace for better
compatibility.)
- The use of stop_machine() will enable an important not-yet-implemented
feature to call a user-supplied callback function at loading time
which can be used to atomically update data structures when applying a
patch. I don't see how such a feature would be possible with the
kGraft approach.
Kpatch first implemented load and unload hooks. These were ported to
the livepatch API in the form of pre and post patch and unpatch
callbacks as described by Documentation/livepatch/callbacks.txt.
- kpatch applies patches immediately without having to send signals to
sleeping processes, and without having to hope that those processes
handle the signal appropriately.
I believe this is still accurate.
- kpatch's patching behavior is more deterministic because
stop_machine() ensures that all tasks are sleeping and interrupts are
disabled when the patching occurs.
kpatch is deterministic at the expense of sometimes not completing a
patch operation, while a similar livepatch might eventually "converge"
on a patched state.
- kpatch already supports other cool features like:
- removing patches and rolling back to the original functions
- atomically replacing existing patches
- incremental patching
- loading multiple patch modules
Livepatch supports all of these (atomic replace as of 5.1)
-- Joe
_______________________________________________
kpatch mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/kpatch
