On 30/11/2021 14:49, Michał Górny via lldb-dev wrote:
Hi,I'm working on a FreeBSD-sponsored project aiming at improving LLDB's support for debugging FreeBSD kernel to achieve feature parity with KGDB. As a part of that, I'd like to improve LLDB's ability of working with kernel coredumps ("vmcores"), plus add the ability to read kernel memory via special character device /dev/mem. The FreeBSD kernel supports two coredump formats that are of interest to us: 1. The (older) "full memory" coredumps that use an ELF container. 2. The (newer) minidumps that dump only the active memory and use a custom format. At this point, LLDB recognizes the ELF files but doesn't handle them correctly, and outright rejects the FreeBSD minidump format. In both cases some additional logic is required. This is because kernel coredumps contain physical contents of memory, and for user convenience the debugger needs to be able to read memory maps from the physical memory and use them to translate virtual addresses to physical addresses. Unless I'm mistaken, the rationale for using this format is that coredumps are -- after all -- usually created when something goes wrong with the kernel. In that case, we want the process for dumping core to be as simple as possible, and coredumps need to be small enough to fit in swap space (that's where they're being usually written). The complexity of memory translation should then naturally fall into userspace processes used to debug them. FreeBSD (following Solaris and other BSDs) provides a helper libkvm library that can be used by userspace programs to access both coredumps and running kernel memory. Additionally, we have split the routines related to coredumps and made them portable to other operating systems via libfbsdvmcore [1]. We have also included a program that can convert minidump into a debugger-compatible ELF core file. We'd like to discuss the possible approaches to integrating this additional functionality to LLDB. At this point, our goal is to make it possible for LLDB to correctly read memory from coredumps and live system. Plan A: new FreeBSDKernel plugin ================================ I think the preferable approach is to write a new plugin that would enable out-of-the-box support for the new functions in LLDB. The plugin would be based on using both libraries. When available, libfbsdvmcore will be used as the primary provider for vmcore support on all operating systems. Additionally, libkvm will be usable on FreeBSD as a fallback provider for coredump support, and as the provider of live memory support. support using system-installed libfbsdvmcore to read coredumps and libkvm to read coredumps (as a fallback) and to read live memory. The two main challenges with this approach are: 1) "Full memory" vmcores are currently recognized by LLDB's elf-core plugin. I haven't investigated LLDB's plugin architecture in detail yet but I think the cleanest solution here would be to teach elf-core to distinguish and reject FreeBSD vmcores, in order to have the new plugin handle them. 2) How to integrate "live kernel" support into the current user interface? I don't think we should make major UI modifications to support this specific case but I'd also like to avoid gross hacks. My initial thought is to allow specifying "/dev/mem" as core path, that would match how libkvm handles it. Nevertheless, I think this is the cleanest approach and I think we should go with it if possible. Plan B: GDB Remote Protocol-based wrapper ========================================= If we cannot integrate FreeBSD vmcore support into LLDB directly, I think the next best approach is to create a minimal GDB Remote Protocol server for it. The rough idea is that the server implements the minimal subset of the protocol necessary for LLDB to connect, and implements memory read operations via the aforementioned libraries. The advantage of this solution is that it is still relatively clean and can be implemented outside LLDB. It still provides quite good performance but probably requires more work than the alternatives and does not provide out-of-box support in LLDB. Plan C: converting vmcores ========================== Our final option, one that's practically implemented already is to require the user to explicitly convert vmcore into an ELF core understood by LLDB. This is the simplest solution but it has a few drawbacks: 1. it is limited to minidumps right now 2. it requires storing a converted coredump which means that at least temporarily it doubles the disk space use 3. there is possibility of cleanly supporting live kernel memory operations and therefore reaching KGDB feature parity We could create a wrapper to avoid having users convert coredumps explicitly but well, we think other options are better. WDYT? [1] https://github.com/Moritz-Systems/libfbsdvmcore
Having a new plugin for opening these kinds of core files seems reasonable to me. The extra dependency is unfortunate, but I guess that's what we have to work with.
Since this is still an elf file, you might still be able to use yaml2obj to create realistic-looking core files (it's support for program headers is pretty good these days).
The live kernel debugging sounds... scary. Can you explain how would this actually work? Like, what would be the supported operations? I presume you won't be able to actually "stop" the kernel, but what will you actually be able to do?
Without more details its hard for me to say whether this should be a separate plugin (or even a server, since that's what we tend to use for live debugging).
pl _______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
