Re: [PATCH v2 0/9] Function Granular KASLR
On Thu, 2020-05-21 at 16:30 -0700, Kees Cook wrote: > On Fri, May 22, 2020 at 12:26:30AM +0200, Thomas Gleixner wrote: > > I understand how this is supposed to work, but I fail to find an > > explanation how all of this is preserving the text subsections we > > have, > > i.e. .kprobes.text, .entry.text ...? > > I had the same question when I first started looking at earlier > versions > of this series! :) Thanks for responding - clearly I do need to update the cover letter and documentation. > > > I assume that the functions in these subsections are reshuffled > > within > > their own randomized address space so that __xxx_text_start and > > __xxx_text_end markers still make sense, right? > > No, but perhaps in the future. Right now, they are entirely ignored > and > left untouched. The current series only looks at the sections > produced > by -ffunction-sections, which is to say only things named > ".text.$thing" > (e.g. ".text.func1", ".text.func2"). Since the "special" text > sections > in the kernel are named ".$thing.text" (specifically to avoid other > long-standing linker logic that does similar .text.* pattern matches) > they get ignored by FGKASLR right now too. > > Even more specifically, they're ignored because all of these special > _input_ sections are actually manually collected by the linker script > into the ".text" _output_ section, which FGKASLR ignores -- it can > only > randomize the final output sections (and has no basic block > visibility > into the section contents), so everything in .text is untouched. > Because > these special sections are collapsed into the single .text output > section is why we've needed the __$thing_start and __$thing_end > symbols > manually constructed by the linker scripts: we lose input section > location/size details once the linker collects them into an output > section. > > > I'm surely too tired to figure it out from the patches, but you > > really > > want to explain that very detailed for mere mortals who are not > > deep > > into this magic as you are. > > Yeah, it's worth calling out, especially since it's an area of future > work -- I think if we can move the special sections out of .text into > their own output sections that can get randomized and we'll have > section > position/size information available without the manual ..._start/_end > symbols. But this will require work with the compiler and linker to > get > what's needed relative to -ffunction-sections, teach the kernel about > the new way of getting _start/_end, etc etc. > > So, before any of that, just .text.* is a good first step, and after > that I think next would be getting .text randomized relative to the > other > .text.* sections (IIUC, it is entirely untouched currently, so only > the > standard KASLR base offset moves it around). Only after that do we > start > poking around trying to munge the special section contents (which > requires use solving a few problems simultaneously). :) > That's right - we keep .text unrandomized, so any special sections that are collected into .text are still in their original layout. Like you said, they still get to take advantage of normal KASLR (base address randomization).
Re: [PATCH v2 0/9] Function Granular KASLR
Kees, Kees Cook writes: > On Fri, May 22, 2020 at 12:26:30AM +0200, Thomas Gleixner wrote: >> I understand how this is supposed to work, but I fail to find an >> explanation how all of this is preserving the text subsections we have, >> i.e. .kprobes.text, .entry.text ...? > > I had the same question when I first started looking at earlier versions > of this series! :) > >> I assume that the functions in these subsections are reshuffled within >> their own randomized address space so that __xxx_text_start and >> __xxx_text_end markers still make sense, right? > > No, but perhaps in the future. Right now, they are entirely ignored and > left untouched. I'm fine with that restriction, but for a moment I got worried that this might screw up explicit subsections. This really want's to be clearly expressed in the cover letter and the changelogs so that such questions don't arise again. > So, before any of that, just .text.* is a good first step, and after > that I think next would be getting .text randomized relative to the other > .text.* sections (IIUC, it is entirely untouched currently, so only the > standard KASLR base offset moves it around). Only after that do we start > poking around trying to munge the special section contents (which > requires use solving a few problems simultaneously). :) Thanks for the detailed explanation! tglx
Re: [PATCH v2 0/9] Function Granular KASLR
On Fri, May 22, 2020 at 12:26:30AM +0200, Thomas Gleixner wrote: > I understand how this is supposed to work, but I fail to find an > explanation how all of this is preserving the text subsections we have, > i.e. .kprobes.text, .entry.text ...? I had the same question when I first started looking at earlier versions of this series! :) > I assume that the functions in these subsections are reshuffled within > their own randomized address space so that __xxx_text_start and > __xxx_text_end markers still make sense, right? No, but perhaps in the future. Right now, they are entirely ignored and left untouched. The current series only looks at the sections produced by -ffunction-sections, which is to say only things named ".text.$thing" (e.g. ".text.func1", ".text.func2"). Since the "special" text sections in the kernel are named ".$thing.text" (specifically to avoid other long-standing linker logic that does similar .text.* pattern matches) they get ignored by FGKASLR right now too. Even more specifically, they're ignored because all of these special _input_ sections are actually manually collected by the linker script into the ".text" _output_ section, which FGKASLR ignores -- it can only randomize the final output sections (and has no basic block visibility into the section contents), so everything in .text is untouched. Because these special sections are collapsed into the single .text output section is why we've needed the __$thing_start and __$thing_end symbols manually constructed by the linker scripts: we lose input section location/size details once the linker collects them into an output section. > I'm surely too tired to figure it out from the patches, but you really > want to explain that very detailed for mere mortals who are not deep > into this magic as you are. Yeah, it's worth calling out, especially since it's an area of future work -- I think if we can move the special sections out of .text into their own output sections that can get randomized and we'll have section position/size information available without the manual ..._start/_end symbols. But this will require work with the compiler and linker to get what's needed relative to -ffunction-sections, teach the kernel about the new way of getting _start/_end, etc etc. So, before any of that, just .text.* is a good first step, and after that I think next would be getting .text randomized relative to the other .text.* sections (IIUC, it is entirely untouched currently, so only the standard KASLR base offset moves it around). Only after that do we start poking around trying to munge the special section contents (which requires use solving a few problems simultaneously). :) -- Kees Cook
Re: [PATCH v2 0/9] Function Granular KASLR
Kristen, Kristen Carlson Accardi writes: sorry for not following this work and a maybe stupid question. > Proposed Improvement > > This patch set proposes adding function reordering on top of the existing > KASLR base address randomization. The over-arching objective is incremental > improvement over what we already have. It is designed to work in combination > with the existing solution. The implementation is really pretty simple, and > there are 2 main area where changes occur: > > * Build time > > GCC has had an option to place functions into individual .text sections for > many years now. This option can be used to implement function reordering at > load time. The final compiled vmlinux retains all the section headers, which > can be used to help find the address ranges of each function. Using this > information and an expanded table of relocation addresses, individual text > sections can be suffled immediately after decompression. Some data tables > inside the kernel that have assumptions about order require re-sorting > after being updated when applying relocations. In order to modify these > tables, > a few key symbols are excluded from the objcopy symbol stripping process for > use after shuffling the text segments. I understand how this is supposed to work, but I fail to find an explanation how all of this is preserving the text subsections we have, i.e. .kprobes.text, .entry.text ...? I assume that the functions in these subsections are reshuffled within their own randomized address space so that __xxx_text_start and __xxx_text_end markers still make sense, right? I'm surely too tired to figure it out from the patches, but you really want to explain that very detailed for mere mortals who are not deep into this magic as you are. Thanks, tglx
Re: [PATCH v2 0/9] Function Granular KASLR
On Thu, May 21, 2020 at 09:56:31AM -0700, Kristen Carlson Accardi wrote: > Changes in v2: > -- > * Fix to address i386 build failure > * Allow module reordering patch to be configured separately so that > arm (or other non-x86_64 arches) can take advantage of module function > reordering. This support has not be tested by me, but smoke tested by > Ard Biesheuvel on arm. > * Fix build issue when building on arm as reported by > Ard Biesheuvel > * minor chages for certain checkpatch warnings and review feedback. I successfully built and booted this on top of linux-next. For my builds I include: CONFIG_LOCK_DEBUGGING_SUPPORT=y CONFIG_PROVE_LOCKING=y CONFIG_DEBUG_RT_MUTEXES=y CONFIG_DEBUG_SPINLOCK=y CONFIG_DEBUG_MUTEXES=y CONFIG_DEBUG_WW_MUTEX_SLOWPATH=y CONFIG_DEBUG_RWSEMS=y CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_LOCKDEP=y CONFIG_DEBUG_ATOMIC_SLEEP=y which catches various things. One of those (I assume either CONFIG_LOCKDEP or CONFIG_DEBUG_MUTEXES) has found an issue with kallsyms: [ 34.112989] [ cut here ] [ 34.113560] WARNING: CPU: 1 PID: 1997 at kernel/module.c:260 module_assert_mutex+0x29/0x30 [ 34.114479] Modules linked in: [ 34.114831] CPU: 1 PID: 1997 Comm: grep Tainted: GW 5.7.0-rc6-next-20200519+ #497 ... [ 34.128556] Call Trace: [ 34.128867] module_kallsyms_on_each_symbol+0x1d/0xa0 [ 34.130238] kallsyms_on_each_symbol+0xbd/0xd0 [ 34.131642] kallsyms_sorted_open+0x3f/0x70 [ 34.132160] proc_reg_open+0x99/0x180 [ 34.133222] do_dentry_open+0x176/0x400 [ 34.134182] vfs_open+0x2d/0x30 [ 34.134579] do_open.isra.0+0x2a0/0x410 [ 34.135058] path_openat+0x175/0x620 [ 34.135506] do_filp_open+0x91/0x100 [ 34.136912] do_sys_openat2+0x210/0x2d0 [ 34.137388] do_sys_open+0x46/0x80 [ 34.137818] __x64_sys_openat+0x20/0x30 [ 34.138288] do_syscall_64+0x55/0x1d0 [ 34.138720] entry_SYSCALL_64_after_hwframe+0x49/0xb3 Triggering it is easy, just "cat /proc/kallsyms" (and I'd note that I don't even have any modules loaded). Tracking this down, it just looks like kallsyms needs to hold a lock while sorting: diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c index 558963b275ec..182b16a6079b 100644 --- a/kernel/kallsyms.c +++ b/kernel/kallsyms.c @@ -772,7 +772,9 @@ static int kallsyms_sorted_open(struct inode *inode, struct file *file) INIT_LIST_HEAD(list); + mutex_lock(_mutex); ret = kallsyms_on_each_symbol(get_all_symbol_name, list); + mutex_unlock(_mutex); if (ret != 0) return ret; This fixes it for me. Everything else seems to be lovely. :) Nice work! -- Kees Cook
[PATCH v2 0/9] Function Granular KASLR
Function Granular Kernel Address Space Layout Randomization (fgkaslr) - This patch set is an implementation of finer grained kernel address space randomization. It rearranges your kernel code at load time on a per-function level granularity, with only around a second added to boot time. Changes in v2: -- * Fix to address i386 build failure * Allow module reordering patch to be configured separately so that arm (or other non-x86_64 arches) can take advantage of module function reordering. This support has not be tested by me, but smoke tested by Ard Biesheuvel on arm. * Fix build issue when building on arm as reported by Ard Biesheuvel * minor chages for certain checkpatch warnings and review feedback. Patches to objtool are included because they are dependencies for this patchset, however they have been submitted by their maintainer separately. Background -- KASLR was merged into the kernel with the objective of increasing the difficulty of code reuse attacks. Code reuse attacks reused existing code snippets to get around existing memory protections. They exploit software bugs which expose addresses of useful code snippets to control the flow of execution for their own nefarious purposes. KASLR moves the entire kernel code text as a unit at boot time in order to make addresses less predictable. The order of the code within the segment is unchanged - only the base address is shifted. There are a few shortcomings to this algorithm. 1. Low Entropy - there are only so many locations the kernel can fit in. This means an attacker could guess without too much trouble. 2. Knowledge of a single address can reveal the offset of the base address, exposing all other locations for a published/known kernel image. 3. Info leaks abound. Finer grained ASLR has been proposed as a way to make ASLR more resistant to info leaks. It is not a new concept at all, and there are many variations possible. Function reordering is an implementation of finer grained ASLR which randomizes the layout of an address space on a function level granularity. We use the term "fgkaslr" in this document to refer to the technique of function reordering when used with KASLR, as well as finer grained KASLR in general. Proposed Improvement This patch set proposes adding function reordering on top of the existing KASLR base address randomization. The over-arching objective is incremental improvement over what we already have. It is designed to work in combination with the existing solution. The implementation is really pretty simple, and there are 2 main area where changes occur: * Build time GCC has had an option to place functions into individual .text sections for many years now. This option can be used to implement function reordering at load time. The final compiled vmlinux retains all the section headers, which can be used to help find the address ranges of each function. Using this information and an expanded table of relocation addresses, individual text sections can be suffled immediately after decompression. Some data tables inside the kernel that have assumptions about order require re-sorting after being updated when applying relocations. In order to modify these tables, a few key symbols are excluded from the objcopy symbol stripping process for use after shuffling the text segments. Some highlights from the build time changes to look for: The top level kernel Makefile was modified to add the gcc flag if it is supported. Currently, I am applying this flag to everything it is possible to randomize. Anything that is written in C and is a function is randomized. Future work could turn off this flags for selected files or even entire subsystems, although obviously at the cost of security. The relocs tool is updated to add relative relocations. This information previously wasn't included because it wasn't necessary when moving the entire .text segment as a unit. A new file was created to contain a list of symbols that objcopy should keep. We use those symbols at load time as described below. * Load time The boot kernel was modified to parse the vmlinux elf file after decompression to check for our interesting symbols that we kept, and to look for any .text.* sections to randomize. We then shuffle the sections and update any tables that need to be updated or resorted. The existing code which updated relocation addresses was modified to account for not just a fixed delta from the load address, but the offset that the function section was moved to. This requires inspection of each address to see if it was impacted by a randomization. We use a bsearch to make this less horrible on performance. In order to hide our new layout, symbols reported through /proc/kallsyms will be sorted by name alphabetically rather than by address. Security Considerations --- The objective of this patch set
