Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Tue, 2020-05-19 at 18:04 -0700, Andy Lutomirski wrote: > On Mon, May 18, 2020 at 6:35 PM Andy Lutomirski wrote: > > [...] > > > On May 18, 2020, at 5:38 PM, Dave Hansen wrote: > > > [...] > > > The sadistic parts of selftests/x86 come from real bugs. Either bugs > > > where the kernel fell over, or where behavior changed that broke apps. > > > I'd suggest doing some research on where that particular test case came > > > from. Find the author of the test, look at the changelogs. > > > > > > If this is something that a real app does, this is a problem. If it's a > > > sadistic test that Andy L added because it was an attack vector against > > > the entry code, it's a different story. > > > > There are quite a few tests that do these horrible things in there. IN my > > personal opinion, sigreturn.c is one of the most important tests we have — > > it does every horrible thing to the entry code that I thought of and that I > > could come up with a way of doing. We have been saved from regressing many > > times by these tests. CET, and especially the CPL0 version of CET, is its > > own set of entry horror, and we need to keep these tests working. > > > > I assume the basic issue is that we call raise(), the context magically > > changes to 32-bit, but SSP has a 64-bit value, and horrors happen. So I > > think two things need to happen: > > > > 1. Someone needs to document what happens when IRET tries to put a 64-bit > > value into SSP but CS is compat. Because Intel has plenty of history of > > doing colossally broken things here. IOW you could easily be hitting a > > hardware design problem, not a software issue per se. > > > > 2. The test needs to work. Assuming the hardware doesn’t do something > > utterly broken, either the 32-bit code needs to be adjusted to avoid any > > CALL > > or RET, or you need to write a little raise_on_32bit_shstk() func that > > switches to an SSP that fits in 32 bits, calls raise(), and switches back. > > From memory, I didn’t think there was a CALl or RET, so I’m guessing that > > SSP is getting truncated when we round trip through CPL3 compat mode and > > the result is that the kernel invoked the signal handler with the wrong > > SSP. Whoops. > > > > Following up here, I think this needs attention from the H/W architects. > > From the SDM: > > SYSRET and SYSEXIT: > > IF ShadowStackEnabled(CPL) > SSP ← IA32_PL3_SSP; > FI; > > IRET: > > IF ShadowStackEnabled(CPL) > IF CPL = 3 > THEN tempSSP ← IA32_PL3_SSP; FI; > IF ((EFER.LMA AND CS.L) = 0 AND tempSSP[63:32] != 0) > THEN #GP(0); FI; > SSP ← tempSSP > > The semantics of actually executing in compat mode with SSP >= 2^32 > are unclear. If nothing else, VM exit will save the full SSP and a > subsequent VM entry will fail. Here is what I got after talking to the architect. If the guest is in 32-bit mode, but its VM guest state SSP field is 64-bit, the CPU only uses the lower 32 bits. The SDM currently states a consistency check of the guest SSP field, but that will be removed in the next version. Upon VM entry, the CPU only requires the guest SSP to be pseudo-canonical like the RIP and RSP. > I don't know what the actual effect of operand-size-32 SYSRET or > SYSEXIT with too big a PL3_SSP will be, but I think it needs to be > documented. Ideally it will not put the CPU in an invalid state. > Ideally it will also not fault, because SYSRET faults in particular > are fatal unless the vector uses IST, and please please please don't > force more ISTs on anyone. On SYSRET/SYSEXIT to a 32-bit context, the CPU only uses the lower 32 bits of the user-mode SSP, and will not go into an invalid state and will not fault. The SDM will be explicit about this. Yu-cheng > > So I think we may need to put this entire series on hold until we get > some answers, because I suspect we're going to have a nice little root > hole otherwise.
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On 22/05/2020 17:49, Peter Zijlstra wrote: > On Sat, May 16, 2020 at 03:09:22PM +0100, Andrew Cooper wrote: > >> Sadly, the same is not true for kernel shadow stacks. >> >> SSP is 0 after SYSCALL, SYSENTER and CLRSSBSY, and you've got to be >> careful to re-establish the shadow stack before a CALL, interrupt or >> exception tries pushing a word onto the shadow stack at 0xfff8. > Oh man, I can only imagine the joy that brings to #NM and friends :-( Establishing a supervisor shadow stack for the first time involves a large leap of faith, even by usual x86 standards. You need to have prepared MSR_PL0_SSP with correct mappings and supervisor tokens, such that when you enable CR4.CET and MSR_S_CET.SHSTK_EN, your SETSSBSY instruction succeeds at its atomic "check the token and set the busy bit" shadow stack access. Any failure here tends to be a triple fault, and I didn't get around to figuring out why #DF wasn't taken cleanly. You also need to have prepared MSR_IST_SSP beforehand with the IST shadow stack pointers matching any IST configuration in the IDT, lest a NMI ruins your day on the instruction boundary before SETSSBSY. A less obvious side effect of these "windows with an SSP of 0" is that you're now forced to use IST for all non-maskable interrupts/exceptions, even if you choose not to use SYSCALL, and you no longer need IST to remove the risks of a userspace privilege escalation, and would prefer not to use IST because of its problematic reentrancy characteristics. For anyone counting the number of IST-necessary vectors across all potential configurations in modern hardware, its #DB, NMI, #DF, #MC, #VE, #HV, #VC and #SX, and an architectural limit of 7. There are several other amusing aspects, such as iret-to-self needing to use call-oriented-programming to keep itself shadow-stack-safe, or the fact that IRET to user mode doesn't fault if it fails to clear the supervisor busy bit, instead leaving you to double fault at some point in the future at the next syscall/interrupt/exception because the stack is still busy. ~Andrew P.S. For anyone interested, https://lore.kernel.org/xen-devel/20200501225838.9866-1-andrew.coop...@citrix.com/T/#u for getting supervisor shadow stacks working on Xen, which is far simpler to manage than Linux. I do not envy whomever has the fun of trying to make this work for Linux.
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Sat, May 16, 2020 at 03:09:22PM +0100, Andrew Cooper wrote: > Sadly, the same is not true for kernel shadow stacks. > > SSP is 0 after SYSCALL, SYSENTER and CLRSSBSY, and you've got to be > careful to re-establish the shadow stack before a CALL, interrupt or > exception tries pushing a word onto the shadow stack at 0xfff8. Oh man, I can only imagine the joy that brings to #NM and friends :-(
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Mon, May 18, 2020 at 6:35 PM Andy Lutomirski wrote: > > > > > On May 18, 2020, at 5:38 PM, Dave Hansen wrote: > > > > On 5/18/20 4:47 PM, Yu-cheng Yu wrote: > >>> On Fri, 2020-05-15 at 19:53 -0700, Yu-cheng Yu wrote: > >>> On Fri, 2020-05-15 at 16:56 -0700, Dave Hansen wrote: > On 5/15/20 4:29 PM, Yu-cheng Yu wrote: > > [...] > > I have run them with CET enabled. All of them pass, except for the > > following: > > Sigreturn from 64-bit to 32-bit fails, because shadow stack is at a > > 64-bit > > address. This is understandable. > [...] > One a separate topic: You ran the selftests and one failed. This is a > *MASSIVE* warning sign. It should minimally be described in your cover > letter, and accompanied by a fix to the test case. It is absolutely > unacceptable to introduce a kernel feature that causes a test to fail. > You must either fix your kernel feature or you fix the test. > > This code can not be accepted until this selftests issue is rectified. > >> The x86/sigreturn test constructs 32-bit ldt entries, and does sigreturn > >> from > >> 64-bit to 32-bit context. We do not have a way to construct a static > >> 32-bit > >> shadow stack. > > > > Why? What's the limiting factor? Hardware architecture? Something in > > the kernel? > > > >> Why do we want that? I think we can simply run the test with CET > >> disabled. > > > > The sadistic parts of selftests/x86 come from real bugs. Either bugs > > where the kernel fell over, or where behavior changed that broke apps. > > I'd suggest doing some research on where that particular test case came > > from. Find the author of the test, look at the changelogs. > > > > If this is something that a real app does, this is a problem. If it's a > > sadistic test that Andy L added because it was an attack vector against > > the entry code, it's a different story. > > There are quite a few tests that do these horrible things in there. IN my > personal opinion, sigreturn.c is one of the most important tests we have — it > does every horrible thing to the entry code that I thought of and that I > could come up with a way of doing. We have been saved from regressing many > times by these tests. CET, and especially the CPL0 version of CET, is its > own set of entry horror, and we need to keep these tests working. > > I assume the basic issue is that we call raise(), the context magically > changes to 32-bit, but SSP has a 64-bit value, and horrors happen. So I > think two things need to happen: > > 1. Someone needs to document what happens when IRET tries to put a 64-bit > value into SSP but CS is compat. Because Intel has plenty of history of doing > colossally broken things here. IOW you could easily be hitting a hardware > design problem, not a software issue per se. > > 2. The test needs to work. Assuming the hardware doesn’t do something utterly > broken, either the 32-bit code needs to be adjusted to avoid any CALL > or RET, or you need to write a little raise_on_32bit_shstk() func that > switches to an SSP that fits in 32 bits, calls raise(), and switches back. > From memory, I didn’t think there was a CALl or RET, so I’m guessing that SSP > is getting truncated when we round trip through CPL3 compat mode and the > result is that the kernel invoked the signal handler with the wrong SSP. > Whoops. > Following up here, I think this needs attention from the H/W architects. >From the SDM: SYSRET and SYSEXIT: IF ShadowStackEnabled(CPL) SSP ← IA32_PL3_SSP; FI; IRET: IF ShadowStackEnabled(CPL) IF CPL = 3 THEN tempSSP ← IA32_PL3_SSP; FI; IF ((EFER.LMA AND CS.L) = 0 AND tempSSP[63:32] != 0) THEN #GP(0); FI; SSP ← tempSSP The semantics of actually executing in compat mode with SSP >= 2^32 are unclear. If nothing else, VM exit will save the full SSP and a subsequent VM entry will fail. I don't know what the actual effect of operand-size-32 SYSRET or SYSEXIT with too big a PL3_SSP will be, but I think it needs to be documented. Ideally it will not put the CPU in an invalid state. Ideally it will also not fault, because SYSRET faults in particular are fatal unless the vector uses IST, and please please please don't force more ISTs on anyone. So I think we may need to put this entire series on hold until we get some answers, because I suspect we're going to have a nice little root hole otherwise.
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
> On May 18, 2020, at 5:38 PM, Dave Hansen wrote: > > On 5/18/20 4:47 PM, Yu-cheng Yu wrote: >>> On Fri, 2020-05-15 at 19:53 -0700, Yu-cheng Yu wrote: >>> On Fri, 2020-05-15 at 16:56 -0700, Dave Hansen wrote: On 5/15/20 4:29 PM, Yu-cheng Yu wrote: > [...] > I have run them with CET enabled. All of them pass, except for the > following: > Sigreturn from 64-bit to 32-bit fails, because shadow stack is at a 64-bit > address. This is understandable. [...] One a separate topic: You ran the selftests and one failed. This is a *MASSIVE* warning sign. It should minimally be described in your cover letter, and accompanied by a fix to the test case. It is absolutely unacceptable to introduce a kernel feature that causes a test to fail. You must either fix your kernel feature or you fix the test. This code can not be accepted until this selftests issue is rectified. >> The x86/sigreturn test constructs 32-bit ldt entries, and does sigreturn from >> 64-bit to 32-bit context. We do not have a way to construct a static 32-bit >> shadow stack. > > Why? What's the limiting factor? Hardware architecture? Something in > the kernel? > >> Why do we want that? I think we can simply run the test with CET >> disabled. > > The sadistic parts of selftests/x86 come from real bugs. Either bugs > where the kernel fell over, or where behavior changed that broke apps. > I'd suggest doing some research on where that particular test case came > from. Find the author of the test, look at the changelogs. > > If this is something that a real app does, this is a problem. If it's a > sadistic test that Andy L added because it was an attack vector against > the entry code, it's a different story. There are quite a few tests that do these horrible things in there. IN my personal opinion, sigreturn.c is one of the most important tests we have — it does every horrible thing to the entry code that I thought of and that I could come up with a way of doing. We have been saved from regressing many times by these tests. CET, and especially the CPL0 version of CET, is its own set of entry horror, and we need to keep these tests working. I assume the basic issue is that we call raise(), the context magically changes to 32-bit, but SSP has a 64-bit value, and horrors happen. So I think two things need to happen: 1. Someone needs to document what happens when IRET tries to put a 64-bit value into SSP but CS is compat. Because Intel has plenty of history of doing colossally broken things here. IOW you could easily be hitting a hardware design problem, not a software issue per se. 2. The test needs to work. Assuming the hardware doesn’t do something utterly broken, either the 32-bit code needs to be adjusted to avoid any CALL or RET, or you need to write a little raise_on_32bit_shstk() func that switches to an SSP that fits in 32 bits, calls raise(), and switches back. From memory, I didn’t think there was a CALl or RET, so I’m guessing that SSP is getting truncated when we round trip through CPL3 compat mode and the result is that the kernel invoked the signal handler with the wrong SSP. Whoops. > > I don't personally know the background, but the changelogs can help you > find the person that does.
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On 5/18/20 4:47 PM, Yu-cheng Yu wrote: > On Fri, 2020-05-15 at 19:53 -0700, Yu-cheng Yu wrote: >> On Fri, 2020-05-15 at 16:56 -0700, Dave Hansen wrote: >>> On 5/15/20 4:29 PM, Yu-cheng Yu wrote: [...] I have run them with CET enabled. All of them pass, except for the following: Sigreturn from 64-bit to 32-bit fails, because shadow stack is at a 64-bit address. This is understandable. >>> [...] >>> One a separate topic: You ran the selftests and one failed. This is a >>> *MASSIVE* warning sign. It should minimally be described in your cover >>> letter, and accompanied by a fix to the test case. It is absolutely >>> unacceptable to introduce a kernel feature that causes a test to fail. >>> You must either fix your kernel feature or you fix the test. >>> >>> This code can not be accepted until this selftests issue is rectified. > The x86/sigreturn test constructs 32-bit ldt entries, and does sigreturn from > 64-bit to 32-bit context. We do not have a way to construct a static 32-bit > shadow stack. Why? What's the limiting factor? Hardware architecture? Something in the kernel? > Why do we want that? I think we can simply run the test with CET > disabled. The sadistic parts of selftests/x86 come from real bugs. Either bugs where the kernel fell over, or where behavior changed that broke apps. I'd suggest doing some research on where that particular test case came from. Find the author of the test, look at the changelogs. If this is something that a real app does, this is a problem. If it's a sadistic test that Andy L added because it was an attack vector against the entry code, it's a different story. I don't personally know the background, but the changelogs can help you find the person that does.
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Fri, 2020-05-15 at 19:53 -0700, Yu-cheng Yu wrote: > On Fri, 2020-05-15 at 16:56 -0700, Dave Hansen wrote: > > On 5/15/20 4:29 PM, Yu-cheng Yu wrote: > > > [...] > > > I have run them with CET enabled. All of them pass, except for the > > > following: > > > Sigreturn from 64-bit to 32-bit fails, because shadow stack is at a 64-bit > > > address. This is understandable. > > [...] > > One a separate topic: You ran the selftests and one failed. This is a > > *MASSIVE* warning sign. It should minimally be described in your cover > > letter, and accompanied by a fix to the test case. It is absolutely > > unacceptable to introduce a kernel feature that causes a test to fail. > > You must either fix your kernel feature or you fix the test. > > > > This code can not be accepted until this selftests issue is rectified. The x86/sigreturn test constructs 32-bit ldt entries, and does sigreturn from 64-bit to 32-bit context. We do not have a way to construct a static 32-bit shadow stack. Why do we want that? I think we can simply run the test with CET disabled. Yu-cheng
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On 5/18/20 7:01 AM, H.J. Lu wrote: >> Could some of this information be added to the documentation, please? >> It would also be nice to have some more details about how apps end up >> using ARCH_X86_CET_STATUS. Why would they care that CET is on? > CET software spec is at > > https://gitlab.com/x86-psABIs/x86-64-ABI/-/wikis/Intel-CET-extension > > My CET presentation at 2018 LPC is at > > https://www.linuxplumbersconf.org/event/2/contributions/147/attachments/72/83/CET-LPC-2018.pdf > > I am working on an updated CET presentation for 2020 LPC. Let me know > if you want to see the early draft. There's a lot of great information in there! However, please remember that presentations are no substitute for old fashioned documentation in the kernel tree and changelogs. The fact that we need to lean on them to answer basic questions about new interfaces is not a great sign.
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Mon, 2020-05-18 at 06:41 -0700, Dave Hansen wrote: > On 5/15/20 7:53 PM, Yu-cheng Yu wrote: > > On Fri, 2020-05-15 at 16:56 -0700, Dave Hansen wrote: > > > What's my recourse as an end user? I want to run my app and turn off > > > CET for that app. How can I do that? > > > > GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-IBT > > Like I mentioned to H.J., this is something that we need to at least > acknowledge the existence of in the changelog and probably even the > Documentation/. Sure. I will do that. > > > > > > I think you're saying that the CET-enabled binary would do > > > > > arch_setup_elf_property() when it was first exec()'d. Later, it could > > > > > use the new prctl(ARCH_X86_CET_DISABLE) to disable its shadow stack, > > > > > then fork() and the child would not be using CET. Right? > > > > > > > > > > What is ARCH_X86_CET_DISABLE used for, anyway? > > > > > > > > Both the parent and the child can do ARCH_X86_CET_DISABLE, if CET is > > > > not locked. > > > > > > Could you please describe a real-world example of why > > > ARCH_X86_CET_DISABLE exists? What kinds of apps will use it, or *are* > > > using it? Why was it created in the first place? > > > > Currently, ld-linux turns off CET if the binary being loaded does not > > support > > CET. > > Great! Could this please be immortalized in the documentation for the > prctl()? Yes. > > > > > > > > Does this *code* work? Could you please indicate which JITs have > > > > > > > been > > > > > > > enabled to use the code in this series? How much of the new ABI > > > > > > > is in use? > > > > > > > > > > > > JIT does not necessarily use all of the ABI. The JIT changes > > > > > > mainly fix stack > > > > > > frames and insert ENDBRs. I do not work on JIT. What I found is > > > > > > LLVM JIT fixes > > > > > > are tested and in the master branch. Sljit fixes are in the > > > > > > release. > > > > > > > > > > Huh, so who is using the new prctl() ABIs? > > > > > > > > Any code can use the ABI, but JIT code CET-enabling part mostly do not > > > > use these > > > > new prctl()'s, except, probably to get CET status. > > > > > > Which applications specifically are going to use the new prctl()s which > > > this series adds? How are they going to use them? > > > > > > "Any code can use them" is not a specific enough answer. > > > > We have four arch_ptctl() calls. ARCH_X86_CET_DISABLE and > > ARCH_X86_CET_LOCK are > > used by ld-linux. ARCH_X86_CET_STATUS are used in many places to determine > > if > > CET is on. ARCH_X86_CET_ALLOC_SHSTK is used in ucontext related handling, > > but > > it can be use by any application to switch shadow stacks. > > Could some of this information be added to the documentation, please? > It would also be nice to have some more details about how apps end up > using ARCH_X86_CET_STATUS. Why would they care that CET is on? Yes. Yu-cheng
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Mon, May 18, 2020 at 6:41 AM Dave Hansen wrote: > > On 5/15/20 7:53 PM, Yu-cheng Yu wrote: > > On Fri, 2020-05-15 at 16:56 -0700, Dave Hansen wrote: > >> What's my recourse as an end user? I want to run my app and turn off > >> CET for that app. How can I do that? > > > > GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-IBT > > Like I mentioned to H.J., this is something that we need to at least > acknowledge the existence of in the changelog and probably even the > Documentation/. > > I think you're saying that the CET-enabled binary would do > arch_setup_elf_property() when it was first exec()'d. Later, it could > use the new prctl(ARCH_X86_CET_DISABLE) to disable its shadow stack, > then fork() and the child would not be using CET. Right? > > What is ARCH_X86_CET_DISABLE used for, anyway? > >>> > >>> Both the parent and the child can do ARCH_X86_CET_DISABLE, if CET is > >>> not locked. > >> > >> Could you please describe a real-world example of why > >> ARCH_X86_CET_DISABLE exists? What kinds of apps will use it, or *are* > >> using it? Why was it created in the first place? > > > > Currently, ld-linux turns off CET if the binary being loaded does not > > support > > CET. > > Great! Could this please be immortalized in the documentation for the > prctl()? > > >> Does this *code* work? Could you please indicate which JITs have been > >> enabled to use the code in this series? How much of the new ABI is in > >> use? > > > > JIT does not necessarily use all of the ABI. The JIT changes mainly > > fix stack > > frames and insert ENDBRs. I do not work on JIT. What I found is LLVM > > JIT fixes > > are tested and in the master branch. Sljit fixes are in the release. > > Huh, so who is using the new prctl() ABIs? > >>> > >>> Any code can use the ABI, but JIT code CET-enabling part mostly do not > >>> use these > >>> new prctl()'s, except, probably to get CET status. > >> > >> Which applications specifically are going to use the new prctl()s which > >> this series adds? How are they going to use them? > >> > >> "Any code can use them" is not a specific enough answer. > > > > We have four arch_ptctl() calls. ARCH_X86_CET_DISABLE and > > ARCH_X86_CET_LOCK are > > used by ld-linux. ARCH_X86_CET_STATUS are used in many places to determine > > if > > CET is on. ARCH_X86_CET_ALLOC_SHSTK is used in ucontext related handling, > > but > > it can be use by any application to switch shadow stacks. > > Could some of this information be added to the documentation, please? > It would also be nice to have some more details about how apps end up > using ARCH_X86_CET_STATUS. Why would they care that CET is on? CET software spec is at https://gitlab.com/x86-psABIs/x86-64-ABI/-/wikis/Intel-CET-extension My CET presentation at 2018 LPC is at https://www.linuxplumbersconf.org/event/2/contributions/147/attachments/72/83/CET-LPC-2018.pdf I am working on an updated CET presentation for 2020 LPC. Let me know if you want to see the early draft. -- H.J.
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On 5/15/20 7:53 PM, Yu-cheng Yu wrote: > On Fri, 2020-05-15 at 16:56 -0700, Dave Hansen wrote: >> What's my recourse as an end user? I want to run my app and turn off >> CET for that app. How can I do that? > > GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-IBT Like I mentioned to H.J., this is something that we need to at least acknowledge the existence of in the changelog and probably even the Documentation/. I think you're saying that the CET-enabled binary would do arch_setup_elf_property() when it was first exec()'d. Later, it could use the new prctl(ARCH_X86_CET_DISABLE) to disable its shadow stack, then fork() and the child would not be using CET. Right? What is ARCH_X86_CET_DISABLE used for, anyway? >>> >>> Both the parent and the child can do ARCH_X86_CET_DISABLE, if CET is >>> not locked. >> >> Could you please describe a real-world example of why >> ARCH_X86_CET_DISABLE exists? What kinds of apps will use it, or *are* >> using it? Why was it created in the first place? > > Currently, ld-linux turns off CET if the binary being loaded does not support > CET. Great! Could this please be immortalized in the documentation for the prctl()? >> Does this *code* work? Could you please indicate which JITs have been >> enabled to use the code in this series? How much of the new ABI is in >> use? > > JIT does not necessarily use all of the ABI. The JIT changes mainly fix > stack > frames and insert ENDBRs. I do not work on JIT. What I found is LLVM > JIT fixes > are tested and in the master branch. Sljit fixes are in the release. Huh, so who is using the new prctl() ABIs? >>> >>> Any code can use the ABI, but JIT code CET-enabling part mostly do not use >>> these >>> new prctl()'s, except, probably to get CET status. >> >> Which applications specifically are going to use the new prctl()s which >> this series adds? How are they going to use them? >> >> "Any code can use them" is not a specific enough answer. > > We have four arch_ptctl() calls. ARCH_X86_CET_DISABLE and ARCH_X86_CET_LOCK > are > used by ld-linux. ARCH_X86_CET_STATUS are used in many places to determine if > CET is on. ARCH_X86_CET_ALLOC_SHSTK is used in ucontext related handling, but > it can be use by any application to switch shadow stacks. Could some of this information be added to the documentation, please? It would also be nice to have some more details about how apps end up using ARCH_X86_CET_STATUS. Why would they care that CET is on?
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On 5/15/20 7:51 PM, H.J. Lu wrote: > On Fri, May 15, 2020 at 4:56 PM Dave Hansen wrote: >> Let's say we have an app doing silly things like retpolines. (Lots of >> app do lots of silly things). It gets compiled in a distro but never >> runs on a system with CET. The app gets run for the first time on a >> system with CET. App goes boom. Not init, just some random app, say >> /usr/bin/ldapsearch. > > I designed and implemented CET toolchain and run-time in such a way > for it very difficult to happen. Basically, CET won't be enabled on such > an app. Would you care to share any specifics about how this is implemented? That would be great information to include in the kernel documentation because it informs us about the reasons why we don't need a kernel-based "kill switch". >> What's my recourse as an end user? I want to run my app and turn off >> CET for that app. How can I do that? > > The CET OS I designed turns CET off for you and you don't have to do > anything. OK, cool! Could you share some of the specifics about how it does that? >> Is it possible with the patches in this series to run a single- >> threaded binary which was has GNU_PROPERTY_X86_FEATURE_1_SHSTK >> unset to run with shadow stack protection? > > Yes, you can. I added such capabilities for testing purpose. But > you application will crash as soon as there is a CET violation. My > CET software design is very flexible. Yu-cheng speficially referred to the: GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-IBT option. Is that the option you're talking about? >>> I have a quick test that checks shadow stack and ibt in both main program >>> and in >>> signals. Currently it is public on Github. If that is desired, I can >>> submit it >>> to the mailing list. >> >> Yes, that is desired. It must accompany this submission. It must also >> exercise all of the new ABIs. > > Our CET smoke test is for quick validation of CET OS, not just > kernel. It requires the complete CET implementation. It does > nothing if your OS isn't CET enabled. I think requiring the complete CET implementation to be present for this test to work is a mistake. We don't require anything other than an enabled kernel and the selftests that ship with that kernel. MPX required toolchain, library and compiler changes. Yet, we had a totally standalone kernel test that found real bugs. It sounds like this smoke test as it stands wouldn't be a great fit. But, that shouldn't discourage us from finding something that _is_ a good fit for the kernel-shipped selftests.
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On 16/05/2020 03:37, H.J. Lu wrote: > On Fri, May 15, 2020 at 5:13 PM Andrew Cooper > wrote: >> Finally seeing as the question was asked but not answered, it is >> actually quite easy to figure out whether shadow stacks are enabled in >> the current thread. >> >> mov $1, %eax >> rdsspd %eax > This is for 32-bit mode. It actually works for both, if all you need is a shstk yes/no check. Usually, you also want SSP in the yes case, so substitute rdsspq %rax as appropriate. (On a tangent - binutils mandating the D/Q suffixes is very irritating with mixed 32/64bit code because you have to #ifdef your instructions despite the register operands being totally unambiguous. Also, D is the wrong suffix for AT syntax, and should be L. Frankly - the Intel manuals are wrong and should not have the operand size suffix included in the opcode name, as they are consistent with all the other instructions in this regard.) > I use > > /* Check if shadow stack is in use. */ > xorl%esi, %esi > rdsspq %rsi > testq %rsi, %rsi > /* Normal return if shadow stack isn't in use. */ > je L(no_shstk) This is probably fine for user code, as I don't think it would be legitimate for shstk to be enabled, with SSP being 0. Sadly, the same is not true for kernel shadow stacks. SSP is 0 after SYSCALL, SYSENTER and CLRSSBSY, and you've got to be careful to re-establish the shadow stack before a CALL, interrupt or exception tries pushing a word onto the shadow stack at 0xfff8. It is a very good (lucky?) thing that frame is unmapped for other reasons, because this corner case does not protect against multiple threads/cores using the same shadow stack concurrently. ~Andrew
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Fri, May 15, 2020 at 4:56 PM Dave Hansen wrote: > > On 5/15/20 4:29 PM, Yu-cheng Yu wrote: > > On Fri, 2020-05-15 at 15:43 -0700, Dave Hansen wrote: > >> Basically, if there ends up being a bug in an app that violates the > >> shadow stack rules, the app is broken, period. The only recourse is to > >> have the kernel disable CET and reboot. > >> > >> Is that right? > > > > You must be talking about init or any of the system daemons, right? > > Assuming we let the app itself start CET with an arch_prctl(), why would > > that be > > different from the current approach? > > You're getting ahead of me a bit here. > > I'm actually not asking directly about the prctls() or advocating for a > different approach. The MPX approach of _requiring the app to make a > prctl() was actually pretty nasty because sometimes threads got created > before the prctl() could get called. Apps ended up inadvertently > half-MPX-enabled. Not fun. > > Let's say we have an app doing silly things like retpolines. (Lots of > app do lots of silly things). It gets compiled in a distro but never > runs on a system with CET. The app gets run for the first time on a > system with CET. App goes boom. Not init, just some random app, say > /usr/bin/ldapsearch. I designed and implemented CET toolchain and run-time in such a way for it very difficult to happen. Basically, CET won't be enabled on such an app. > What's my recourse as an end user? I want to run my app and turn off > CET for that app. How can I do that? The CET OS I designed turns CET off for you and you don't have to do anything. > Can a binary compiled without CET run CET-enabled code? > >>> > >>> Partially yes, but in reality somewhat difficult. > >> ... > >>> - If a not-CET application does fork(), and the child wants to turn on > >>> CET, it > >>> would be difficult to manage the stack frames, unless the child knows > >>> what is is > >>> doing. > >> > >> It might be hard to do, but it is possible with the patches you posted? > > > > It is possible to add an arch_prctl() to turn on CET. That is simple from > > the > > kernel's perspective, but difficult for the application. Once the app > > enables > > shadow stack, it has to take care not to return beyond the function call > > layers > > before that point. It can no longer do longjmp or ucontext swaps to > > anything > > before that point. It will also be complicated if the app enables shadow > > stack > > in a signal handler. > > Yu-cheng, I'm having a very hard time getting direct answers to my > questions. Could you endeavor to give succinct, direct answers? If you > want to give a longer, conditioned answer, that's great. But, I'd > appreciate if you could please focus first on clearly answering the > questions that I'm asking. > > Let me try again: > > Is it possible with the patches in this series to run a single- > threaded binary which was has GNU_PROPERTY_X86_FEATURE_1_SHSTK > unset to run with shadow stack protection? Yes, you can. I added such capabilities for testing purpose. But you application will crash as soon as there is a CET violation. My CET software design is very flexible. It can accommodate different requirements. We are working with 2 OSVs to enable CET in their OSes. So far we haven't run into any unexpected issues. > I think the answer is an unambiguous: "No". But I'd like to hear it > from you. > > >> I think you're saying that the CET-enabled binary would do > >> arch_setup_elf_property() when it was first exec()'d. Later, it could > >> use the new prctl(ARCH_X86_CET_DISABLE) to disable its shadow stack, > >> then fork() and the child would not be using CET. Right? > >> > >> What is ARCH_X86_CET_DISABLE used for, anyway? > > > > Both the parent and the child can do ARCH_X86_CET_DISABLE, if CET is > > not locked. > > Could you please describe a real-world example of why > ARCH_X86_CET_DISABLE exists? What kinds of apps will use it, or *are* > using it? Why was it created in the first place? > > >>> The JIT examples I mentioned previously run with CET enabled from the > >>> beginning. Do you have a reason to do this? In other words, if the JIT > >>> code > >>> needs CET, the app could have started with CET in the first place. > >> > >> Let's say I have a JIT'd sandbox. I want the sandbox to be > >> CET-protected, but the JIT engine itself not to be. > > > > I do not have any objections to this use case, but it needs some cautions as > > stated above. It will be much easier and cleaner if the sandbox is in a > > separate exec'ed task with CET on. > > OK, great suggestion! Could you do some research and look at the > various sandboxing techniques? Is imposing this requirement for a > separate exec'd task reasonable? Does it fit nicely with their existing > models? How about the Chrome browser and Firefox sandboxs? > > Does this *code* work? Could you please indicate which JITs have been > enabled to use
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Fri, 2020-05-15 at 16:56 -0700, Dave Hansen wrote: > On 5/15/20 4:29 PM, Yu-cheng Yu wrote: > > On Fri, 2020-05-15 at 15:43 -0700, Dave Hansen wrote: > > > Basically, if there ends up being a bug in an app that violates the > > > shadow stack rules, the app is broken, period. The only recourse is to > > > have the kernel disable CET and reboot. > > > > > > Is that right? > > > > You must be talking about init or any of the system daemons, right? > > Assuming we let the app itself start CET with an arch_prctl(), why would > > that be > > different from the current approach? > > You're getting ahead of me a bit here. > > I'm actually not asking directly about the prctls() or advocating for a > different approach. The MPX approach of _requiring the app to make a > prctl() was actually pretty nasty because sometimes threads got created > before the prctl() could get called. Apps ended up inadvertently > half-MPX-enabled. Not fun. > > Let's say we have an app doing silly things like retpolines. (Lots of > app do lots of silly things). It gets compiled in a distro but never > runs on a system with CET. The app gets run for the first time on a > system with CET. App goes boom. Not init, just some random app, say > /usr/bin/ldapsearch. > > What's my recourse as an end user? I want to run my app and turn off > CET for that app. How can I do that? GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-IBT > > > > > Can a binary compiled without CET run CET-enabled code? > > > > > > > > Partially yes, but in reality somewhat difficult. > > > ... > > > > - If a not-CET application does fork(), and the child wants to turn on > > > > CET, it > > > > would be difficult to manage the stack frames, unless the child knows > > > > what is is > > > > doing. > > > > > > It might be hard to do, but it is possible with the patches you posted? > > > > It is possible to add an arch_prctl() to turn on CET. That is simple from > > the > > kernel's perspective, but difficult for the application. Once the app > > enables > > shadow stack, it has to take care not to return beyond the function call > > layers > > before that point. It can no longer do longjmp or ucontext swaps to > > anything > > before that point. It will also be complicated if the app enables shadow > > stack > > in a signal handler. > > Yu-cheng, I'm having a very hard time getting direct answers to my > questions. Could you endeavor to give succinct, direct answers? If you > want to give a longer, conditioned answer, that's great. But, I'd > appreciate if you could please focus first on clearly answering the > questions that I'm asking. > > Let me try again: > > Is it possible with the patches in this series to run a single- > threaded binary which was has GNU_PROPERTY_X86_FEATURE_1_SHSTK > unset to run with shadow stack protection? > > I think the answer is an unambiguous: "No". But I'd like to hear it > from you. No! > > > I think you're saying that the CET-enabled binary would do > > > arch_setup_elf_property() when it was first exec()'d. Later, it could > > > use the new prctl(ARCH_X86_CET_DISABLE) to disable its shadow stack, > > > then fork() and the child would not be using CET. Right? > > > > > > What is ARCH_X86_CET_DISABLE used for, anyway? > > > > Both the parent and the child can do ARCH_X86_CET_DISABLE, if CET is > > not locked. > > Could you please describe a real-world example of why > ARCH_X86_CET_DISABLE exists? What kinds of apps will use it, or *are* > using it? Why was it created in the first place? Currently, ld-linux turns off CET if the binary being loaded does not support CET. > > > > The JIT examples I mentioned previously run with CET enabled from the > > > > beginning. Do you have a reason to do this? In other words, if the > > > > JIT code > > > > needs CET, the app could have started with CET in the first place. > > > > > > Let's say I have a JIT'd sandbox. I want the sandbox to be > > > CET-protected, but the JIT engine itself not to be. > > > > I do not have any objections to this use case, but it needs some cautions as > > stated above. It will be much easier and cleaner if the sandbox is in a > > separate exec'ed task with CET on. > > OK, great suggestion! Could you do some research and look at the > various sandboxing techniques? Is imposing this requirement for a > separate exec'd task reasonable? Does it fit nicely with their existing > models? How about the Chrome browser and Firefox sandboxs? I will check. > > > > > Does this *code* work? Could you please indicate which JITs have been > > > > > enabled to use the code in this series? How much of the new ABI is > > > > > in use? > > > > > > > > JIT does not necessarily use all of the ABI. The JIT changes mainly > > > > fix stack > > > > frames and insert ENDBRs. I do not work on JIT. What I found is LLVM > > > > JIT fixes > > > > are tested and in the master branch. Sljit fixes are in the
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Fri, May 15, 2020 at 5:13 PM Andrew Cooper wrote: > > On 15/05/2020 23:43, Dave Hansen wrote: > > On 5/15/20 2:33 PM, Yu-cheng Yu wrote: > >> On Fri, 2020-05-15 at 11:39 -0700, Dave Hansen wrote: > >>> On 5/12/20 4:20 PM, Yu-cheng Yu wrote: > >>> Can a binary compiled with CET run without CET? > >> Yes, but a few details: > >> > >> - The shadow stack is transparent to the application. A CET application > >> does > >> not have anything different from a non-CET application. However, if a CET > >> application uses any CET instructions (e.g. INCSSP), it must first check > >> if CET > >> is turned on. > >> - If an application is compiled for IBT, the compiler inserts ENDBRs at > >> branch > >> targets. These are nops if IBT is not on. > > I appreciate the detailed response, but it wasn't quite what I was > > asking. Let's ignore IBT for now and just talk about shadow stacks. > > > > An app compiled with the new ELF flags and running on a CET-enabled > > kernel and CPU will start off with shadow stacks allocated and enabled, > > right? It can turn its shadow stack off per-thread with the new prctl. > > But, otherwise, it's stuck, the only way to turn shadow stacks off at > > startup would be editing the binary. > > > > Basically, if there ends up being a bug in an app that violates the > > shadow stack rules, the app is broken, period. The only recourse is to > > have the kernel disable CET and reboot. > > > > Is that right? > > If I may interject with the experience of having got supervisor shadow > stacks working for Xen. > > Turning shadow stacks off is quite easy - clear MSR_U_CET.SHSTK_EN and > the shadow stack will stay in whatever state it was in, and you can > largely forget about it. (Of course, in a sandbox scenario, it would be > prudent to prevent the ability to disable shadow stacks.) > > Turning shadow stacks on is much more tricky. You cannot enable it in > any function you intend to return from, as the divergence between the > stack and shadow stack will constitute a control flow violation. > > > When it comes to binaries, you can reasonably arrange for clone() to > start a thread on a new stack/shstk, as you can prepare both stacks > suitably before execution starts. > > You cannot reasonably implement a system call for "turn shadow stacks on > for me", because you'll crash on the ret out of the VDSO from the system > call. It would be possible to conceive of an exec()-like system call > which is "discard my current stack, turn on shstk, and start me on this > new stack/shstk". > > In principle, with a pair of system calls to atomically manage the ststk > settings and stack switching, it might possible to construct a > `run_with_shstk_enabled(func, stack, shstk)` API which executes in the > current threads context and doesn't explode. > > Fork() is a problem when shadow stacks are disabled in the parent. The > moment shadow stacks are disabled, the regular stack diverges from the > shadow stack. A CET-enabled app which turns off shstk and then fork()'s > must have the child inherit the shstk-off property. If the child were > to start with shstk enabled, it would explode almost immediately due to > the parent's stack divergence which it inherited. > > > Finally seeing as the question was asked but not answered, it is > actually quite easy to figure out whether shadow stacks are enabled in > the current thread. > > mov $1, %eax > rdsspd %eax This is for 32-bit mode. I use /* Check if shadow stack is in use. */ xorl%esi, %esi rdsspq %rsi testq %rsi, %rsi /* Normal return if shadow stack isn't in use. */ je L(no_shstk) > cmp $1, %eax > je no_shstk > ... > no_shsk: > > rdssp is allocated from the hint nop encoding space, and the minimum > alignment of the shadow stack pointer is 4. On older parts, or with > shstk disabled (either at the system level, or for the thread), the $1 > will be preserved in %eax, while if CET is active, it will be clobbered > with something that has the bottom two bits clear. > > It turns out this is a lifesaver for codepaths (e.g. the NMI handler) > which need to use other CET instructions which aren't from the hint nop > space, and run before the BSP can set everything up. > > ~Andrew -- H.J.
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On 15/05/2020 23:43, Dave Hansen wrote: > On 5/15/20 2:33 PM, Yu-cheng Yu wrote: >> On Fri, 2020-05-15 at 11:39 -0700, Dave Hansen wrote: >>> On 5/12/20 4:20 PM, Yu-cheng Yu wrote: >>> Can a binary compiled with CET run without CET? >> Yes, but a few details: >> >> - The shadow stack is transparent to the application. A CET application does >> not have anything different from a non-CET application. However, if a CET >> application uses any CET instructions (e.g. INCSSP), it must first check if >> CET >> is turned on. >> - If an application is compiled for IBT, the compiler inserts ENDBRs at >> branch >> targets. These are nops if IBT is not on. > I appreciate the detailed response, but it wasn't quite what I was > asking. Let's ignore IBT for now and just talk about shadow stacks. > > An app compiled with the new ELF flags and running on a CET-enabled > kernel and CPU will start off with shadow stacks allocated and enabled, > right? It can turn its shadow stack off per-thread with the new prctl. > But, otherwise, it's stuck, the only way to turn shadow stacks off at > startup would be editing the binary. > > Basically, if there ends up being a bug in an app that violates the > shadow stack rules, the app is broken, period. The only recourse is to > have the kernel disable CET and reboot. > > Is that right? If I may interject with the experience of having got supervisor shadow stacks working for Xen. Turning shadow stacks off is quite easy - clear MSR_U_CET.SHSTK_EN and the shadow stack will stay in whatever state it was in, and you can largely forget about it. (Of course, in a sandbox scenario, it would be prudent to prevent the ability to disable shadow stacks.) Turning shadow stacks on is much more tricky. You cannot enable it in any function you intend to return from, as the divergence between the stack and shadow stack will constitute a control flow violation. When it comes to binaries, you can reasonably arrange for clone() to start a thread on a new stack/shstk, as you can prepare both stacks suitably before execution starts. You cannot reasonably implement a system call for "turn shadow stacks on for me", because you'll crash on the ret out of the VDSO from the system call. It would be possible to conceive of an exec()-like system call which is "discard my current stack, turn on shstk, and start me on this new stack/shstk". In principle, with a pair of system calls to atomically manage the ststk settings and stack switching, it might possible to construct a `run_with_shstk_enabled(func, stack, shstk)` API which executes in the current threads context and doesn't explode. Fork() is a problem when shadow stacks are disabled in the parent. The moment shadow stacks are disabled, the regular stack diverges from the shadow stack. A CET-enabled app which turns off shstk and then fork()'s must have the child inherit the shstk-off property. If the child were to start with shstk enabled, it would explode almost immediately due to the parent's stack divergence which it inherited. Finally seeing as the question was asked but not answered, it is actually quite easy to figure out whether shadow stacks are enabled in the current thread. mov $1, %eax rdsspd %eax cmp $1, %eax je no_shstk ... no_shsk: rdssp is allocated from the hint nop encoding space, and the minimum alignment of the shadow stack pointer is 4. On older parts, or with shstk disabled (either at the system level, or for the thread), the $1 will be preserved in %eax, while if CET is active, it will be clobbered with something that has the bottom two bits clear. It turns out this is a lifesaver for codepaths (e.g. the NMI handler) which need to use other CET instructions which aren't from the hint nop space, and run before the BSP can set everything up. ~Andrew
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On 5/15/20 4:29 PM, Yu-cheng Yu wrote: > On Fri, 2020-05-15 at 15:43 -0700, Dave Hansen wrote: >> Basically, if there ends up being a bug in an app that violates the >> shadow stack rules, the app is broken, period. The only recourse is to >> have the kernel disable CET and reboot. >> >> Is that right? > > You must be talking about init or any of the system daemons, right? > Assuming we let the app itself start CET with an arch_prctl(), why would that > be > different from the current approach? You're getting ahead of me a bit here. I'm actually not asking directly about the prctls() or advocating for a different approach. The MPX approach of _requiring the app to make a prctl() was actually pretty nasty because sometimes threads got created before the prctl() could get called. Apps ended up inadvertently half-MPX-enabled. Not fun. Let's say we have an app doing silly things like retpolines. (Lots of app do lots of silly things). It gets compiled in a distro but never runs on a system with CET. The app gets run for the first time on a system with CET. App goes boom. Not init, just some random app, say /usr/bin/ldapsearch. What's my recourse as an end user? I want to run my app and turn off CET for that app. How can I do that? Can a binary compiled without CET run CET-enabled code? >>> >>> Partially yes, but in reality somewhat difficult. >> ... >>> - If a not-CET application does fork(), and the child wants to turn on CET, >>> it >>> would be difficult to manage the stack frames, unless the child knows what >>> is is >>> doing. >> >> It might be hard to do, but it is possible with the patches you posted? > > It is possible to add an arch_prctl() to turn on CET. That is simple from the > kernel's perspective, but difficult for the application. Once the app enables > shadow stack, it has to take care not to return beyond the function call > layers > before that point. It can no longer do longjmp or ucontext swaps to anything > before that point. It will also be complicated if the app enables shadow > stack > in a signal handler. Yu-cheng, I'm having a very hard time getting direct answers to my questions. Could you endeavor to give succinct, direct answers? If you want to give a longer, conditioned answer, that's great. But, I'd appreciate if you could please focus first on clearly answering the questions that I'm asking. Let me try again: Is it possible with the patches in this series to run a single- threaded binary which was has GNU_PROPERTY_X86_FEATURE_1_SHSTK unset to run with shadow stack protection? I think the answer is an unambiguous: "No". But I'd like to hear it from you. >> I think you're saying that the CET-enabled binary would do >> arch_setup_elf_property() when it was first exec()'d. Later, it could >> use the new prctl(ARCH_X86_CET_DISABLE) to disable its shadow stack, >> then fork() and the child would not be using CET. Right? >> >> What is ARCH_X86_CET_DISABLE used for, anyway? > > Both the parent and the child can do ARCH_X86_CET_DISABLE, if CET is > not locked. Could you please describe a real-world example of why ARCH_X86_CET_DISABLE exists? What kinds of apps will use it, or *are* using it? Why was it created in the first place? >>> The JIT examples I mentioned previously run with CET enabled from the >>> beginning. Do you have a reason to do this? In other words, if the JIT >>> code >>> needs CET, the app could have started with CET in the first place. >> >> Let's say I have a JIT'd sandbox. I want the sandbox to be >> CET-protected, but the JIT engine itself not to be. > > I do not have any objections to this use case, but it needs some cautions as > stated above. It will be much easier and cleaner if the sandbox is in a > separate exec'ed task with CET on. OK, great suggestion! Could you do some research and look at the various sandboxing techniques? Is imposing this requirement for a separate exec'd task reasonable? Does it fit nicely with their existing models? How about the Chrome browser and Firefox sandboxs? Does this *code* work? Could you please indicate which JITs have been enabled to use the code in this series? How much of the new ABI is in use? >>> >>> JIT does not necessarily use all of the ABI. The JIT changes mainly fix >>> stack >>> frames and insert ENDBRs. I do not work on JIT. What I found is LLVM JIT >>> fixes >>> are tested and in the master branch. Sljit fixes are in the release. >> >> Huh, so who is using the new prctl() ABIs? > > Any code can use the ABI, but JIT code CET-enabling part mostly do not use > these > new prctl()'s, except, probably to get CET status. Which applications specifically are going to use the new prctl()s which this series adds? How are they going to use them? "Any code can use them" is not a specific enough answer. Where are the selftests/ for this new ABI? Were you planning on submitting any with this series? >>>
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Fri, 2020-05-15 at 15:43 -0700, Dave Hansen wrote: > On 5/15/20 2:33 PM, Yu-cheng Yu wrote: > > On Fri, 2020-05-15 at 11:39 -0700, Dave Hansen wrote: > > > On 5/12/20 4:20 PM, Yu-cheng Yu wrote: > > > Can a binary compiled with CET run without CET? > > > > Yes, but a few details: > > > > - The shadow stack is transparent to the application. A CET application > > does > > not have anything different from a non-CET application. However, if a CET > > application uses any CET instructions (e.g. INCSSP), it must first check if > > CET > > is turned on. > > - If an application is compiled for IBT, the compiler inserts ENDBRs at > > branch > > targets. These are nops if IBT is not on. > > I appreciate the detailed response, but it wasn't quite what I was > asking. Let's ignore IBT for now and just talk about shadow stacks. > > An app compiled with the new ELF flags and running on a CET-enabled > kernel and CPU will start off with shadow stacks allocated and enabled, > right? It can turn its shadow stack off per-thread with the new prctl. > But, otherwise, it's stuck, the only way to turn shadow stacks off at > startup would be editing the binary. > > Basically, if there ends up being a bug in an app that violates the > shadow stack rules, the app is broken, period. The only recourse is to > have the kernel disable CET and reboot. > > Is that right? You must be talking about init or any of the system daemons, right? Assuming we let the app itself start CET with an arch_prctl(), why would that be different from the current approach? > > > Can a binary compiled without CET run CET-enabled code? > > > > Partially yes, but in reality somewhat difficult. > ... > > - If a not-CET application does fork(), and the child wants to turn on CET, > > it > > would be difficult to manage the stack frames, unless the child knows what > > is is > > doing. > > It might be hard to do, but it is possible with the patches you posted? It is possible to add an arch_prctl() to turn on CET. That is simple from the kernel's perspective, but difficult for the application. Once the app enables shadow stack, it has to take care not to return beyond the function call layers before that point. It can no longer do longjmp or ucontext swaps to anything before that point. It will also be complicated if the app enables shadow stack in a signal handler. > I think you're saying that the CET-enabled binary would do > arch_setup_elf_property() when it was first exec()'d. Later, it could > use the new prctl(ARCH_X86_CET_DISABLE) to disable its shadow stack, > then fork() and the child would not be using CET. Right? > > What is ARCH_X86_CET_DISABLE used for, anyway? Both the parent and the child can do ARCH_X86_CET_DISABLE, if CET is not locked. > > The JIT examples I mentioned previously run with CET enabled from the > > beginning. Do you have a reason to do this? In other words, if the JIT > > code > > needs CET, the app could have started with CET in the first place. > > Let's say I have a JIT'd sandbox. I want the sandbox to be > CET-protected, but the JIT engine itself not to be. I do not have any objections to this use case, but it needs some cautions as stated above. It will be much easier and cleaner if the sandbox is in a separate exec'ed task with CET on. > > - If you are asking about dlopen(), the library will have the same setting > > as > > the main application. Do you have any reason to have a library running with > > CET, but the application does not have CET? > > Sure, using old binaries. That's why IBT has a legacy bitmap and things > like MPX had ways of jumping into old non-enabled binaries. If the app has CET, but libs do not, then bitmap can help. If the app does not have CET, we don't run the libs with CET, right? This is the case right now. > > > Can different threads in a process have different CET enabling state? > > > > Yes, if the parent starts with CET, children can turn it off. > > How would that work, though? clone() by default will copy the parent > xsave state, which means it will be CET-enabled, which means it needs a > shadow stack. So, if I want a CET-free child thread, I need to clone(), > then turn CET off, then free the shadow stack? Yes, the child itself turns off CET. > > > Does this *code* work? Could you please indicate which JITs have been > > > enabled to use the code in this series? How much of the new ABI is in > > > use? > > > > JIT does not necessarily use all of the ABI. The JIT changes mainly fix > > stack > > frames and insert ENDBRs. I do not work on JIT. What I found is LLVM JIT > > fixes > > are tested and in the master branch. Sljit fixes are in the release. > > Huh, so who is using the new prctl() ABIs? Any code can use the ABI, but JIT code CET-enabling part mostly do not use these new prctl()'s, except, probably to get CET status. > > > Where are the selftests/ for this new ABI? Were you planning on > > > submitting any
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On 5/15/20 2:33 PM, Yu-cheng Yu wrote: > On Fri, 2020-05-15 at 11:39 -0700, Dave Hansen wrote: >> On 5/12/20 4:20 PM, Yu-cheng Yu wrote: >> Can a binary compiled with CET run without CET? > > Yes, but a few details: > > - The shadow stack is transparent to the application. A CET application does > not have anything different from a non-CET application. However, if a CET > application uses any CET instructions (e.g. INCSSP), it must first check if > CET > is turned on. > - If an application is compiled for IBT, the compiler inserts ENDBRs at branch > targets. These are nops if IBT is not on. I appreciate the detailed response, but it wasn't quite what I was asking. Let's ignore IBT for now and just talk about shadow stacks. An app compiled with the new ELF flags and running on a CET-enabled kernel and CPU will start off with shadow stacks allocated and enabled, right? It can turn its shadow stack off per-thread with the new prctl. But, otherwise, it's stuck, the only way to turn shadow stacks off at startup would be editing the binary. Basically, if there ends up being a bug in an app that violates the shadow stack rules, the app is broken, period. The only recourse is to have the kernel disable CET and reboot. Is that right? >> Can a binary compiled without CET run CET-enabled code? > > Partially yes, but in reality somewhat difficult. ... > - If a not-CET application does fork(), and the child wants to turn on CET, it > would be difficult to manage the stack frames, unless the child knows what is > is > doing. It might be hard to do, but it is possible with the patches you posted? I think you're saying that the CET-enabled binary would do arch_setup_elf_property() when it was first exec()'d. Later, it could use the new prctl(ARCH_X86_CET_DISABLE) to disable its shadow stack, then fork() and the child would not be using CET. Right? What is ARCH_X86_CET_DISABLE used for, anyway? > The JIT examples I mentioned previously run with CET enabled from the > beginning. Do you have a reason to do this? In other words, if the JIT code > needs CET, the app could have started with CET in the first place. Let's say I have a JIT'd sandbox. I want the sandbox to be CET-protected, but the JIT engine itself not to be. > - If you are asking about dlopen(), the library will have the same setting as > the main application. Do you have any reason to have a library running with > CET, but the application does not have CET? Sure, using old binaries. That's why IBT has a legacy bitmap and things like MPX had ways of jumping into old non-enabled binaries. >> Can different threads in a process have different CET enabling state? > > Yes, if the parent starts with CET, children can turn it off. How would that work, though? clone() by default will copy the parent xsave state, which means it will be CET-enabled, which means it needs a shadow stack. So, if I want a CET-free child thread, I need to clone(), then turn CET off, then free the shadow stack? >> Does this *code* work? Could you please indicate which JITs have been >> enabled to use the code in this series? How much of the new ABI is in use? > > JIT does not necessarily use all of the ABI. The JIT changes mainly fix stack > frames and insert ENDBRs. I do not work on JIT. What I found is LLVM JIT > fixes > are tested and in the master branch. Sljit fixes are in the release. Huh, so who is using the new prctl() ABIs? >> Where are the selftests/ for this new ABI? Were you planning on >> submitting any with this series? > > The ABI is more related to the application side, and therefore most suitable > for > GLIBC unit tests. I was mostly concerned with the kernel selftests. The things in tools/testing/selftests/x86 in the kernel tree. > The more complicated areas such as pthreads, signals, ucontext, > fork() are all included there. I have been constantly running these > tests without any problems. I can provide more details if testing is > the concern. For something this complicated, with new kernel ABIs, we need an in-kernel sefltest. MPX was not that much different from this feature. It required a boatload of compiler and linker changes to function. Yet, there was a simple in-kernel test for it that didn't require *any* of that big pile of toolchain bits. Is there a reason we don't have one of those for CET?
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Fri, 2020-05-15 at 11:39 -0700, Dave Hansen wrote: > On 5/12/20 4:20 PM, Yu-cheng Yu wrote: > > On Wed, 2020-04-29 at 16:02 -0700, Yu-cheng Yu wrote: > > > On Wed, 2020-04-29 at 15:53 -0700, Dave Hansen wrote: > > > > On 4/29/20 3:07 PM, Yu-cheng Yu wrote: > > > > > +Note: > > > > > + There is no CET-enabling arch_prctl function. By design, CET is > > > > > enabled > > > > > + automatically if the binary and the system can support it. > > > > > > > > I think Andy and I danced around this last time. Let me try to say it > > > > more explicitly. > > > > > > > > I want CET kernel enabling to able to be disconnected from the on-disk > > > > binary. I want a binary compiled with CET to be able to disable it, and > > > > I want a binary not compiled with CET to be able to enable it. I want > > > > different threads in a process to be able to each have different CET > > > > status. > > > > > > The kernel patches we have now can be modified to support this model. If > > > after > > > discussion this is favorable, I will modify code accordingly. > > > > To turn on/off and to lock CET are application-level decisions. The kernel > > does > > not prevent any of those. Should there be a need to provide an > > arch_prctl() to > > turn on CET, it can be added without any conflict to this series. > > I spelled out what I wanted pretty clearly. On your next post, could > you please directly address each of the things I asked for? Please > directly answer the following questions in your next post with respect > to the code you post: > > Can a binary compiled with CET run without CET? Yes, but a few details: - The shadow stack is transparent to the application. A CET application does not have anything different from a non-CET application. However, if a CET application uses any CET instructions (e.g. INCSSP), it must first check if CET is turned on. - If an application is compiled for IBT, the compiler inserts ENDBRs at branch targets. These are nops if IBT is not on. > Can a binary compiled without CET run CET-enabled code? Partially yes, but in reality somewhat difficult. - If a non-CET application does exec() of a CET binary, then CET is enabled. - If a not-CET application does fork(), and the child wants to turn on CET, it would be difficult to manage the stack frames, unless the child knows what is is doing. The JIT examples I mentioned previously run with CET enabled from the beginning. Do you have a reason to do this? In other words, if the JIT code needs CET, the app could have started with CET in the first place. - If you are asking about dlopen(), the library will have the same setting as the main application. Do you have any reason to have a library running with CET, but the application does not have CET? > Can different threads in a process have different CET enabling state? Yes, if the parent starts with CET, children can turn it off. But for the same reason described above, it is difficult to turn on CET from the middle. > > > > Which JITs was this tested with? I think as a bare minimum we need to > > > > know that this design can accommodate _a_ modern JIT. It would be > > > > horrible if the browser javascript engines couldn't use this design, for > > > > instance. > > > > > > JIT work is still in progress. When that is available I will test it. > > > > I found CET has been enabled in LLVM JIT, Mesa JIT as well as sljit which is > > used by jit. So the current model works with JIT. > > Great! I'm glad the model works. That's not what I asked, though. > > Does this *code* work? Could you please indicate which JITs have been > enabled to use the code in this series? How much of the new ABI is in use? JIT does not necessarily use all of the ABI. The JIT changes mainly fix stack frames and insert ENDBRs. I do not work on JIT. What I found is LLVM JIT fixes are tested and in the master branch. Sljit fixes are in the release. > Where are the selftests/ for this new ABI? Were you planning on > submitting any with this series? The ABI is more related to the application side, and therefore most suitable for GLIBC unit tests. The more complicated areas such as pthreads, signals, ucontext, fork() are all included there. I have been constantly running these tests without any problems. I can provide more details if testing is the concern. Yu-cheng
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On 5/12/20 4:20 PM, Yu-cheng Yu wrote: > On Wed, 2020-04-29 at 16:02 -0700, Yu-cheng Yu wrote: >> On Wed, 2020-04-29 at 15:53 -0700, Dave Hansen wrote: >>> On 4/29/20 3:07 PM, Yu-cheng Yu wrote: +Note: + There is no CET-enabling arch_prctl function. By design, CET is enabled + automatically if the binary and the system can support it. >>> >>> I think Andy and I danced around this last time. Let me try to say it >>> more explicitly. >>> >>> I want CET kernel enabling to able to be disconnected from the on-disk >>> binary. I want a binary compiled with CET to be able to disable it, and >>> I want a binary not compiled with CET to be able to enable it. I want >>> different threads in a process to be able to each have different CET status. >> >> The kernel patches we have now can be modified to support this model. If >> after >> discussion this is favorable, I will modify code accordingly. > > To turn on/off and to lock CET are application-level decisions. The kernel > does > not prevent any of those. Should there be a need to provide an arch_prctl() > to > turn on CET, it can be added without any conflict to this series. I spelled out what I wanted pretty clearly. On your next post, could you please directly address each of the things I asked for? Please directly answer the following questions in your next post with respect to the code you post: Can a binary compiled with CET run without CET? Can a binary compiled without CET run CET-enabled code? Can different threads in a process have different CET enabling state? >>> Which JITs was this tested with? I think as a bare minimum we need to >>> know that this design can accommodate _a_ modern JIT. It would be >>> horrible if the browser javascript engines couldn't use this design, for >>> instance. >> >> JIT work is still in progress. When that is available I will test it. > > I found CET has been enabled in LLVM JIT, Mesa JIT as well as sljit which is > used by jit. So the current model works with JIT. Great! I'm glad the model works. That's not what I asked, though. Does this *code* work? Could you please indicate which JITs have been enabled to use the code in this series? How much of the new ABI is in use? Where are the selftests/ for this new ABI? Were you planning on submitting any with this series?
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Wed, 2020-04-29 at 16:02 -0700, Yu-cheng Yu wrote: > On Wed, 2020-04-29 at 15:53 -0700, Dave Hansen wrote: > > On 4/29/20 3:07 PM, Yu-cheng Yu wrote: > > > +Note: > > > + There is no CET-enabling arch_prctl function. By design, CET is > > > enabled > > > + automatically if the binary and the system can support it. > > > > I think Andy and I danced around this last time. Let me try to say it > > more explicitly. > > > > I want CET kernel enabling to able to be disconnected from the on-disk > > binary. I want a binary compiled with CET to be able to disable it, and > > I want a binary not compiled with CET to be able to enable it. I want > > different threads in a process to be able to each have different CET status. > > The kernel patches we have now can be modified to support this model. If > after > discussion this is favorable, I will modify code accordingly. To turn on/off and to lock CET are application-level decisions. The kernel does not prevent any of those. Should there be a need to provide an arch_prctl() to turn on CET, it can be added without any conflict to this series. > > Which JITs was this tested with? I think as a bare minimum we need to > > know that this design can accommodate _a_ modern JIT. It would be > > horrible if the browser javascript engines couldn't use this design, for > > instance. > > JIT work is still in progress. When that is available I will test it. I found CET has been enabled in LLVM JIT, Mesa JIT as well as sljit which is used by jit. So the current model works with JIT. Yu-cheng
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On Wed, 2020-04-29 at 15:53 -0700, Dave Hansen wrote: > On 4/29/20 3:07 PM, Yu-cheng Yu wrote: > > +Note: > > + There is no CET-enabling arch_prctl function. By design, CET is enabled > > + automatically if the binary and the system can support it. > > I think Andy and I danced around this last time. Let me try to say it > more explicitly. > > I want CET kernel enabling to able to be disconnected from the on-disk > binary. I want a binary compiled with CET to be able to disable it, and > I want a binary not compiled with CET to be able to enable it. I want > different threads in a process to be able to each have different CET status. The kernel patches we have now can be modified to support this model. If after discussion this is favorable, I will modify code accordingly. > Which JITs was this tested with? I think as a bare minimum we need to > know that this design can accommodate _a_ modern JIT. It would be > horrible if the browser javascript engines couldn't use this design, for > instance. JIT work is still in progress. When that is available I will test it. Yu-cheng
Re: [PATCH v10 01/26] Documentation/x86: Add CET description
On 4/29/20 3:07 PM, Yu-cheng Yu wrote: > +Note: > + There is no CET-enabling arch_prctl function. By design, CET is enabled > + automatically if the binary and the system can support it. I think Andy and I danced around this last time. Let me try to say it more explicitly. I want CET kernel enabling to able to be disconnected from the on-disk binary. I want a binary compiled with CET to be able to disable it, and I want a binary not compiled with CET to be able to enable it. I want different threads in a process to be able to each have different CET status. Which JITs was this tested with? I think as a bare minimum we need to know that this design can accommodate _a_ modern JIT. It would be horrible if the browser javascript engines couldn't use this design, for instance.
