On Wed, Nov 05, 2025 at 01:16:19PM -0800, Deepak Gupta wrote:
On Wed, Nov 05, 2025 at 12:24:41PM -0800, Deepak Gupta wrote:
On Tue, Nov 04, 2025 at 05:34:11PM +1030, Joel Stanley wrote:
Hello Deepak,
On Fri, 24 Oct 2025 at 03:31, Deepak Gupta via B4 Relay
<[email protected]> wrote:
v22: fixing build error due to -march=zicfiss being picked in gcc-13 and above
but not actually doing any codegen or recognizing instruction for zicfiss.
Change in v22 makes dependence on `-fcf-protection=full` compiler flag to
ensure that toolchain has support and then only CONFIG_RISCV_USER_CFI will be
visible in menuconfig.
Following our discussion at the riscv summit I spent some time with
this patch set with the goal of giving a test run on emulation. I only
got as far as qemu, as I couldn't get the selftests passing there.
I had trouble running the podman container so I built a toolchain
using the riscv-gnu-toolchain branch (cfi-dev, d19f3009f6c2) you
pointed to.
The opensbi branch was a bit old and wouldn't build with GCC 15, so I
tried to rebase and noticed the patches were already upstream. Have
you tested using v1.7 (or newer) there? Is there something I missed,
do we need more patches on upstream opensbi?
I booted it in qemu 10.1.2 with the zicfi extensions both on and off.
qemu-system-riscv64 -M virt,aia=aplic-imsic,aia-guests=5 \
-cpu rv64,zicfilp=true,zicfiss=true,zimop=true,zcmop=true
-smp 8 -nographic -bios fw_dynamic.elf
-m 1024M -kernel arch/riscv/boot/Image \
-initrd selftests/selftests.cpio \
-append 'init=mini-init command="cfitests"'
My results:
no zicfi, no z*mop (crash, as expected):
-------------------------------------------------
Running command: cfitests
system_opcode_insn: Invalid opcode for CSR read/write instruction[
0.462709] cfitests[85]: unhandled signal 4 code 0x1 at
0x0000000000011c44 in cfitests[1c44,10000+6d000]
[ 0.463141] CPU: 4 UID: 0 PID: 85 Comm: cfitests Not tainted
6.18.0-rc3-tt-defconfig-jms-00090-g6e2297f1edbc #93 NONE
[ 0.463338] Hardware name: riscv-virtio,qemu (DT)
[ 0.463573] epc : 0000000000011c44 ra : 00000000000104e0 sp :
00007fffebd0ddb0
...
[ 0.465177] status: 0000000200004020 badaddr: 00000000ce104073
cause: 0000000000000002
[ 0.465410] Code: 0893 05d0 4501 0073 0000 b7f5 4501 b7f9 0017 0000
(4073) ce10
no zicfi, z*mop (failed to start, as expected):
-----------------------------------------------------------
Running command: cfitests
TAP version 13
# Starting risc-v CFI tests
Bail out! Get landing pad status failed with -22
zicfi, z*mop (failed to start, unexpected):
-------------------------------------------------------
Running command: cfitests
TAP version 13
# Starting risc-v CFI tests
Bail out! Get landing pad status failed with -22
I went digging to see why the zicfi enabled kernel failed. The
userspace binary was built with CFI:
$ riscv64-unknown-linux-gnu-readelf -n selftests/cfitests
Displaying notes found in: .note.gnu.property
Owner Data size Description
GNU 0x00000010 NT_GNU_PROPERTY_TYPE_0
Properties: RISC-V AND feature: CFI_LP_UNLABELED, CFI_SS
I then tested your opensbi tree with some hacks to get it built with a
newer compiler. This produced different results, which was unexpected:
Running command: cfitests
TAP version 13
# Starting risc-v CFI tests
Bail out! Landing pad is not enabled, should be enabled via glibc
# Totals: pass:0 fail:0 xfail:0 xpass:0 skip:0 error:0
Ok so I think atleast I have narrowed down the issue. It seems like
something is going on with how libc will initialize cfi in case of
static compile.
I enabled logging whenever prctl set/get are called and my rootfs
binaries which are all dynamically linked are setting cfi prctls
correctly. Except when I run statically compiled `cfitest`. It bails
on "Shadow stack is not enabled" (unlike in your case where it bails
saying landing pad is not enabled).
I think I have figured out why its happening on my end.
In case of, riscv-gnu-toolchain's "cfi-dev" branch's glibc submodule
is not setting rest of the parameters for prctl to `0` (for static compile)
https://github.com/sifive/riscv-glibc/blob/20b27db7d9aa8f25419b0cbaf62004e53713315e/sysdeps/riscv/libc-start.h#L73
Kernel will reject the prctl for cfi set/get, if any of the not required
parameter is set to `0`.
Although latest glibc patches from Jesse on libc-alpha do that correctly
https://public-inbox.org/libc-alpha/[email protected]/T/#u
Shortlog
"""
Welcome to Buildroot
buildroot login: root
[ 51.361564] login[129]: arch_set_shadow_stack_status set successfully,
shadow stack base 7fffbb977000size 800000
[ 51.362708] login[129]: arch_set_indir_br_lp_status set successfully
[ 51.363805] login[129]: arch_get_indir_br_lp_status called
[ 51.364296] login[129]: arch_get_shadow_stack_status called
[ 51.448918] bash[129]: arch_set_shadow_stack_status set successfully, shadow
stack base 7fff9b516000size 800000
[ 51.450030] bash[129]: arch_set_indir_br_lp_status set successfully
[ 51.450568] bash[129]: arch_get_indir_br_lp_status called
[ 51.451047] bash[129]: arch_get_shadow_stack_status called
[ 51.552710] id[130]: arch_set_shadow_stack_status set successfully, shadow
stack base 7fff7fa2a000size 800000
[ 51.553798] id[130]: arch_set_indir_br_lp_status set successfully
[ 51.554306] id[130]: arch_get_indir_br_lp_status called
[ 51.554757] id[130]: arch_get_shadow_stack_status called
# mount -t 9p -o trans=virtio,version=9p2000.L host0 /mnt
[ 65.484939] mount[131]: arch_set_shadow_stack_status set successfully,
shadow stack base 7fffa71a0000size 800000
[ 65.487002] mount[131]: arch_set_indir_br_lp_status set successfully
[ 65.488482] mount[131]: arch_get_indir_br_lp_status called
[ 65.489346] mount[131]: arch_get_shadow_stack_status called
# /mnt/cfitests
[ 74.235075] cfitests[132]: arch_set_indir_br_lp_status set successfully
[ 74.249763] cfitests[132]: arch_get_indir_br_lp_status called
[ 74.250823] cfitests[132]: arch_get_shadow_stack_status called
TAP version 13
# Starting risc-v tests
[ 74.374286] cfitests[132]: arch_get_indir_br_lp_status called
[ 74.375497] cfitests[132]: arch_get_shadow_stack_status called
Bail out! Shadow stack is not enabled, should be enabled via glibc
# Totals: pass:0 fail:0 xfail:0 xpass:0 skip:0 error:0
#
"""
Full log (with cfi prctls get/set) of boot at pastebin below
https://pastebin.com/JbX2Yv5W
CCed: Jesse Huang who is leading glibc patches. Can help us figure out
why statically compiled binaries are failing to set cfi prctls appropriately.
Paul,
Given that it's not an issue in kernel changes, we should still be going ahead
with it for 6.19
The selftest binary and the little toy init that starts it are both
statically linked and built against the toolchain's glibc, so I would
expect this to work.
$ riscv64-unknown-linux-gnu-readelf -n sifive-cfi-build/sysroot/usr/lib/libc.a
File: sifive-cfi-build/sysroot/usr/lib/libc.a(init-first.o)
Displaying notes found in: .note.gnu.property
Owner Data size Description
GNU 0x00000010 NT_GNU_PROPERTY_TYPE_0
Properties: RISC-V AND feature: CFI_LP_UNLABELED, CFI_SS
The kernel seems to have detected that CFI is available and is built with it:
$ grep CFI .config
CONFIG_RISCV_USER_CFI=y
CONFIG_ARCH_SUPPORTS_CFI=y
I did notice the func-sig-dev gcc branch is a few commits ahead of
what the sifive riscv-gnu-toolchain points to.
I had to context switch to some other tasks at this point. I wanted to
do some more digging to work out what was wrong, but I haven't found
time, so here are my notes in the hope that they are useful. I'll let
you know if I discover anything further.
I have it working on my end with latest upstream opensbi (no hacks, same
compiler)
"""
$ git log
commit 38a6106b1099646f25657bba53cefb80886721a7 (HEAD -> master, origin/master,
origin/HEAD)
Author: Benoît Monin <[email protected]>
Date: Mon Oct 27 14:12:17 2025 +0100
lib: utils/ipi: mswi: add MIPS P8700 compatible
....
"""
I am surprised that change of compiler on opensbi changed errorcode for
userspace
in your setup. That's quite bizarre.
Output from cfitests (with toolchain that's on docker. I didn't compile from
cfi-dev branch).
# /mnt/cfitests
TAP version 13
# Starting risc-v tests
# Landing pad and shadow stack are enabled for binary
# cfi_ptrace_test, ptrace test succeeded
# Executing RISC-V shadow stack self tests
1..5
# Exercising shadow stack fork test
# Parent pid 133 and child pid 135
# dummy calls for sspush and sspopchk in context of parent
# Spewing out shadow stack ptr: 7fffbf4a9fb8
This is to ensure shadow stack is indeed enabled and working
# Waiting on child to finish
# dummy calls for sspush and sspopchk in context of child
# Spewing out shadow stack ptr: 7fffbf4a9fb8
This is to ensure shadow stack is indeed enabled and working
ok 1 shstk fork test
# Exercising shadow stack map test
ok 2 map shadow stack syscall
# Exercising shadow stack gup tests
ok 3 shadow stack gup tests
# Exercising shadow stack signal test
ok 4 shadow stack signal tests
# Exercising shadow stack protection test (WPT)
ok 5 memory protections of shadow stack memory
# Totals: pass:5 fail:0 xfail:0 xpass:0 skip:0 error:0
#
Is there a place where I can grab your kselftest `cfitests` binary?
Only difference I can see is that `cfitests` in my case is not statically
compiled
"""
$ riscv64-unknown-linux-gnu-readelf -d
/scratch/debug/sources/spectacles/cfitests | grep NEEDED
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
"""
Cheers,
Joel
How to test this series
=======================
Toolchain
---------
$ git clone [email protected]:sifive/riscv-gnu-toolchain.git -b cfi-dev
$ riscv-gnu-toolchain/configure --prefix=<path-to-where-to-build>
--with-arch=rv64gc_zicfilp_zicfiss --enable-linux --disable-gdb
--with-extra-multilib-test="rv64gc_zicfilp_zicfiss-lp64d:-static"
$ make -j$(nproc)
Qemu
----
Get the lastest qemu
$ cd qemu
$ mkdir build
$ cd build
$ ../configure --target-list=riscv64-softmmu
$ make -j$(nproc)
Opensbi
-------
$ git clone [email protected]:deepak0414/opensbi.git -b v6_cfi_spec_split_opensbi
$ make CROSS_COMPILE=<your riscv toolchain> -j$(nproc) PLATFORM=generic
Linux
-----
Running defconfig is fine. CFI is enabled by default if the toolchain
supports it.
$ make ARCH=riscv
CROSS_COMPILE=<path-to-cfi-riscv-gnu-toolchain>/build/bin/riscv64-unknown-linux-gnu-
-j$(nproc) defconfig
$ make ARCH=riscv
CROSS_COMPILE=<path-to-cfi-riscv-gnu-toolchain>/build/bin/riscv64-unknown-linux-gnu-
-j$(nproc)
Running
-------
Modify your qemu command to have:
-bios <path-to-cfi-opensbi>/build/platform/generic/firmware/fw_dynamic.bin
-cpu rv64,zicfilp=true,zicfiss=true,zimop=true,zcmop=true
