casaroli commented on code in PR #20132: URL: https://github.com/apache/nuttx/pull/20132#discussion_r4208180021
########## Documentation/components/fdpic.rst: ########## @@ -0,0 +1,513 @@ +.. _fdpic: + +============= +FDPIC Modules +============= + +Overview +======== + +An FDPIC module is an ELF shared object whose read-only and writable +segments are placed independently of one another. NuttX uses the +read-only segment where it already lies on the media and never copies +it; only the writable segment is copied to RAM, once per running +instance. A module's code and ``.rodata`` therefore cost no RAM at all, +and several instances of one module share them. + +FDPIC is not a separate binary format and has no loader of its own. An +object announces itself in its OS/ABI byte, +``e_ident[EI_OSABI] == ELFOSABI_ARM_FDPIC`` (65), which ``readelf -h`` +reports as *OS/ABI: ARM FDPIC*, and the ELF loader takes it from there. +Everything else -- ``exec()``, ``posix_spawn()``, ``dlopen()``, the +symbol table -- is the ordinary ELF path. + +What FDPIC adds over the position independent ELF support already in the +tree is a function pointer that carries its own data base. That is what +lets a module be called back on a thread it did not create, and what +lets a module and the libraries it uses hold distinct data bases at the +same time. + +Function descriptors +-------------------- + +Code reaches its own data through a base register -- **r9** on ARM -- +holding the address of that object's GOT. Because code and data are +placed independently, a bare code address is not enough to call a +function: the callee needs its data base too. FDPIC therefore +represents a function pointer as a two word *descriptor*: + +=========== ============================================================== +Word Contents +=========== ============================================================== +``entry`` Code address, including its Thumb bit +``got`` Data base to install in the PIC base register before + branching +=========== ============================================================== + +Building those descriptors is most of what relocation does. Because +each one names its own base, a pointer handed to the base firmware +carries everything needed to call back into the module later, from any +thread. + +A module links against nothing. libc and everything else are undefined +imports, resolved at load time against the globally registered symbols +first, then any shared libraries the module names, then the symbol table +``exec()`` supplied. + +Placement +--------- + +The loader asks the filesystem where the file lies on its media. Two +mechanisms exist and they are not interchangeable: + +* ``XIPFSIOC_PIN`` is for a filesystem that can move a file's blocks. It + returns an address together with a pin that holds the extent still, and + the pin is given back with ``XIPFSIOC_UNPIN`` when the module is + unloaded. :doc:`XIPFS <filesystem/xipfs>` is the one in tree. + +* ``FIOC_XIPBASE`` is for a filesystem whose layout never changes, which + has nothing to hold and answers with a bare address. ROMFS and TMPFS + are those. + +The pin is asked for first, because a filesystem that needs one is not +safe without it. The loader asks for a pin only if it can hold one, +which is the flat build, or the pin would stay for ever. + +A filesystem that answers neither is still usable. The loader then +copies the text to RAM, as it does for any other module. The module +loses the shared text and the flash saving, but it runs. + +The writable segment is allocated and copied per instance, and a pool of +function descriptors is reserved behind it for the relocations that ask +the loader to manufacture one. When the task starts, +``up_initial_state()`` installs the object's data base -- ``DT_PLTGOT``, +or the GOT immediately after ``PT_DYNAMIC`` in an object with no +imports -- into the PIC base register. + +Shared libraries +---------------- + +A module may name shared libraries in ``DT_NEEDED``. The loader loads +each one during relocation, through ``libelf_insert()`` as ``dlopen()`` +does, and then binds the module's undefined symbols against that +library's exports. ``CONFIG_LIBC_ELF_MAXDEPEND`` caps how many one module +may name, and a module that names more is refused. With +``CONFIG_ARCH_ADDRENV`` a module carrying ``DT_NEEDED`` is refused, because +the library would not be in the address space of the program. + +Libraries are found the way ``dlopen()`` finds them: an absolute path is +used as given, and a bare name is searched for along ``LD_LIBRARY_PATH``, +which needs ``CONFIG_LIBC_ENVPATH`` and is seeded from +``CONFIG_LDPATH_INITIAL``. + +A library lands in the module registry, which holds one instance per +name, so its data is shared by everything that opens it. A module +started with ``exec()`` is different: that path loads a fresh copy each +time, so two running instances of one module have separate data while +sharing one copy of the text in flash. + +Comparison with NXFLAT and PIC ELF +================================== + +All three run position independent code from flash on a target with no +MMU, and all three give several instances of one module a shared +``.text`` with private ``.data``. They differ in what a *pointer* can +express and in what the toolchain has to provide. + +========================= ============== ============== ============= +Property NXFLAT PIC ELF FDPIC +========================= ============== ============== ============= +Format NuttX only ELF ELF +Extra build tools yes none linker +Data base per task task object +Shared libraries no no yes +Foreign-thread callback no no yes +Instruction set ARM, Thumb-2 unrestricted Thumb-2 only +========================= ============== ============== ============= + +:ref:`NXFLAT <nxflat>` is a NuttX-specific format. A module imports +symbols from the base firmware but cannot export any, so shared +libraries are not possible, and the build needs ``mknxflat`` to generate +a thunk, ``ldnxflat`` to link, and one of the ``binfmt/libnxflat`` linker +scripts to place the sections. + +**PIC ELF** needs no extra tools. With ``CONFIG_PIC`` the ELF loader +allocates the writable sections separately and, when the filesystem +answers ``FIOC_XIPBASE``, leaves the read-only ones on the media. Two +limits follow from having one base register per task: a shared object is +loaded as a single allocation, because the distance between its text and +its data is compiled into it, and the data base is installed once per +task, so every object in a task shares one. + +**FDPIC** pays for its descriptors with an ``arm-uclinuxfdpiceabi`` +linker, and gets back the two things a single register +cannot express. A task or pthread that a module starts inherits the +module's D-Space, so a register would be enough there; a work queue +worker was created at boot and carries no module base, and a descriptor +supplies one, which is how ``SIGEV_THREAD`` notifications reach module +code. + +Requirements +============ + +**An ARM Thumb-2 core.** The boundary is the instruction set, not the +core profile: GCC rejects FDPIC in Thumb-1 mode. + +========================= ========================== ===== +Core Architecture FDPIC +========================= ========================== ===== +Cortex-M3 / M4 / M7 ARMv7-M / ARMv7E-M yes +Cortex-M33 ARMv8-M Mainline yes +Cortex-M0 / M0+ / M23 ARMv6-M / ARMv8-M Baseline no +========================= ========================== ===== + +RISC-V has no FDPIC ABI -- the psABI addendum is an unmerged proposal and +no ``EI_OSABI`` value is assigned -- so a RISC-V target cannot use this. + +**Flash that is memory mapped and executable**, exposed by a filesystem +that answers ``XIPFSIOC_PIN`` or ``FIOC_XIPBASE``. This is what gives +execute in place. Without it the module still loads, but from RAM. + +**An FDPIC linker.** A stock ``arm-none-eabi`` GCC compiles correct +FDPIC code for both C and C++. Its assembler accepts the relocations that +code produces in FDPIC mode only: GCC 14 and later select that mode for +``-mfdpic``, and for an older GCC the build passes ``-Wa,--fdpic``. Only +``arm-uclinuxfdpiceabi`` binutils carry the ``armelf_linux_fdpiceabi`` +emulation that the link needs, and ``arm-none-eabi-ld`` rejects it. + +No distribution packages that target, so build binutils for it -- which +takes about a minute and needs nothing else:: + + configure --target=arm-uclinuxfdpiceabi --prefix=$HOME/fdpic \ + --disable-nls --disable-werror + make && make install + export PATH=$HOME/fdpic/bin:$PATH + +An FDPIC GCC is not needed. + +**The base firmware must reserve r9.** It is not enough for the module +to be well behaved: a firmware routine calling back into module code +arrives with the module's data base in r9 only if the compiler was never +free to allocate that register elsewhere. ``CONFIG_FDPIC`` selects +``CONFIG_PIC``, under which ``arch/arm/src/common/Toolchain.defs`` adds +``--fixed-r9``; see :ref:`nxflat` for why it goes into ``ARCHCFLAGS`` +rather than ``CFLAGS`` and how to check that it arrived. + +Configuration +============= + +``CONFIG_FDPIC`` lives under ``CONFIG_ELF``. A working configuration +also needs a symbol table for modules to import from and a filesystem +that can expose its media:: + + CONFIG_ELF=y + CONFIG_FDPIC=y + CONFIG_LIBC_EXECFUNCS=y + CONFIG_EXECFUNCS_HAVE_SYMTAB=y + CONFIG_EXECFUNCS_SYSTEM_SYMTAB=y + CONFIG_FS_XIPFS=y + +``crt0`` runs the constructors of a module only with +``CONFIG_HAVE_CXXINITIALIZE``. + +Shared libraries need three more. A library resolves its own imports +against the table of ``CONFIG_LIBC_ELF_HAVE_SYMTAB``, and the last two let +a library be named rather than spelled out as an absolute path:: + + CONFIG_LIBC_ELF_HAVE_SYMTAB=y + CONFIG_LIBC_ENVPATH=y + CONFIG_LDPATH_INITIAL="/mnt/xipfs" + +``dlopen()`` needs ``CONFIG_LIBC_DLFCN`` too. + +Only the make build generates the table of +``CONFIG_EXECFUNCS_SYSTEM_SYMTAB``. A CMake configuration needs a symbol +table from elsewhere. + +``CONFIG_ELF_STACKSIZE`` gives the stack a module runs with. A module +that needs a different one can export an ``nx_stacksize`` symbol, which +the loader prefers when present. + +Building a module +================= + +Select ``CONFIG_FDPIC`` and a module is built by the ordinary in-tree ELF +build: the same ``MODULE = m`` in the same application Makefile as any +other, and the same ``crt0``. Nothing else is needed. + +What the build does differently is add compiler flags and use a +different linker:: + + arm-none-eabi-gcc -mcpu=cortex-m33 -mthumb -mfdpic -fPIC -Wa,--fdpic \ + -Os -fno-builtin -D__NuttX__ -I$NUTTX/include -c mod.c -o mod.o + + arm-uclinuxfdpiceabi-ld -m armelf_linux_fdpiceabi -shared -z now \ + -e _start -T $NUTTX/libs/libc/elf/gnu-elf.ld \ + -o mod $NUTTX/arch/arm/src/crt0.o mod.o + +Only the link needs the FDPIC toolchain. The stock compiler emits correct +FDPIC objects for both C and C++, its assembler included once it is in +FDPIC mode. That linker is in the NuttX CI image; +``tools/ci/docker/linux/Dockerfile`` shows how it is built. +``FDPIC_CROSSDEV`` names a different prefix, and the build says so if it +is missing. + +Six flags carry weight: + +* ``-mfdpic`` is stated rather than assumed, so a mis-set toolchain fails + loudly instead of producing a plain ELF the loader will not recognize. + +* ``-fPIC`` is not implied by ``-mfdpic`` on a bare-metal target, and + without it the link emits ``TEXTREL``. Text relocations cannot work + against text executed from read-only flash. + +* ``-Wa,--fdpic`` puts the assembler in FDPIC mode, which GCC before 14 + does not do for ``-mfdpic``. Without it the assembler stops with + "Relocation supported only in FDPIC mode". + +* ``-shared`` preserves the ``R_ARM_FUNCDESC_VALUE`` relocations for + imported symbols. A PIE link with ``--unresolved-symbols=ignore-all`` + appears to work but degrades every import to ``R_ARM_NONE``, and the + module branches to zero on its first call into the firmware. + +* ``-m armelf_linux_fdpiceabi`` is required: this linker supports several + emulations and will not guess. + +* ``-e _start`` names the entry point. ``crt0.c`` is the module's own + start-up file, the one every other module uses: it walks ``.init_array`` + and then calls ``main``. A shared library is never entered, so it is + linked without ``crt0``. + +A module links with ``-shared``, so importing something the firmware does +not export links cleanly and fails only on the target. Checking the +undefined symbols of the module against the generated +``libs/libc/exec_symtab.c`` catches that at build time. + +When the toolchain cannot make an FDPIC module, the build stops with one +of these: + +``CONFIG_FDPIC needs arm-uclinuxfdpiceabi-ld, which is not on PATH`` + The FDPIC linker is missing. The CMake build stops with the same + message when it configures. + +``unrecognised emulation mode: armelf_linux_fdpiceabi`` + ``FDPIC_CROSSDEV`` names a linker without the FDPIC emulation, such as + ``arm-none-eabi-``. + +``Relocation supported only in FDPIC mode`` + The assembler is not in FDPIC mode. A compile that does not use the + module flags has lost ``-Wa,--fdpic``. + +``unknown argument: '-mfdpic'`` + The compiler cannot make FDPIC code. Clang is one. + +Building a shared library +------------------------- + +A library is an ordinary shared library of the application build, +``BUILD_SHARED_LIBRARY`` in make and ``DYNLIB y`` in CMake, with a +soname:: + + LOCAL_MODULE := libfoo + LOCAL_MODULE_FILENAME := libfoo.so + LOCAL_SRC_FILES := libfoo.c + LOCAL_LDFLAGS := -soname libfoo.so + + include $(BUILD_SHARED_LIBRARY) + +The module flags hide every symbol, so a library marks what it exports +with ``visibility_default``. A module that uses the library names it on +its link line, which records the soname in ``DT_NEEDED``; in a module +Makefile, ``LDLIBS`` for the module target does that. +``apps/examples/fdpicxip/modules`` has both kinds. + +At run time the library must be reachable under its soname along +``LD_LIBRARY_PATH``. + +Calling back into a module +========================== + +A module's function pointer is the address of a descriptor in its +writable segment. Firmware that stores one and later branches to it +would jump into RAM data, so an entry point that accepts a callback from +a module has to resolve the descriptor first. ``CONFIG_FDPIC`` makes +these do so: + +``qsort``, ``bsearch``, ``pthread_create``, ``signal``/``sigaction``, +``task_create``/``task_create_with_stack``, ``task_spawn``, +``pthread_once``, ``scandir``, and ``mq_notify``/``timer_create`` with +``SIGEV_THREAD``. + +Whether a pointer is a descriptor is decided by reading the PIC base +register: a module's task runs with its data base there, a firmware task +with zero, so a kernel caller is unaffected. + +A new entry point that takes a module callback must resolve it too, under +three rules: + +* **Resolve once, in the innermost common routine.** Resolving twice + treats a code address as a descriptor. ``qsort()`` recurses, so its + public entry resolves and the recursive body does not; ``signal()`` + does not resolve because ``nxsig_action()`` does it for both paths; + ``scandir()`` resolves its filter but not the comparison function it + hands to ``qsort()``. + +* **Exclude sentinel values by hand.** ``fdpic_callback()`` declines to + dereference NULL and nothing else. ``sigaction()`` excludes + ``SIG_IGN``, ``SIG_DFL``, ``SIG_HOLD`` and ``SIG_ERR`` -- the integers + 0, 1, 2 and -1. + +* **A callback on a shared thread needs its base installed.** A + ``SIGEV_THREAD`` notification runs on a work queue worker that carries + no module base, so resolving the entry is not enough. Capture the base + at registration with ``fdpic_base()``, in the module's own context, and + install it around the call with ``fdpic_invoke()``. + +Everywhere else the callback runs in a task that inherited the module's +D-Space, so only the code address needs resolving. + +Examples and tests +================== + +``apps/examples/fdpicxip`` writes modules into XIPFS at run time and runs +them. ``fdpicxip qsort`` runs one module twice, with one copy of its +text; ``solib`` adds a shared library; ``cxx`` does the same in C++ and +checks that the constructors ran; ``jmprel`` runs a module whose imports +are bound through ``DT_JMPREL``. + +``apps/testing/fs/xipfs`` asserts what the demo shows. ``xipfs_test +fdpic`` loads modules and checks the loader, and ``xipfs_test reject`` +checks that a malformed or unloadable module is refused. Both apps build +their modules from ``apps/examples/fdpicxip/modules`` with the module +flags of the tree. + +``pimoroni-pico-2-plus:xipfs-fdpic`` is a board configuration with all of +this. Without hardware, ``mps2-an500:xipfs`` runs it under QEMU with +these added:: + + CONFIG_ELF=y + CONFIG_FDPIC=y + CONFIG_EXAMPLES_FDPICXIP=y + CONFIG_LIBC_EXECFUNCS=y + CONFIG_EXECFUNCS_HAVE_SYMTAB=y + CONFIG_EXECFUNCS_SYSTEM_SYMTAB=y + CONFIG_LIBC_DLFCN=y + CONFIG_LIBC_ELF_HAVE_SYMTAB=y + CONFIG_LIBC_ENVPATH=y + CONFIG_LDPATH_INITIAL="/mnt/xipfs" + CONFIG_SIG_EVTHREAD=y + CONFIG_SCHED_HPWORK=y + CONFIG_INIT_STACKSIZE=16384 + +and with ``CONFIG_DISABLE_POSIX_TIMERS``, ``CONFIG_PROFILE_ALL``, +``CONFIG_PROFILE_MINI`` and ``CONFIG_SYSTEM_GPROF`` off. + +Limitations +=========== + +**Tested in the flat build only.** In a protected build, a module that +``exec()`` started faulted at its entry point when last tested, because +the kernel side of the loader places the module in the kernel heap. A +kernel build needs an MMU, which the FDPIC cores do not have, and an +address environment refuses ``DT_NEEDED``. + +**A library's constructors run on the loading task, not on the task that Review Comment: done ########## Documentation/components/fdpic.rst: ########## @@ -0,0 +1,513 @@ +.. _fdpic: + +============= +FDPIC Modules +============= + +Overview +======== + +An FDPIC module is an ELF shared object whose read-only and writable +segments are placed independently of one another. NuttX uses the +read-only segment where it already lies on the media and never copies +it; only the writable segment is copied to RAM, once per running +instance. A module's code and ``.rodata`` therefore cost no RAM at all, +and several instances of one module share them. + +FDPIC is not a separate binary format and has no loader of its own. An +object announces itself in its OS/ABI byte, +``e_ident[EI_OSABI] == ELFOSABI_ARM_FDPIC`` (65), which ``readelf -h`` +reports as *OS/ABI: ARM FDPIC*, and the ELF loader takes it from there. +Everything else -- ``exec()``, ``posix_spawn()``, ``dlopen()``, the +symbol table -- is the ordinary ELF path. + +What FDPIC adds over the position independent ELF support already in the +tree is a function pointer that carries its own data base. That is what +lets a module be called back on a thread it did not create, and what +lets a module and the libraries it uses hold distinct data bases at the +same time. + +Function descriptors +-------------------- + +Code reaches its own data through a base register -- **r9** on ARM -- +holding the address of that object's GOT. Because code and data are +placed independently, a bare code address is not enough to call a +function: the callee needs its data base too. FDPIC therefore +represents a function pointer as a two word *descriptor*: + +=========== ============================================================== +Word Contents +=========== ============================================================== +``entry`` Code address, including its Thumb bit +``got`` Data base to install in the PIC base register before + branching +=========== ============================================================== + +Building those descriptors is most of what relocation does. Because +each one names its own base, a pointer handed to the base firmware +carries everything needed to call back into the module later, from any +thread. + +A module links against nothing. libc and everything else are undefined +imports, resolved at load time against the globally registered symbols +first, then any shared libraries the module names, then the symbol table +``exec()`` supplied. + +Placement +--------- + +The loader asks the filesystem where the file lies on its media. Two +mechanisms exist and they are not interchangeable: + +* ``XIPFSIOC_PIN`` is for a filesystem that can move a file's blocks. It + returns an address together with a pin that holds the extent still, and + the pin is given back with ``XIPFSIOC_UNPIN`` when the module is + unloaded. :doc:`XIPFS <filesystem/xipfs>` is the one in tree. + +* ``FIOC_XIPBASE`` is for a filesystem whose layout never changes, which + has nothing to hold and answers with a bare address. ROMFS and TMPFS + are those. + +The pin is asked for first, because a filesystem that needs one is not +safe without it. The loader asks for a pin only if it can hold one, +which is the flat build, or the pin would stay for ever. + +A filesystem that answers neither is still usable. The loader then +copies the text to RAM, as it does for any other module. The module +loses the shared text and the flash saving, but it runs. + +The writable segment is allocated and copied per instance, and a pool of +function descriptors is reserved behind it for the relocations that ask +the loader to manufacture one. When the task starts, +``up_initial_state()`` installs the object's data base -- ``DT_PLTGOT``, +or the GOT immediately after ``PT_DYNAMIC`` in an object with no +imports -- into the PIC base register. + +Shared libraries +---------------- + +A module may name shared libraries in ``DT_NEEDED``. The loader loads +each one during relocation, through ``libelf_insert()`` as ``dlopen()`` +does, and then binds the module's undefined symbols against that +library's exports. ``CONFIG_LIBC_ELF_MAXDEPEND`` caps how many one module +may name, and a module that names more is refused. With +``CONFIG_ARCH_ADDRENV`` a module carrying ``DT_NEEDED`` is refused, because +the library would not be in the address space of the program. + +Libraries are found the way ``dlopen()`` finds them: an absolute path is +used as given, and a bare name is searched for along ``LD_LIBRARY_PATH``, +which needs ``CONFIG_LIBC_ENVPATH`` and is seeded from +``CONFIG_LDPATH_INITIAL``. + +A library lands in the module registry, which holds one instance per +name, so its data is shared by everything that opens it. A module +started with ``exec()`` is different: that path loads a fresh copy each +time, so two running instances of one module have separate data while +sharing one copy of the text in flash. + +Comparison with NXFLAT and PIC ELF +================================== + +All three run position independent code from flash on a target with no +MMU, and all three give several instances of one module a shared +``.text`` with private ``.data``. They differ in what a *pointer* can +express and in what the toolchain has to provide. + +========================= ============== ============== ============= +Property NXFLAT PIC ELF FDPIC +========================= ============== ============== ============= +Format NuttX only ELF ELF +Extra build tools yes none linker +Data base per task task object +Shared libraries no no yes +Foreign-thread callback no no yes +Instruction set ARM, Thumb-2 unrestricted Thumb-2 only +========================= ============== ============== ============= + +:ref:`NXFLAT <nxflat>` is a NuttX-specific format. A module imports +symbols from the base firmware but cannot export any, so shared +libraries are not possible, and the build needs ``mknxflat`` to generate +a thunk, ``ldnxflat`` to link, and one of the ``binfmt/libnxflat`` linker +scripts to place the sections. + +**PIC ELF** needs no extra tools. With ``CONFIG_PIC`` the ELF loader +allocates the writable sections separately and, when the filesystem +answers ``FIOC_XIPBASE``, leaves the read-only ones on the media. Two +limits follow from having one base register per task: a shared object is +loaded as a single allocation, because the distance between its text and +its data is compiled into it, and the data base is installed once per +task, so every object in a task shares one. + +**FDPIC** pays for its descriptors with an ``arm-uclinuxfdpiceabi`` +linker, and gets back the two things a single register +cannot express. A task or pthread that a module starts inherits the +module's D-Space, so a register would be enough there; a work queue +worker was created at boot and carries no module base, and a descriptor +supplies one, which is how ``SIGEV_THREAD`` notifications reach module +code. + +Requirements +============ + +**An ARM Thumb-2 core.** The boundary is the instruction set, not the +core profile: GCC rejects FDPIC in Thumb-1 mode. + +========================= ========================== ===== +Core Architecture FDPIC +========================= ========================== ===== +Cortex-M3 / M4 / M7 ARMv7-M / ARMv7E-M yes +Cortex-M33 ARMv8-M Mainline yes +Cortex-M0 / M0+ / M23 ARMv6-M / ARMv8-M Baseline no +========================= ========================== ===== + +RISC-V has no FDPIC ABI -- the psABI addendum is an unmerged proposal and +no ``EI_OSABI`` value is assigned -- so a RISC-V target cannot use this. + +**Flash that is memory mapped and executable**, exposed by a filesystem +that answers ``XIPFSIOC_PIN`` or ``FIOC_XIPBASE``. This is what gives +execute in place. Without it the module still loads, but from RAM. + +**An FDPIC linker.** A stock ``arm-none-eabi`` GCC compiles correct +FDPIC code for both C and C++. Its assembler accepts the relocations that +code produces in FDPIC mode only: GCC 14 and later select that mode for +``-mfdpic``, and for an older GCC the build passes ``-Wa,--fdpic``. Only +``arm-uclinuxfdpiceabi`` binutils carry the ``armelf_linux_fdpiceabi`` +emulation that the link needs, and ``arm-none-eabi-ld`` rejects it. + +No distribution packages that target, so build binutils for it -- which +takes about a minute and needs nothing else:: + + configure --target=arm-uclinuxfdpiceabi --prefix=$HOME/fdpic \ + --disable-nls --disable-werror + make && make install + export PATH=$HOME/fdpic/bin:$PATH + +An FDPIC GCC is not needed. + +**The base firmware must reserve r9.** It is not enough for the module +to be well behaved: a firmware routine calling back into module code +arrives with the module's data base in r9 only if the compiler was never +free to allocate that register elsewhere. ``CONFIG_FDPIC`` selects +``CONFIG_PIC``, under which ``arch/arm/src/common/Toolchain.defs`` adds +``--fixed-r9``; see :ref:`nxflat` for why it goes into ``ARCHCFLAGS`` +rather than ``CFLAGS`` and how to check that it arrived. + +Configuration +============= + +``CONFIG_FDPIC`` lives under ``CONFIG_ELF``. A working configuration +also needs a symbol table for modules to import from and a filesystem +that can expose its media:: + + CONFIG_ELF=y + CONFIG_FDPIC=y + CONFIG_LIBC_EXECFUNCS=y + CONFIG_EXECFUNCS_HAVE_SYMTAB=y + CONFIG_EXECFUNCS_SYSTEM_SYMTAB=y + CONFIG_FS_XIPFS=y + +``crt0`` runs the constructors of a module only with +``CONFIG_HAVE_CXXINITIALIZE``. + +Shared libraries need three more. A library resolves its own imports +against the table of ``CONFIG_LIBC_ELF_HAVE_SYMTAB``, and the last two let +a library be named rather than spelled out as an absolute path:: + + CONFIG_LIBC_ELF_HAVE_SYMTAB=y + CONFIG_LIBC_ENVPATH=y + CONFIG_LDPATH_INITIAL="/mnt/xipfs" + +``dlopen()`` needs ``CONFIG_LIBC_DLFCN`` too. + +Only the make build generates the table of +``CONFIG_EXECFUNCS_SYSTEM_SYMTAB``. A CMake configuration needs a symbol +table from elsewhere. + +``CONFIG_ELF_STACKSIZE`` gives the stack a module runs with. A module +that needs a different one can export an ``nx_stacksize`` symbol, which +the loader prefers when present. + +Building a module +================= + +Select ``CONFIG_FDPIC`` and a module is built by the ordinary in-tree ELF +build: the same ``MODULE = m`` in the same application Makefile as any +other, and the same ``crt0``. Nothing else is needed. + +What the build does differently is add compiler flags and use a +different linker:: + + arm-none-eabi-gcc -mcpu=cortex-m33 -mthumb -mfdpic -fPIC -Wa,--fdpic \ + -Os -fno-builtin -D__NuttX__ -I$NUTTX/include -c mod.c -o mod.o + + arm-uclinuxfdpiceabi-ld -m armelf_linux_fdpiceabi -shared -z now \ + -e _start -T $NUTTX/libs/libc/elf/gnu-elf.ld \ + -o mod $NUTTX/arch/arm/src/crt0.o mod.o + +Only the link needs the FDPIC toolchain. The stock compiler emits correct +FDPIC objects for both C and C++, its assembler included once it is in +FDPIC mode. That linker is in the NuttX CI image; +``tools/ci/docker/linux/Dockerfile`` shows how it is built. +``FDPIC_CROSSDEV`` names a different prefix, and the build says so if it +is missing. + +Six flags carry weight: + +* ``-mfdpic`` is stated rather than assumed, so a mis-set toolchain fails + loudly instead of producing a plain ELF the loader will not recognize. + +* ``-fPIC`` is not implied by ``-mfdpic`` on a bare-metal target, and + without it the link emits ``TEXTREL``. Text relocations cannot work + against text executed from read-only flash. + +* ``-Wa,--fdpic`` puts the assembler in FDPIC mode, which GCC before 14 + does not do for ``-mfdpic``. Without it the assembler stops with + "Relocation supported only in FDPIC mode". + +* ``-shared`` preserves the ``R_ARM_FUNCDESC_VALUE`` relocations for + imported symbols. A PIE link with ``--unresolved-symbols=ignore-all`` + appears to work but degrades every import to ``R_ARM_NONE``, and the + module branches to zero on its first call into the firmware. + +* ``-m armelf_linux_fdpiceabi`` is required: this linker supports several + emulations and will not guess. + +* ``-e _start`` names the entry point. ``crt0.c`` is the module's own + start-up file, the one every other module uses: it walks ``.init_array`` + and then calls ``main``. A shared library is never entered, so it is + linked without ``crt0``. + +A module links with ``-shared``, so importing something the firmware does +not export links cleanly and fails only on the target. Checking the +undefined symbols of the module against the generated +``libs/libc/exec_symtab.c`` catches that at build time. + +When the toolchain cannot make an FDPIC module, the build stops with one +of these: + +``CONFIG_FDPIC needs arm-uclinuxfdpiceabi-ld, which is not on PATH`` + The FDPIC linker is missing. The CMake build stops with the same + message when it configures. + +``unrecognised emulation mode: armelf_linux_fdpiceabi`` + ``FDPIC_CROSSDEV`` names a linker without the FDPIC emulation, such as + ``arm-none-eabi-``. + +``Relocation supported only in FDPIC mode`` + The assembler is not in FDPIC mode. A compile that does not use the + module flags has lost ``-Wa,--fdpic``. + +``unknown argument: '-mfdpic'`` + The compiler cannot make FDPIC code. Clang is one. + +Building a shared library +------------------------- + +A library is an ordinary shared library of the application build, +``BUILD_SHARED_LIBRARY`` in make and ``DYNLIB y`` in CMake, with a +soname:: + + LOCAL_MODULE := libfoo + LOCAL_MODULE_FILENAME := libfoo.so + LOCAL_SRC_FILES := libfoo.c + LOCAL_LDFLAGS := -soname libfoo.so + + include $(BUILD_SHARED_LIBRARY) + +The module flags hide every symbol, so a library marks what it exports +with ``visibility_default``. A module that uses the library names it on +its link line, which records the soname in ``DT_NEEDED``; in a module +Makefile, ``LDLIBS`` for the module target does that. +``apps/examples/fdpicxip/modules`` has both kinds. + +At run time the library must be reachable under its soname along +``LD_LIBRARY_PATH``. + +Calling back into a module +========================== + +A module's function pointer is the address of a descriptor in its +writable segment. Firmware that stores one and later branches to it +would jump into RAM data, so an entry point that accepts a callback from +a module has to resolve the descriptor first. ``CONFIG_FDPIC`` makes +these do so: + +``qsort``, ``bsearch``, ``pthread_create``, ``signal``/``sigaction``, +``task_create``/``task_create_with_stack``, ``task_spawn``, +``pthread_once``, ``scandir``, and ``mq_notify``/``timer_create`` with +``SIGEV_THREAD``. + +Whether a pointer is a descriptor is decided by reading the PIC base +register: a module's task runs with its data base there, a firmware task +with zero, so a kernel caller is unaffected. + +A new entry point that takes a module callback must resolve it too, under +three rules: + +* **Resolve once, in the innermost common routine.** Resolving twice + treats a code address as a descriptor. ``qsort()`` recurses, so its + public entry resolves and the recursive body does not; ``signal()`` + does not resolve because ``nxsig_action()`` does it for both paths; + ``scandir()`` resolves its filter but not the comparison function it + hands to ``qsort()``. + +* **Exclude sentinel values by hand.** ``fdpic_callback()`` declines to + dereference NULL and nothing else. ``sigaction()`` excludes + ``SIG_IGN``, ``SIG_DFL``, ``SIG_HOLD`` and ``SIG_ERR`` -- the integers + 0, 1, 2 and -1. + +* **A callback on a shared thread needs its base installed.** A + ``SIGEV_THREAD`` notification runs on a work queue worker that carries + no module base, so resolving the entry is not enough. Capture the base + at registration with ``fdpic_base()``, in the module's own context, and + install it around the call with ``fdpic_invoke()``. + +Everywhere else the callback runs in a task that inherited the module's +D-Space, so only the code address needs resolving. + +Examples and tests +================== + +``apps/examples/fdpicxip`` writes modules into XIPFS at run time and runs +them. ``fdpicxip qsort`` runs one module twice, with one copy of its +text; ``solib`` adds a shared library; ``cxx`` does the same in C++ and +checks that the constructors ran; ``jmprel`` runs a module whose imports +are bound through ``DT_JMPREL``. + +``apps/testing/fs/xipfs`` asserts what the demo shows. ``xipfs_test +fdpic`` loads modules and checks the loader, and ``xipfs_test reject`` +checks that a malformed or unloadable module is refused. Both apps build +their modules from ``apps/examples/fdpicxip/modules`` with the module +flags of the tree. + +``pimoroni-pico-2-plus:xipfs-fdpic`` is a board configuration with all of +this. Without hardware, ``mps2-an500:xipfs`` runs it under QEMU with +these added:: + + CONFIG_ELF=y + CONFIG_FDPIC=y + CONFIG_EXAMPLES_FDPICXIP=y + CONFIG_LIBC_EXECFUNCS=y + CONFIG_EXECFUNCS_HAVE_SYMTAB=y + CONFIG_EXECFUNCS_SYSTEM_SYMTAB=y + CONFIG_LIBC_DLFCN=y + CONFIG_LIBC_ELF_HAVE_SYMTAB=y + CONFIG_LIBC_ENVPATH=y + CONFIG_LDPATH_INITIAL="/mnt/xipfs" + CONFIG_SIG_EVTHREAD=y + CONFIG_SCHED_HPWORK=y + CONFIG_INIT_STACKSIZE=16384 + +and with ``CONFIG_DISABLE_POSIX_TIMERS``, ``CONFIG_PROFILE_ALL``, +``CONFIG_PROFILE_MINI`` and ``CONFIG_SYSTEM_GPROF`` off. + +Limitations +=========== + +**Tested in the flat build only.** In a protected build, a module that +``exec()`` started faulted at its entry point when last tested, because +the kernel side of the loader places the module in the kernel heap. A Review Comment: what do you mean? per-module data base (r9) should live in the task TLS? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
