MainframeReboot commented on issue #12356:
URL: https://github.com/apache/nuttx/issues/12356#issuecomment-2138234172
@patacongo
I sat down and RTFM to fully understand the differences between crt0, crt1,
crti, crtbegin, crtn and crtend. This has helped me immensely and I have
managed to get static global instances to run on NuttX kernel by making the
necessary changes to NuttX.
I will explain my patches below but the executive summary is _use crt0 for
C, use crt1 for C++_ while also manually linking crti, crtbegin, crtn and
crtend for C++ applications only.
### crt0.c
The first modification was to remove C++ logic from crt0. This means that
all of the .ctors/.dtors arrays at the top (_sctors, sdtors etc.) are removed,
along with the the calls to exec_ctors and atexit(exec_dtors). The result is a
crt0.c file that looks like this (some comments stripped for size):
``` C
/****************************************************************************
* arch/risc-v/src/common/crt0.c
****************************************************************************/
#include <nuttx/arch.h>
#include <nuttx/addrenv.h>
#include <nuttx/compiler.h>
#include <nuttx/config.h>
#include <sys/types.h>
#include <syscall.h>
#include <stdlib.h>
#include <stdio.h>
#include "riscv_internal.h"
#ifdef CONFIG_BUILD_KERNEL
int main(int argc, char *argv[]);
static void sig_trampoline(void) naked_function;
static void sig_trampoline(void)
{
__asm__ __volatile__
(
" addi sp, sp, -" STACK_FRAME_SIZE "\n" /* Save ra on the stack */
REGSTORE " ra, 0(sp)\n"
" mv t0, a0\n" /* t0=sighand */
" mv a0, a1\n" /* a0=signo */
" mv a1, a2\n" /* a1=info */
" mv a2, a3\n" /* a2=ucontext */
" jalr t0\n" /* Call the signal handler (modifies ra) */
REGLOAD " ra, 0(sp)\n" /* Recover ra in sp */
" addi sp, sp, " STACK_FRAME_SIZE "\n"
" li a0, %0\n" /* SYS_signal_handler_return */
" ecall\n" /* Return from the SYSCALL */
" nop\n"
:
: "i" (SYS_signal_handler_return)
:
);
}
void __start(int argc, char *argv[])
{
int ret;
/* Initialize the reserved area at the beginning of the .bss/.data region
* that is visible to the RTOS.
*/
ARCH_DATA_RESERVE->ar_sigtramp = (addrenv_sigtramp_t)sig_trampoline;
/******************************************************************
* Do NOT include C++ constructor/destructor calls in this file.
* This file is for C applications only. Refer to crt1.c for C++.
*******************************************************************/
/* Call the main() entry point passing argc and argv. */
ret = main(argc, argv);
/* Call exit() if/when the main() returns */
exit(ret);
}
#endif /* CONFIG_BUILD_KERNEL */
```
### crt1.c
All C++ constructor/destructor logic was moved to the new file crt1.c. Note
that this is dfference from the original C++ logic in crt0.c. I have learned,
during my research, that .ctors and .dtors is the "old" way of doing things and
that the new recommended way of doing things is to use .preinit_array,
.init_array and .fini_array. The RISC-V toolchain that I use (xPack 13.2.0)
supports the new way so I have decided to follow the new convention. From my
reading of old RISC-V NuttX issues, it appears the NuttX team has moved on to
xPack 13.2.0 for testing so it _should_ work with other RISC-V boards as well.
Here is the new crt1.c file:
``` C
/****************************************************************************
* arch/risc-v/src/common/crt1.c
****************************************************************************/
#include <nuttx/arch.h>
#include <nuttx/addrenv.h>
#include <nuttx/compiler.h>
#include <nuttx/config.h>
#include <sys/types.h>
#include <syscall.h>
#include <stdlib.h>
#include "riscv_internal.h"
#ifdef CONFIG_BUILD_KERNEL
int main(int argc, char *argv[]);
static void sig_trampoline(void) naked_function;
static void sig_trampoline(void)
{
__asm__ __volatile__
(
" addi sp, sp, -" STACK_FRAME_SIZE "\n" /* Save ra on the stack */
REGSTORE " ra, 0(sp)\n"
" mv t0, a0\n" /* t0=sighand */
" mv a0, a1\n" /* a0=signo */
" mv a1, a2\n" /* a1=info */
" mv a2, a3\n" /* a2=ucontext */
" jalr t0\n" /* Call the signal handler (modifies ra) */
REGLOAD " ra, 0(sp)\n" /* Recover ra in sp */
" addi sp, sp, " STACK_FRAME_SIZE "\n"
" li a0, %0\n" /* SYS_signal_handler_return */
" ecall\n" /* Return from the SYSCALL */
" nop\n"
:
: "i" (SYS_signal_handler_return)
:
);
}
/****************************************************************************
* Public Data
****************************************************************************/
/*
Linker defined symbols to .preinit_array, .init_array and .fini_array.
.ctors and .dtors are not used by RISC-V.
*/
extern initializer_t __preinit_array_start[];
extern initializer_t __preinit_array_end[];
extern initializer_t __init_array_start[];
extern initializer_t __init_array_end[];
extern initializer_t __fini_array_start[];
extern initializer_t __fini_array_end[];
/****************************************************************************
* Private Functions
****************************************************************************/
#ifdef CONFIG_HAVE_CXX
/****************************************************************************
* Name: exec_preinit
*
* Description:
* Calls startup functions prior to main entry point
*
****************************************************************************/
static void exec_preinit(void)
{
initializer_t *preinit;
for(preinit = __preinit_array_start; preinit < __preinit_array_end;
++preinit)
{
initializer_t initializer = *preinit;
if (initializer)
{
initializer();
}
}
}
/****************************************************************************
* Name: exec_preinit
*
* Description:
* Calls static constructors prior to main entry point
*
****************************************************************************/
static void exec_init(void)
{
initializer_t *init;
for(init = __init_array_start; init < __init_array_end; ++init)
{
initializer_t initializer = *init;
if (initializer)
{
initializer();
}
}
}
/****************************************************************************
* Name: exec_fini
*
* Description:
* Calls static destructors using atexit
*
****************************************************************************/
static void exec_fini(void)
{
initializer_t *fini;
for(fini = __fini_array_start; fini < __fini_array_end; ++fini)
{
initializer_t initializer = *fini;
if (initializer)
{
initializer();
}
}
}
#endif
void __start(int argc, char *argv[])
{
int ret;
/* Initialize the reserved area at the beginning of the .bss/.data region
* that is visible to the RTOS.
*/
ARCH_DATA_RESERVE->ar_sigtramp = (addrenv_sigtramp_t)sig_trampoline;
#ifdef CONFIG_HAVE_CXX
/* Call preinit functions */
exec_preinit();
/* Call C++ constructors */
exec_init();
/* Setup so that C++ destructors called on task exit */
atexit(exec_fini);
#endif
/* Call the main() entry point passing argc and argv. */
ret = main(argc, argv);
/* Call exit() if/when the main() returns */
exit(ret);
}
#endif /* CONFIG_BUILD_KERNEL */
```
### gnu-elf.ld
In order to make sure the arrays in the crt1.c file are populated, the
linker script was updated to match:
```
/****************************************************************************
* boards/risc-v/mpfs/icicle/scripts/gnu-elf.ld
****************************************************************************/
SECTIONS
{
.text 0xC0000000 :
{
_stext = . ;
*(.text)
*(.text.*)
*(.gnu.warning)
*(.stub)
*(.glue_7)
*(.glue_7t)
*(.jcr)
/* C++ support: The .init and .fini sections contain specific logic
* to manage static constructors and destructors.
*/
*(.gnu.linkonce.t.*)
*(.init) /* Old ABI */
*(.fini) /* Old ABI */
_etext = . ;
}
.rodata :
{
_srodata = . ;
*(.rodata)
*(.rodata1)
*(.rodata.*)
*(.gnu.linkonce.r*)
_erodata = . ;
}
.data 0xC0101000:
{
_sdata = . ;
*(.data)
*(.data1)
*(.data.*)
*(.gnu.linkonce.d*)
. = ALIGN(4);
_edata = . ;
}
/* C++ support. For each global and static local C++ object,
* GCC creates a small subroutine to construct the object. Pointers
* to these routines (not the routines themselves) are stored as
* simple, linear arrays in the .ctors section of the object file.
* Similarly, pointers to global/static destructor routines are
* stored in .dtors.
*/
.preinit_array :
{
PROVIDE(__preinit_array_start = .);
KEEP(*(.preinit_array*))
PROVIDE(__preinit_array_end = .);
}
.init_array :
{
PROVIDE(__init_array_start = .);
KEEP(*(SORT_BY_INIT_PRIORITY(.init_array.*)
SORT_BY_INIT_PRIORITY(.ctors.*)))
KEEP(*(.init_array .ctors))
PROVIDE(__init_array_end = .);
}
.fini_array :
{
PROVIDE(__fini_array_start = .);
KEEP(*(SORT_BY_INIT_PRIORITY(.fini_array.*)
SORT_BY_INIT_PRIORITY(.dtors.*)))
KEEP(*(.fini_array .dtors))
PROVIDE(__fini_array_end = .);
}
/* Thread local storage support */
.tdata :
{
_stdata = ABSOLUTE(.);
KEEP (*(.tdata .tdata.* .gnu.linkonce.td.*));
_etdata = ABSOLUTE(.);
}
.tbss :
{
_stbss = ABSOLUTE(.);
KEEP (*(.tbss .tbss.* .gnu.linkonce.tb.* .tcommon));
_etbss = ABSOLUTE(.);
}
.bss :
{
_sbss = . ;
*(.bss)
*(.bss.*)
*(.sbss)
*(.sbss.*)
*(.gnu.linkonce.b*)
*(COMMON)
_ebss = . ;
}
/* Stabs debugging sections. */
.stab 0 : { *(.stab) }
.stabstr 0 : { *(.stabstr) }
.stab.excl 0 : { *(.stab.excl) }
.stab.exclstr 0 : { *(.stab.exclstr) }
.stab.index 0 : { *(.stab.index) }
.stab.indexstr 0 : { *(.stab.indexstr) }
.comment 0 : { *(.comment) }
.debug_abbrev 0 : { *(.debug_abbrev) }
.debug_info 0 : { *(.debug_info) }
.debug_line 0 : { *(.debug_line) }
.debug_pubnames 0 : { *(.debug_pubnames) }
.debug_aranges 0 : { *(.debug_aranges) }
}
```
### Compilation & Linking
I had to modify how my kernel applications were compiled and linked in order
to support the new crt0.c and crt1.c files. To do this, I modified the
`toolchain.cmake.export` file under `nuttx/tools`. Below is a snippet of the
modification:
``` CMake
file(GLOB CSTARTUP_OBJS ${NUTTX_PATH}/startup/*)
file(GLOB CXXSTARTUP_OBJS ${NUTTX_PATH}/startup/*)
add_compile_options($<$COMPILE_LANGUAGE:C>:-nostdlib>)
add_compile_options($<$COMPILE_LANGUAGE:CXX>:-nodefaultlibs$<SEMICONON>-nostartfiles>)
set(CMAKE_C_LINK_EXECUTABLE
"<CMAKE_LINKER> ${LDFLAGS} --entry=__start -T${LINKER_SCRIPT} <OBJECTS>
${CSTARTUP_OBJS} -o <TARGET> <LINK_LIBRARIES> -L${NUTTX_PATH}/libs
--start-group ${LDLIBS} ${EXTRA_LIBS} --end-group"
)
set(CMAKE_CXX_LINK_EXECUTABLE
"<CMAKE_LINKER> ${LDFLAGS} --entry=__start -T${LINKER_SCRIPT} <OBJECTS>
${CXXSTARTUP_OBJS} -o <TARGET> <LINK_LIBRARIES> -L${NUTTX_PATH}/libs
--start-group ${LDLIBS} ${EXTRA_LIBS} --end-group"
)
```
The above modifications ensure that C++ is not linked with any standard
libraries nor is it linked with any toolchain start files (I do this manually
later on) while C is. It also splits the STARTUP_OBJS variable into a
CSTARTUP_OBJS and a CXXSTARTUP_OBJS variable so I can set one to crt0.o and the
other to crt1.o. The modification to STARTUP_OBJS is done in
`arch/risc-v/src/common/Make.defs`.
Inside the `Makefile` found within `arch/risc-v/src`, I added a section to
ensure crt1 was compiled, right under crt0:
``` Makefile
crt0$(OBJEXT): %$(OBJEXT): %.c
$(call COMPILE, $<, $@)
crt1$(OBJEXT): %$(OBJEXT): %.c
$(call COMPILE, $<, $@)
```
Additionally, I had to make sure crt1.o was exported when `make export` is
run so that it appears under apps/import/startup. Inside the same `Makefile`
found within `arch/risc-v/src`, I modified `export_startup` to contain
CXXSTARTUP_OBJS:
``` Makefile
export_startup: $(CSTARTUP_OBJS) $(CXXSTARTUP_OBJS)
ifneq ($(CSTARTUP_OBJS),)
$(Q) if [ -d "$(EXPORT_DIR)/startup" ]; then \
cp -f $(CSTARTUP_OBJS) "$(EXPORT_DIR)/startup/."; \
else \
echo "$(EXPORT_DIR)/startup does not exist"; \
exit 1; \
fi
endif
ifneq ($(CXXSTARTUP_OBJS),)
$(Q) if [ -d "$(EXPORT_DIR)/startup" ]; then \
cp -f $(CXXSTARTUP_OBJS) "$(EXPORT_DIR)/startup/."; \
else \
echo "$(EXPORT_DIR)/startup does not exist"; \
exit 1; \
fi
endif
```
Now that my crt0.o and crt1.o files are built and exported, I had to make
sure they were used while building my apps. To do this, I modified the
`Make.defs` file found inside `apps/import/Make.defs` to remove default libs
and start up files from only the C++ apps:
``` Makefile
ARCHCFLAGS += -fno-common -pipe
ARCHCXXFLAGS += -fno-common -nostdinc++ -pipe -nodefaultlibs -nostartfiles
```
As I am manually linking the C runtime files for C++, I also added some
variables to the top of the file for these objects:
``` Makefile
ARCHCRT0OBJ = $(call
CONVERT_PATH,$(TOPDIR)$(DELIM)startup$(DELIM)crt0$(OBJEXT))
ARCHCRT1OBJ = $(call
CONVERT_PATH,$(TOPDIR)$(DELIM)startup$(DELIM)crt1$(OBJEXT))
ARCHCRTIOBJ = $(wildcard $(shell $(CC) $(ARCHCPUFLAGS)
--print-file-name=crti.o))
ARCHCRTBEGINOBJ = $(wildcard $(shell $(CC) $(ARCHCPUFLAGS)
--print-file-name=crtbegin.o))
ARCHCRTENDOBJ = $(wildcard $(shell $(CC) $(ARCHCPUFLAGS)
--print-file-name=crtend.o))
ARCHCRTNOBJ = $(wildcard $(shell $(CC) $(ARCHCPUFLAGS)
--print-file-name=crtn.o))
```
The last modification involves the `Application.mk` file under `/apps`.
Here, I updated the `ELFLD` function to perform a check of the incoming object
and then call another function to perform the linking depending on whether it's
a pure C application or a C++ application:
``` Makefile
define ELFLDC
$(info From ELFLDC: $1 $2)
$(Q) $(LD) $(LDELFFLAGS) $(LDLIBPATH) $(ARCHCRT0OBJ) $1 $(LDSTARTGROUP)
$(LDLIBS) $(LDENDGROUP) -o $2
endef
define ELFLDCXX
$(info From ELFLDCXX: $1 $2)
$(Q) $(LD) $(LDELFFLAGS) $(LDLIBPATH) $(ARCHCRT1OBJ) $(ARCHCRTIOBJ)
$(ARCHCRTBEGINOBJ) $1 $(LDSTARTGROUP) $(LDLIBS) $(LDENDGROUP) $(ARCHCRTENDOBJ)
$(ARCHCRTNOBJ) -o $2
endef
define ELFLD
$(ECHO_BEGIN)"LD: $2 "
$(if $(MAINCOBJ), $(call ELFLDC, $1, $2), $(call ELFLDCXX, $1, $2))
$(ECHO_END)
endef
```
Now when I perform a `make import`, the C applications are linked to crt0.o
while the C++ applications are linked to crt1.o as well as to crti.o,
crtbegin.o, crtn.o and crtend.o. The result is C++ applications now work with
static global instances:
The additional benefit is that C applications no longer have empty
.preinit_array/.init_array/.fini_array sections and no longer call
`exec_preinit`, `exec_init` and `atexit(exec_fini)` as there is no reason for
them to, they will never have any constructors/destructors in them.
I apologize for this insanely long post, but I wanted to document this in
the event anyone else wants to use NuttX kernel build with C++ apps and static
global instances. I also apologize if my "hacks" are unsightly, I am not an
expert at makefiles so I am sure I broke a number of best practices while
coming up with this solution. I am hoping now that I have a solution it can be
critiqued, and someone can let me know if this is an OK way of doing it.
Nonetheless, I have learned a ton getting the NuttX kernel running on the
Polarfire Icicle board and I couldn't have done it without your support, so
thank you again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
