I have some new insights about what happened here. The compiler warnings in nx_start.c that were the reason I started this thread:
init/nx_start.c: In function 'nx_start': init/nx_start.c:552:14: warning: unused variable 'heap_size' [-Wunused-variable] size_t heap_size; ^~~~~~~~~ init/nx_start.c:551:17: warning: unused variable 'heap_start' [-Wunused-variable] FAR void *heap_start; ^~~~~~~~~~ These warnings were appearing when I built my custom board configuration, based on a Tiva chip. When I switched to the tm4c1294-launchpad:nsh configuration, based on the same Tiva chip, the warnings disappeared! Comparing the two boards' defconfig files, and then their .config files, I saw no differences that should change the presence of the define MM_KERNEL_USRHEAP_INIT. To help diagnose why the two configurations behaved differently, I added the following to mm.h, just below the define of MM_KERNEL_USRHEAP_INIT: #if defined(CONFIG_BUILD_KERNEL) #warning "CONFIG_BUILD_KERNEL is defined" #else #warning "CONFIG_BUILD_KERNEL is NOT defined" #endif #if defined(__KERNEL__) #warning "__KERNEL__ is defined" #else #warning "__KERNEL__ is NOT defined" #endif #if defined(MM_KERNEL_USRHEAP_INIT) #warning "MM_KERNEL_USRHEAP_INIT is defined" #else #warning "MM_KERNEL_USRHEAP_INIT is NOT defined" #endif and ran make, redirecting stdout and stderr to a file. This produced a ton of output, but I saw that in both configurations, some files were built with __KERNEL__ defined, which causes MM_KERNEL_USRHEAP_INIT to be defined, and others were built without __KERNEL__ defined, which causes MM_KERNEL_USRHEAP_INIT to be undefined. The difference between the two configurations is that in my custom board, nx_start.c was being built without __KERNEL__ defined, whereas in tm4c1294-launchpad, nx_start.c was built *with* __KERNEL__ defined. After much more head-scratching, I finally traced this to a change in commit # 459ad9937377a42785692098ff0d73baaa9551e6: commit 459ad9937377a42785692098ff0d73baaa9551e6 Author: liuhaitao <liuhai...@xiaomi.com> Date: Fri Apr 10 10:37:54 2020 +0800 Use EXTRAFLAGS instead of EXTRADEFINES to be used by make via command line So call 'make EXTRAFLAGS=-Wno-cpp' could suppress the warnings with pre-processor directive #warning in GCC. Change-Id: Iaa618238924c9969bf91db22117b39e6d2fc9bb6 Signed-off-by: liuhaitao <liuhai...@xiaomi.com> All other boards' Make.defs were updated during that change to $(EXTRAFLAGS) but my custom boards remained with $(EXTRADEFINES). So: If anyone else with custom board configurations sees the warnings at the top of this email, look to see if you've updated your custom Make.defs for the EXTRADEFINES -> EXTRAFLAGS change. Apparently, __KERNEL__ gets defined (or not defined) differently for some files than for others. Evidently it comes from EXTRAFLAGS when building nx_start.c, but it comes from elsewhere when building files in arch/arm/src/armv7-m. Anyway mystery solved. I'll go create that PR now. Cheers, Nathan