Gregory, thank you for the analysis.
The crashes happened during system booting up, mostly at "AppBringup" or "rptun" threads, as per the
assertion logs. The other threads existing are the "idle" and the "lpwork" threads as per the sched logs.
There should be no other threads as NSH creation is still ongoing. As for interruptions, the UART and IPI are running
in kernel space and MTIMER are in NuttSBI space. The NSH is loaded from a RPMSGFS volume, thus there are a lot RPMSG
communications.
Is the KASAN proper for use in Kernel mode?
With MM_KASAN_ALL it reports a read access error:
BCkasan_report: kasan detected a read access error, address at 0x708fe90,size
is 8, return address: 0x701aeac
_assert: Assertion failed panic: at file: kasan/kasan.c:117 task: Idle_Task
process: Kernel 0x70023c0
The call stack looks like:
#0 _assert (filename=0x7060f78 "kasan/kasan.c", linenum=117, msg=0x7060ff0 "panic", regs=0x7082720
<g_last_regs>) at misc/assert.c:536#1 0x0000000007010248 in __assert (filename=0x7060f78
"kasan/kasan.c", linenum=117, msg=0x7060ff0 "panic") at assert/lib_assert.c:36
#2 0x00000000070141d6 in kasan_report (addr=0x708fe90, size=8, is_write=false,
return_address=0x701aeac <riscv_swint+114>) at kasan/kasan.c:117
#3 0x0000000007014412 in kasan_check_report (addr=0x708fe90, size=8,
is_write=false, return_address=0x701aeac <riscv_swint+114>) at kasan/kasan.c:190
#4 0x000000000701468c in __asan_load8_noabort (addr=0x708fe90) at
kasan/kasan.c:315
#5 0x000000000701aeac in riscv_swint (irq=0, context=0x708fe40, arg=0x0)
at common/riscv_swint.c:133
#6 0x000000000701b8fe in riscv_perform_syscall (regs=0x708fe40) at
common/supervisor/riscv_perform_syscall.c:45
#7 0x0000000007000570 in sys_call6 ()
With MM_KASAN_DISABLE_READ_CHECKS=y, it reports:
_assert: Assertion failed : at file: mm_heap/mm_malloc.c:245 task: rptun
process: Kernel 0x704a030
The call stack is:
#0 _assert (filename=0x7056060 "mm_heap/mm_malloc.c", linenum=245, msg=0x0, regs=0x7082720
<g_last_regs>) at misc/assert.c:536#1 0x000000000700df18 in __assert (filename=0x7056060
"mm_heap/mm_malloc.c", linenum=245, msg=0x0) at assert/lib_assert.c:36
#2 0x0000000007013082 in mm_malloc (heap=0x7089c00, size=128) at
mm_heap/mm_malloc.c:245
#3 0x0000000007011694 in kmm_malloc (size=128) at kmm_heap/kmm_malloc.c:51
#4 0x000000000704efd4 in metal_allocate_memory (size=128) at
.../nuttx/include/metal/system/nuttx/alloc.h:27
#5 0x000000000704fd8a in rproc_virtio_create_vdev (role=1, notifyid=0,
rsc=0x80200050, rsc_io=0x7080408 <metal_io_region_>,
priv=0x708ecd8,
notify=0x704e6d2 <remoteproc_virtio_notify>, rst_cb=0x0)
at open-amp/lib/remoteproc/remoteproc_virtio.c:356
#6 0x000000000704e956 in remoteproc_create_virtio (rproc=0x708ecd8,
vdev_id=0, role=1, rst_cb=0x0) at
open-amp/lib/remoteproc/remoteproc.c:957
#7 0x000000000704b1ee in rptun_dev_start (rproc=0x708ecd8)
at rptun/rptun.c:757
#8 0x0000000007049ff8 in rptun_start_worker (arg=0x708eac0)
at rptun/rptun.c:233
#9 0x000000000704a0ac in rptun_thread (argc=3, argv=0x7092010)
at rptun/rptun.c:253
#10 0x000000000700437e in nxtask_start () at task/task_start.c:107
This looks like already corrupted.
I also noticed there is a "mm_checkcorruption()" function, not sure how to use
it yet.
Regards,
yf
Original
From:"Gregory Nutt"< spudan...@gmail.com >;
Date:2024/3/11 1:43
To:"dev"< dev@nuttx.apache.org >;
Subject:Re: mm/mm_heap assertion error
On 3/10/2024 4:38 AM, yfliu2008 wrote:
> Dear experts,
>
>
>
>
> When doing regression check on K230 with a previously working Kernel mode
configuration, I got assertion error like below:
>
>
>
> #0 _assert (filename=0x704c598 "mm_heap/mm_malloc.c", linenum=245,
msg=0x0,regs=0x7082730 > #2 0x00000000070110f0 in mm_malloc (heap=0x7089c00, size=112)
at mm_heap/mm_malloc.c:245
> #3 0x000000000700fd74 in kmm_malloc (size=112) at
kmm_heap/kmm_malloc.c:51
> #4 0x0000000007028d4e in elf_loadphdrs (loadinfo=0x7090550) at
libelf/libelf_sections.c:207
> #5 0x0000000007028b0c in elf_load (loadinfo=0x7090550) at
libelf/libelf_load.c:337
> #6 0x00000000070278aa in elf_loadbinary (binp=0x708f5d0, filename=0x704bca8
"/system/bin/init", exports=0x0, nexports=0) at elf.c:257
> #7 0x00000000070293ea in load_absmodule (bin=0x708f5d0, filename=0x704bca8
"/system/bin/init", exports=0x0, nexports=0) at binfmt_loadmodule.c:115
> #8 0x0000000007029504 in load_module (bin=0x708f5d0, filename=0x704bca8
"/system/bin/init", exports=0x0, nexports=0) at binfmt_loadmodule.c:219
> #9 0x0000000007027674 in exec_internal (filename=0x704bca8
"/system/bin/init", argv=0x70907a0, envp=0x0, exports=0x0, nexports=0, actions=0x0,
attr=0x7090788, spawn=true) at binfmt_exec.c:98
> #10 0x000000000702779c in exec_spawn (filename=0x704bca8
"/system/bin/init", argv=0x70907a0, envp=0x0, exports=0x0, nexports=0, actions=0x0,
attr=0x7090788) at binfmt_exec.c:220
> #11 0x000000000700299e in nx_start_application () at init/nx_bringup.c:375
> #12 0x00000000070029f0 in nx_start_task (argc=1, argv=0x7090010) at
init/nx_bringup.c:403
> #13 0x0000000007003f84 in nxtask_start () at task/task_start.c:107
>
>
>
> It looks like mm/mm_heap data structure consistency was broken. As I am
unfamilar with these internals, I am looking forward to any hints about how
to find the root cause.
>
>
>
>
>
>
>
> Regards,
>
> yf
This does indicate heap corruption:
240 /* Node next must be alloced,
otherwise it should be merged.
241 * Its prenode(the founded
node) must be free and
preceding should
242 * match with nodesize.
243 */
244
245 DEBUGASSERT(MM_NODE_IS_ALLOC(next)
&&
MM_PREVNODE_IS_FREE(next) &&
246
next->preceding == nodesize);
Heap corruption normally occurs when that this a wild write outside of
the allocated memory region. These kinds of wild writes may clobber
some other threads data and directory or indirectly clobber the heap
meta data. Trying to traverse the damages heap meta data is probably
the root cause of the problem.
Only a kernel thread or interrupt handler could damage the heap.
The cause of this corruption can be really difficult to find because the
reported error does not occur when the heap is damaged but may not
manifest itself until sometime later.
It is unlikely that anyone will be able to solve this by just talking
about it. It might be worth increasing some kernel thread heap sizes
just to eliminate that common cause.
