On Fri, 2007-09-07 at 11:27 +0200, Peter Soetens wrote: > Just in case you hooked off the long discussion about the issues we found from > Xenomai 2.3.2 on: > > o We are using the xeno_native skin, create Xeno tasks and semaphores, but > have strong indications that the crashes are caused by the memory allocation > scheme of Xenomai in combination with task creation/deletion > o We found two ways to break Xenomai, causing a 'Killed' (rt_task_delete) > and causing an OOPS (rt_task_join). > o They happen on 2.6.20 and 2.6.22 kernels > o On the 2.3 branch, r2429 works, r2433 causes the faults. The patch is > small, and in the ChangLog: > > 2007-05-11 Philippe Gerum <[EMAIL PROTECTED]> > > * include/nucleus/heap.h (xnfreesafe): Use xnpod_current_p() when > checking for deferral. > > * include/nucleus/pod.h (xnpod_current_p): Give exec mode > awareness to this predicate, checking for primary/secondary mode > of shadows. > > 2007-05-11 Gilles Chanteperdrix <[EMAIL PROTECTED]> > > * ksrc/skins: Always defer thread memory release in deletion hook > by calling xnheap_schedule_free() instead of xnfreesafe(). > > o We reverted this patch on HEAD of the 2.3 branch, but got -ENOMEM errors > during Xenomai resource allocations, indicating that later changes depend on > this patch. So we use clean HEAD again further on to find the causes: > o A first test (in Orocos) creates one thread, two semaphores, lets it wait > on them and cleans up the thread.
Please point me at the actual Orocos test code that breaks, with the hope to get a fairly standalone test case from it; if you do have a standalone test case already, this would be even better. I intend to address this issue asap. > o During rt_task_delete, our program gets 'Killed' (without joinable > thread), > hence a user space problem. However, gdb is of no use, all thread info is > lost. SIGKILL is sent from the nucleus upon a call to rt_task_delete() which targets a non-current task, in order to make sure this user-space task will go away from a Linux context, since we don't want the kernel TCB Xenomai maintains for it, to be wiped out before the mated userland thread has really exited. IOW, this case boils down to an asynchronous cancellation, where Linux is asked to kick out the target task first, Xenomai then catches the event and cleans up the TCB on its side afterwise. As per POSIX, a lethal signal sent to a single thread zaps all other threads belonging to the same process, which explains why your process dies. We could be a bit smarter by handling this situation using a hidden exit from a trapped signal handler, I guess. > o We made the thread joinable (T_JOINABLE), and then joined. This bypassed > the Kill on the first run but causes an OOPS the second time the same > application is started: > > Oops: 0000 [#1] > PREEMPT > CPU: 0 > EIP: 0060:[<fef4a1f3>] Not tainted VLI > EFLAGS: 00010002 (2.6.20.9-ipipe-1.8-08 #2) > EIP is at get_free_range+0x56/0x160 [xeno_nucleus] > eax: f3a81d01 ebx: 00000200 ecx: 55550101 edx: fef62b00 > esi: 55550101 edi: 00000200 ebp: f0f33ec4 esp: f0f33e98 > ds: 007b es: 007b ss: 0068 > Process NonPeriodicActi (pid: 3020, ti=f0f32000 task=f7ce61b0 > task.ti=f0f32000) > Stack: 00000000 00000600 fef62b80 f3a81b24 f3a80000 fef62ba4 f3a80720 55550101 > 00000600 f0f33f18 f7ce6360 f0f33ee4 fef4a948 fef62b80 f0f33f08 00000000 > 00000400 f0f33f18 f7ce6360 f0f33f50 ff13e1de 00000282 00000282 bfab6350 > Call Trace: > [<c0103ffb>] show_trace_log_lvl+0x1f/0x35 > [<c01040bb>] show_stack_log_lvl+0xaa/0xcf > [<c01042a9>] show_registers+0x1c9/0x392 > [<c0104588>] die+0x116/0x245 > [<c0110fca>] do_page_fault+0x287/0x61d > [<c010ea35>] __ipipe_handle_exception+0x63/0x136 > [<c029466d>] error_code+0x79/0x88 > [<fef4a948>] xnheap_alloc+0x15b/0x17d [xeno_nucleus] The only explanation looking at this backtrace is that the system heap has been corrupted by the previous exit; likely a side effect of the deferral. > [<ff13e1de>] __rt_task_create+0xe0/0x171 [xeno_native] > [<fef5655f>] losyscall_event+0xaf/0x170 [xeno_nucleus] > [<c0138804>] __ipipe_dispatch_event+0xc0/0x1da > [<c010e90b>] __ipipe_syscall_root+0x43/0x10a > [<c0102e79>] system_call+0x29/0x41 > ======================= > Code: 74 61 85 c0 74 5d c7 45 e0 00 00 00 00 8b 4d e4 8b 49 10 89 4d ec 85 c9 > 74 38 8b 45 dc 8b 78 0c 89 4d f0 89 ce 89 fb eb 02 89 ce <8b> 09 8d 04 3e 39 > c1 0f 94 c2 3b 5d d8 0f 92 c0 01 fb 84 c2 75 > EIP: [<fef4a1f3>] get_free_range+0x56/0x160 [xeno_nucleus] SS:ESP > 0068:f0f33e98 > [hard lockup] > > o Our application is also mixing the original RT_TASK struct and return > value of the rt_task_self() function call when calling rt_ functions. > Switching between one of those influences the crashing behaviour as well, not > further investigated. > This should not make any difference regarding the bug above. Both methods boil down to returning an abstract handler to the task, which serves as an index to the in-kernel TCB maintained by Xenomai. In any case, this handle is fully validated before any use, and does not even carry any memory pointer semantics. We are playing with PTDs to maintain the rt_task_self() information from userland instead of retrieving it systematically from a syscall, but would the associated memory be corrupted, the crash would then be confined to the userland process. > o This was reproduced on two different systems (one with SMI workaround > working) > > You have the patch that broke things, I hope this gives you a hint on what > causes our crashes. Know that Orocos as-is has worked with Xenomai from > Xenomai 2.0 on. > > Peter > -- Philippe. _______________________________________________ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
