On Fri, 2007-09-07 at 11:27 +0200, Peter Soetens wrote:
> Just in case you hooked off the long discussion about the issues we found from
> Xenomai 2.3.2 on:
>   o We are using the xeno_native skin, create Xeno tasks and semaphores, but 
> have strong indications that the crashes are caused by the memory allocation 
> scheme of Xenomai in combination with task creation/deletion
>   o We found two ways to break Xenomai, causing a 'Killed' (rt_task_delete) 
> and causing an OOPS (rt_task_join).
>   o They happen on 2.6.20 and 2.6.22 kernels
>   o On the 2.3 branch, r2429 works, r2433 causes the faults. The patch is 
> small, and in the ChangLog: 
> 2007-05-11  Philippe Gerum  <[EMAIL PROTECTED]>
>     * include/nucleus/heap.h (xnfreesafe): Use xnpod_current_p() when
>     checking for deferral.
>     * include/nucleus/pod.h (xnpod_current_p): Give exec mode
>     awareness to this predicate, checking for primary/secondary mode
>     of shadows.
> 2007-05-11  Gilles Chanteperdrix  <[EMAIL PROTECTED]>
>     * ksrc/skins: Always defer thread memory release in deletion hook
>     by calling xnheap_schedule_free() instead of xnfreesafe().
>   o We reverted this patch on HEAD of the 2.3 branch, but got -ENOMEM errors 
> during Xenomai resource allocations, indicating that later changes depend on 
> this patch. So we use clean HEAD again further on to find the causes:
>  o A first test (in Orocos) creates one thread, two semaphores, lets it wait 
> on them and cleans up the thread.

Please point me at the actual Orocos test code that breaks, with the
hope to get a fairly standalone test case from it; if you do have a
standalone test case already, this would be even better. I intend to
address this issue asap.

>  o During rt_task_delete, our program gets 'Killed' (without joinable 
> thread), 
> hence a user space problem. However, gdb is of no use, all thread info is 
> lost.

SIGKILL is sent from the nucleus upon a call to rt_task_delete() which
targets a non-current task, in order to make sure this user-space task
will go away from a Linux context, since we don't want the kernel TCB
Xenomai maintains for it, to be wiped out before the mated userland
thread has really exited. IOW, this case boils down to an asynchronous
cancellation, where Linux is asked to kick out the target task first,
Xenomai then catches the event and cleans up the TCB on its side

As per POSIX, a lethal signal sent to a single thread zaps all other
threads belonging to the same process, which explains why your process
dies. We could be a bit smarter by handling this situation using a
hidden exit from a trapped signal handler, I guess.

>  o We made the thread joinable (T_JOINABLE), and then joined. This bypassed 
> the Kill on the first run but causes an OOPS the second time the same 
> application is started:
> Oops: 0000 [#1]
> CPU:    0
> EIP:    0060:[<fef4a1f3>]    Not tainted VLI
> EFLAGS: 00010002   ( #2)
> EIP is at get_free_range+0x56/0x160 [xeno_nucleus]
> eax: f3a81d01   ebx: 00000200   ecx: 55550101   edx: fef62b00
> esi: 55550101   edi: 00000200   ebp: f0f33ec4   esp: f0f33e98
> ds: 007b   es: 007b   ss: 0068
> Process NonPeriodicActi (pid: 3020, ti=f0f32000 task=f7ce61b0 
> task.ti=f0f32000)
> Stack: 00000000 00000600 fef62b80 f3a81b24 f3a80000 fef62ba4 f3a80720 55550101
>        00000600 f0f33f18 f7ce6360 f0f33ee4 fef4a948 fef62b80 f0f33f08 00000000
>        00000400 f0f33f18 f7ce6360 f0f33f50 ff13e1de 00000282 00000282 bfab6350
> Call Trace:
>  [<c0103ffb>] show_trace_log_lvl+0x1f/0x35
>  [<c01040bb>] show_stack_log_lvl+0xaa/0xcf
>  [<c01042a9>] show_registers+0x1c9/0x392
>  [<c0104588>] die+0x116/0x245
>  [<c0110fca>] do_page_fault+0x287/0x61d
>  [<c010ea35>] __ipipe_handle_exception+0x63/0x136
>  [<c029466d>] error_code+0x79/0x88
>  [<fef4a948>] xnheap_alloc+0x15b/0x17d [xeno_nucleus]

The only explanation looking at this backtrace is that the system heap
has been corrupted by the previous exit; likely a side effect of the

>  [<ff13e1de>] __rt_task_create+0xe0/0x171 [xeno_native]
>  [<fef5655f>] losyscall_event+0xaf/0x170 [xeno_nucleus]
>  [<c0138804>] __ipipe_dispatch_event+0xc0/0x1da
>  [<c010e90b>] __ipipe_syscall_root+0x43/0x10a
>  [<c0102e79>] system_call+0x29/0x41
>  =======================
> Code: 74 61 85 c0 74 5d c7 45 e0 00 00 00 00 8b 4d e4 8b 49 10 89 4d ec 85 c9 
> 74 38 8b 45 dc 8b 78 0c 89 4d f0 89 ce 89 fb eb 02 89 ce <8b> 09 8d 04 3e 39 
> c1 0f 94 c2 3b 5d d8 0f 92 c0 01 fb 84 c2 75
> EIP: [<fef4a1f3>] get_free_range+0x56/0x160 [xeno_nucleus] SS:ESP 
> 0068:f0f33e98
> [hard lockup]
>   o Our application is also mixing the original RT_TASK struct and return 
> value of the rt_task_self() function call when calling rt_ functions. 
> Switching between one of those influences the crashing behaviour as well, not 
> further investigated.

This should not make any difference regarding the bug above. Both
methods boil down to returning an abstract handler to the task, which
serves as an index to the in-kernel TCB maintained by Xenomai. In any
case, this handle is fully validated before any use, and does not even
carry any memory pointer semantics. We are playing with PTDs to maintain
the rt_task_self() information from userland instead of retrieving it
systematically from a syscall, but would the associated memory be
corrupted, the crash would then be confined to the userland process.

>   o This was reproduced on two different systems (one with SMI workaround 
> working)
> You have the patch that broke things, I hope this gives you a hint on what 
> causes our crashes. Know that Orocos as-is has worked with Xenomai from  
> Xenomai 2.0 on.

> Peter

Xenomai-core mailing list

Reply via email to