RE: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai cobalt context switch event
>-Original Message- >From: Jan Kiszka >Sent: Tuesday, December 14, 2021 3:51 PM >To: Chen, Hongzhan ; xenomai@xenomai.org >Subject: Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai >cobalt context switch event > >On 14.12.21 08:46, Chen, Hongzhan wrote: >> >>>>>> >>>>>> BTW, is that blue hollow box exclusive for our plugin? Or can this be >>>>>> customized? Is it explained somewhere in Kernelshark what hollow boxes >>>>>> mean? >>>>>> >>>>> >>>>> Some visualization issue: >>>>> >>>>>pthelper-912 [000] 753.948271: cobalt_schedule: >>>>> status=0x1000 >>>>>pthelper-912 [000] 753.948275: cobalt_switch_context: >>>>> prev_name=pthelper prev_pid=912 prev_prio=3 prev_state=0xa40042 ==> >>>>> next_name=hi-thread next_pid=913 next_prio=2 >>>>> hi-thread-913 [000] 753.948285: cobalt_shadow_hardened: >>>>> state=0x240040 info=0x0 >>>>> hi-thread-913 [000] 753.948423: cobalt_thread_fault: >>>>> ip=0xb6e34e20 type=0x8 >>>>> hi-thread-913 [000] 753.948737: cobalt_shadow_gorelax: >>>>> reason=syscall >>>>> >>>>> In this case, the task plot of hi-thread-919 does no show the blue box >>>>> between hardened and fault. >>>> >>>> Could you zoom out (click "-" in the top left button) to check if blue box >>>> would be reshowing? >>>> Could you paste more trailing log in your log to check if there another >>>> cobalt_switch_context. Current logic is create hollow box between two >>>> cobalt_switch_context. If there is only cobalt switch_context event >>>> matching, it would not create hollow box correctly. >>>> >>> >>> Yeah, the problem is indeed that the bar is only drawn if beg AND end >>> points are visible - that is sure not useful. Does the drawing concept >>> of kernelshark imposes this limitation? >> >> This should be common issue in kernelshark. I also observed same >> issue on hollow red box for preempt latency implementation. >> > >Ok, another topic to discuss with their community :) It is OK to submit this patch set to kernel-shark community to review if it meet our requirement? Regards Hongzhan Chen > >Jan > >-- >Siemens AG, T RDA IOT >Corporate Competence Center Embedded Linux
Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai cobalt context switch event
On 14.12.21 08:46, Chen, Hongzhan wrote: > > > BTW, is that blue hollow box exclusive for our plugin? Or can this be > customized? Is it explained somewhere in Kernelshark what hollow boxes > mean? > Some visualization issue: pthelper-912 [000] 753.948271: cobalt_schedule: status=0x1000 pthelper-912 [000] 753.948275: cobalt_switch_context: prev_name=pthelper prev_pid=912 prev_prio=3 prev_state=0xa40042 ==> next_name=hi-thread next_pid=913 next_prio=2 hi-thread-913 [000] 753.948285: cobalt_shadow_hardened: state=0x240040 info=0x0 hi-thread-913 [000] 753.948423: cobalt_thread_fault: ip=0xb6e34e20 type=0x8 hi-thread-913 [000] 753.948737: cobalt_shadow_gorelax: reason=syscall In this case, the task plot of hi-thread-919 does no show the blue box between hardened and fault. >>> >>> Could you zoom out (click "-" in the top left button) to check if blue box >>> would be reshowing? >>> Could you paste more trailing log in your log to check if there another >>> cobalt_switch_context. Current logic is create hollow box between two >>> cobalt_switch_context. If there is only cobalt switch_context event >>> matching, it would not create hollow box correctly. >>> >> >> Yeah, the problem is indeed that the bar is only drawn if beg AND end >> points are visible - that is sure not useful. Does the drawing concept >> of kernelshark imposes this limitation? > > This should be common issue in kernelshark. I also observed same > issue on hollow red box for preempt latency implementation. > Ok, another topic to discuss with their community :) Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
RE: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai cobalt context switch event
BTW, is that blue hollow box exclusive for our plugin? Or can this be customized? Is it explained somewhere in Kernelshark what hollow boxes mean? >>> >>> Some visualization issue: >>> >>>pthelper-912 [000] 753.948271: cobalt_schedule: >>> status=0x1000 >>>pthelper-912 [000] 753.948275: cobalt_switch_context: >>> prev_name=pthelper prev_pid=912 prev_prio=3 prev_state=0xa40042 ==> >>> next_name=hi-thread next_pid=913 next_prio=2 >>> hi-thread-913 [000] 753.948285: cobalt_shadow_hardened: >>> state=0x240040 info=0x0 >>> hi-thread-913 [000] 753.948423: cobalt_thread_fault: >>> ip=0xb6e34e20 type=0x8 >>> hi-thread-913 [000] 753.948737: cobalt_shadow_gorelax: >>> reason=syscall >>> >>> In this case, the task plot of hi-thread-919 does no show the blue box >>> between hardened and fault. >> >> Could you zoom out (click "-" in the top left button) to check if blue box >> would be reshowing? >> Could you paste more trailing log in your log to check if there another >> cobalt_switch_context. Current logic is create hollow box between two >> cobalt_switch_context. If there is only cobalt switch_context event >> matching, it would not create hollow box correctly. >> > >Yeah, the problem is indeed that the bar is only drawn if beg AND end >points are visible - that is sure not useful. Does the drawing concept >of kernelshark imposes this limitation? This should be common issue in kernelshark. I also observed same issue on hollow red box for preempt latency implementation. Regards Hongzhan Chen > >Jan > >-- >Siemens AG, T RDA IOT >Corporate Competence Center Embedded Linux
Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai cobalt context switch event
On 14.12.21 08:14, Chen, Hongzhan wrote: > > >> -Original Message- >> From: Jan Kiszka >> Sent: Tuesday, December 14, 2021 3:02 PM >> To: Chen, Hongzhan ; xenomai@xenomai.org >> Subject: Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai >> cobalt context switch event >> >> On 14.12.21 07:49, Jan Kiszka via Xenomai wrote: >>> On 14.12.21 07:44, Chen, Hongzhan wrote: >>>>> >>>>> >>>>> -Original Message- >>>>> From: Jan Kiszka >>>>> Sent: Tuesday, December 14, 2021 2:38 PM >>>>> To: Chen, Hongzhan ; xenomai@xenomai.org >>>>> Subject: Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling >>>>> xenomai cobalt context switch event >>>>> >>>>> On 14.12.21 04:15, Hongzhan Chen wrote: >>>>>> For Xenomai-cobalt enabled system, cobalt_switch_context means that there >>>>>> is context switch in companion core(realtime core), which we may need >>>>>> to do special treatment and take correct action as main kernel >>>>>> sched_switch. >>>>>> We need to update cpu bar regarding cobalt_switch_context event to >>>>>> correct >>>>>> color as switching-in task. >>>>> >>>>> It's tricker... I have trace with the following sequence here: >>>>> >>>>> gdb-898 [000] 753.944137: cobalt_switch_context: >>>>> prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x218008 ==> >>>>> next_name=pthelper next_pid=912 next_prio=3 >>>>> gdb-898 [000] 753.944142: cobalt_timer_start: >>>>> timer=0xef7e8200([watchdog]) value=40 interval=0 mode=0x0 >>>>>pthelper-912 [000] 753.944172: cobalt_shadow_hardened: >>>>> state=0x240840 info=0x0 >>>>> >>>>> Your plugin correctly switches colors on cobalt_switch_context, but that >>>>> cobalt_timer_start is still running under the previous pid, causing a >>>>> color switch-back and than forth on cobalt_shadow_hardened. We need to >>>>> understand why and possibly fix that or rethink the visualization. >>>> >>>> You need to click menu Plots->tasks and then select RT tasks and then >>>> apply. It would tigger plugin shape to redraw >>>> and then the issue you observed would be fixed by plugin. >>> >>> OK, now I also see that blue "OOB" box under the RT task. >>> >>> But the that does not affect the CPU graph, it continues to jump back >>> and forth. And it has no RT mode visualization. >> >> Correction: The CPU graph is fine now as well. Still confusing that you >> have to enable task plots first, but that might be a generic KernelShark >> issue. > > Yes, this is kernelshark issue. Kernel shark have to depend on redraw of > plugin after activation > to correct them. > >> >>> >>> BTW, is that blue hollow box exclusive for our plugin? Or can this be >>> customized? Is it explained somewhere in Kernelshark what hollow boxes mean? >>> >> >> Some visualization issue: >> >>pthelper-912 [000] 753.948271: cobalt_schedule: >> status=0x1000 >>pthelper-912 [000] 753.948275: cobalt_switch_context: >> prev_name=pthelper prev_pid=912 prev_prio=3 prev_state=0xa40042 ==> >> next_name=hi-thread next_pid=913 next_prio=2 >> hi-thread-913 [000] 753.948285: cobalt_shadow_hardened: >> state=0x240040 info=0x0 >> hi-thread-913 [000] 753.948423: cobalt_thread_fault: >> ip=0xb6e34e20 type=0x8 >> hi-thread-913 [000] 753.948737: cobalt_shadow_gorelax: >> reason=syscall >> >> In this case, the task plot of hi-thread-919 does no show the blue box >> between hardened and fault. > > Could you zoom out (click "-" in the top left button) to check if blue box > would be reshowing? > Could you paste more trailing log in your log to check if there another > cobalt_switch_context. Current logic is create hollow box between two > cobalt_switch_context. If there is only cobalt switch_context event matching, > it would not create hollow box correctly. > Yeah, the problem is indeed that the bar is only drawn if beg AND end points are visible - that is sure not useful. Does the drawing concept of kernelshark imposes this limitation? Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
RE: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai cobalt context switch event
>-Original Message- >From: Jan Kiszka >Sent: Tuesday, December 14, 2021 3:02 PM >To: Chen, Hongzhan ; xenomai@xenomai.org >Subject: Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai >cobalt context switch event > >On 14.12.21 07:49, Jan Kiszka via Xenomai wrote: >> On 14.12.21 07:44, Chen, Hongzhan wrote: >>>> >>>> >>>> -Original Message- >>>> From: Jan Kiszka >>>> Sent: Tuesday, December 14, 2021 2:38 PM >>>> To: Chen, Hongzhan ; xenomai@xenomai.org >>>> Subject: Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai >>>> cobalt context switch event >>>> >>>> On 14.12.21 04:15, Hongzhan Chen wrote: >>>>> For Xenomai-cobalt enabled system, cobalt_switch_context means that there >>>>> is context switch in companion core(realtime core), which we may need >>>>> to do special treatment and take correct action as main kernel >>>>> sched_switch. >>>>> We need to update cpu bar regarding cobalt_switch_context event to correct >>>>> color as switching-in task. >>>> >>>> It's tricker... I have trace with the following sequence here: >>>> >>>> gdb-898 [000] 753.944137: cobalt_switch_context: >>>> prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x218008 ==> >>>> next_name=pthelper next_pid=912 next_prio=3 >>>> gdb-898 [000] 753.944142: cobalt_timer_start: >>>> timer=0xef7e8200([watchdog]) value=40 interval=0 mode=0x0 >>>>pthelper-912 [000] 753.944172: cobalt_shadow_hardened: >>>> state=0x240840 info=0x0 >>>> >>>> Your plugin correctly switches colors on cobalt_switch_context, but that >>>> cobalt_timer_start is still running under the previous pid, causing a >>>> color switch-back and than forth on cobalt_shadow_hardened. We need to >>>> understand why and possibly fix that or rethink the visualization. >>> >>> You need to click menu Plots->tasks and then select RT tasks and then >>> apply. It would tigger plugin shape to redraw >>> and then the issue you observed would be fixed by plugin. >> >> OK, now I also see that blue "OOB" box under the RT task. >> >> But the that does not affect the CPU graph, it continues to jump back >> and forth. And it has no RT mode visualization. > >Correction: The CPU graph is fine now as well. Still confusing that you >have to enable task plots first, but that might be a generic KernelShark >issue. Yes, this is kernelshark issue. Kernel shark have to depend on redraw of plugin after activation to correct them. > >> >> BTW, is that blue hollow box exclusive for our plugin? Or can this be >> customized? Is it explained somewhere in Kernelshark what hollow boxes mean? >> > >Some visualization issue: > >pthelper-912 [000] 753.948271: cobalt_schedule: > status=0x1000 >pthelper-912 [000] 753.948275: cobalt_switch_context: > prev_name=pthelper prev_pid=912 prev_prio=3 prev_state=0xa40042 ==> > next_name=hi-thread next_pid=913 next_prio=2 > hi-thread-913 [000] 753.948285: cobalt_shadow_hardened: > state=0x240040 info=0x0 > hi-thread-913 [000] 753.948423: cobalt_thread_fault: ip=0xb6e34e20 > type=0x8 > hi-thread-913 [000] 753.948737: cobalt_shadow_gorelax: > reason=syscall > >In this case, the task plot of hi-thread-919 does no show the blue box >between hardened and fault. Could you zoom out (click "-" in the top left button) to check if blue box would be reshowing? Could you paste more trailing log in your log to check if there another cobalt_switch_context. Current logic is create hollow box between two cobalt_switch_context. If there is only cobalt switch_context event matching, it would not create hollow box correctly. Regards Hongzhan Chen > >Jan > >-- >Siemens AG, T RDA IOT >Corporate Competence Center Embedded Linux
Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai cobalt context switch event
On 14.12.21 07:49, Jan Kiszka via Xenomai wrote: > On 14.12.21 07:44, Chen, Hongzhan wrote: >>> >>> >>> -Original Message- >>> From: Jan Kiszka >>> Sent: Tuesday, December 14, 2021 2:38 PM >>> To: Chen, Hongzhan ; xenomai@xenomai.org >>> Subject: Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai >>> cobalt context switch event >>> >>> On 14.12.21 04:15, Hongzhan Chen wrote: >>>> For Xenomai-cobalt enabled system, cobalt_switch_context means that there >>>> is context switch in companion core(realtime core), which we may need >>>> to do special treatment and take correct action as main kernel >>>> sched_switch. >>>> We need to update cpu bar regarding cobalt_switch_context event to correct >>>> color as switching-in task. >>> >>> It's tricker... I have trace with the following sequence here: >>> >>> gdb-898 [000] 753.944137: cobalt_switch_context: >>> prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x218008 ==> >>> next_name=pthelper next_pid=912 next_prio=3 >>> gdb-898 [000] 753.944142: cobalt_timer_start: >>> timer=0xef7e8200([watchdog]) value=40 interval=0 mode=0x0 >>>pthelper-912 [000] 753.944172: cobalt_shadow_hardened: >>> state=0x240840 info=0x0 >>> >>> Your plugin correctly switches colors on cobalt_switch_context, but that >>> cobalt_timer_start is still running under the previous pid, causing a >>> color switch-back and than forth on cobalt_shadow_hardened. We need to >>> understand why and possibly fix that or rethink the visualization. >> >> You need to click menu Plots->tasks and then select RT tasks and then apply. >> It would tigger plugin shape to redraw >> and then the issue you observed would be fixed by plugin. > > OK, now I also see that blue "OOB" box under the RT task. > > But the that does not affect the CPU graph, it continues to jump back > and forth. And it has no RT mode visualization. Correction: The CPU graph is fine now as well. Still confusing that you have to enable task plots first, but that might be a generic KernelShark issue. > > BTW, is that blue hollow box exclusive for our plugin? Or can this be > customized? Is it explained somewhere in Kernelshark what hollow boxes mean? > Some visualization issue: pthelper-912 [000] 753.948271: cobalt_schedule: status=0x1000 pthelper-912 [000] 753.948275: cobalt_switch_context: prev_name=pthelper prev_pid=912 prev_prio=3 prev_state=0xa40042 ==> next_name=hi-thread next_pid=913 next_prio=2 hi-thread-913 [000] 753.948285: cobalt_shadow_hardened: state=0x240040 info=0x0 hi-thread-913 [000] 753.948423: cobalt_thread_fault: ip=0xb6e34e20 type=0x8 hi-thread-913 [000] 753.948737: cobalt_shadow_gorelax: reason=syscall In this case, the task plot of hi-thread-919 does no show the blue box between hardened and fault. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai cobalt context switch event
On 14.12.21 07:44, Chen, Hongzhan wrote: >> >> >> -Original Message- >> From: Jan Kiszka >> Sent: Tuesday, December 14, 2021 2:38 PM >> To: Chen, Hongzhan ; xenomai@xenomai.org >> Subject: Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai >> cobalt context switch event >> >> On 14.12.21 04:15, Hongzhan Chen wrote: >>> For Xenomai-cobalt enabled system, cobalt_switch_context means that there >>> is context switch in companion core(realtime core), which we may need >>> to do special treatment and take correct action as main kernel sched_switch. >>> We need to update cpu bar regarding cobalt_switch_context event to correct >>> color as switching-in task. >> >> It's tricker... I have trace with the following sequence here: >> >> gdb-898 [000] 753.944137: cobalt_switch_context: >> prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x218008 ==> >> next_name=pthelper next_pid=912 next_prio=3 >> gdb-898 [000] 753.944142: cobalt_timer_start: >> timer=0xef7e8200([watchdog]) value=40 interval=0 mode=0x0 >>pthelper-912 [000] 753.944172: cobalt_shadow_hardened: >> state=0x240840 info=0x0 >> >> Your plugin correctly switches colors on cobalt_switch_context, but that >> cobalt_timer_start is still running under the previous pid, causing a >> color switch-back and than forth on cobalt_shadow_hardened. We need to >> understand why and possibly fix that or rethink the visualization. > > You need to click menu Plots->tasks and then select RT tasks and then apply. > It would tigger plugin shape to redraw > and then the issue you observed would be fixed by plugin. OK, now I also see that blue "OOB" box under the RT task. But the that does not affect the CPU graph, it continues to jump back and forth. And it has no RT mode visualization. BTW, is that blue hollow box exclusive for our plugin? Or can this be customized? Is it explained somewhere in Kernelshark what hollow boxes mean? Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
RE: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai cobalt context switch event
> > >-Original Message- >From: Jan Kiszka >Sent: Tuesday, December 14, 2021 2:38 PM >To: Chen, Hongzhan ; xenomai@xenomai.org >Subject: Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai >cobalt context switch event > >On 14.12.21 04:15, Hongzhan Chen wrote: >> For Xenomai-cobalt enabled system, cobalt_switch_context means that there >> is context switch in companion core(realtime core), which we may need >> to do special treatment and take correct action as main kernel sched_switch. >> We need to update cpu bar regarding cobalt_switch_context event to correct >> color as switching-in task. > >It's tricker... I have trace with the following sequence here: > > gdb-898 [000] 753.944137: cobalt_switch_context: > prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x218008 ==> > next_name=pthelper next_pid=912 next_prio=3 > gdb-898 [000] 753.944142: cobalt_timer_start: > timer=0xef7e8200([watchdog]) value=40 interval=0 mode=0x0 >pthelper-912 [000] 753.944172: cobalt_shadow_hardened: > state=0x240840 info=0x0 > >Your plugin correctly switches colors on cobalt_switch_context, but that >cobalt_timer_start is still running under the previous pid, causing a >color switch-back and than forth on cobalt_shadow_hardened. We need to >understand why and possibly fix that or rethink the visualization. You need to click menu Plots->tasks and then select RT tasks and then apply. It would tigger plugin shape to redraw and then the issue you observed would be fixed by plugin. > >Jan > >-- >Siemens AG, T RDA IOT >Corporate Competence Center Embedded Linux
Re: [RFC PATCH 1/2] kernel-shark: add plugin for handling xenomai cobalt context switch event
On 14.12.21 04:15, Hongzhan Chen wrote: > For Xenomai-cobalt enabled system, cobalt_switch_context means that there > is context switch in companion core(realtime core), which we may need > to do special treatment and take correct action as main kernel sched_switch. > We need to update cpu bar regarding cobalt_switch_context event to correct > color as switching-in task. It's tricker... I have trace with the following sequence here: gdb-898 [000] 753.944137: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x218008 ==> next_name=pthelper next_pid=912 next_prio=3 gdb-898 [000] 753.944142: cobalt_timer_start: timer=0xef7e8200([watchdog]) value=40 interval=0 mode=0x0 pthelper-912 [000] 753.944172: cobalt_shadow_hardened: state=0x240840 info=0x0 Your plugin correctly switches colors on cobalt_switch_context, but that cobalt_timer_start is still running under the previous pid, causing a color switch-back and than forth on cobalt_shadow_hardened. We need to understand why and possibly fix that or rethink the visualization. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
