Re: [Adeos-main] Kernel blocked during send/receive raw ethernet packets.
On Tue, Jul 5, 2011 at 10:45 AM, Ronny Meeus wrote: > On Mon, Jul 4, 2011 at 10:13 PM, Philippe Gerum wrote: >> On Mon, 2011-07-04 at 22:09 +0200, Philippe Gerum wrote: >>> On Mon, 2011-07-04 at 22:04 +0200, Ronny Meeus wrote: >>> > On Mon, Jul 4, 2011 at 1:42 PM, Gilles Chanteperdrix >>> > wrote: >>> > > On 07/04/2011 10:20 AM, Philippe Gerum wrote: >>> > >> Could you try 2.6.36-ipipe as well in case 2.6.35.7 without pipeline >>> > >> does not exhibit the issue? A number of changes went in the IRQ replay >>> > >> code during this time frame, and 2.6.35 was in a state of flux >>> > >> regarding >>> > >> this. >>> > > >>> > > And please try Xenomai 2.5.6. >>> > > >>> > > -- >>> > > Gilles. >>> > > >>> > >>> > Hello >>> > >>> > today we tested tested: "Running the same test on 2.6.35.7 without IPIPE". >>> > The result is also not OK. The problem can be reproduced on this >>> > kernel, so it looks like the issue has nothing to do with the I-PIPE. >>> > We are currently porting the FreeScale patches to the 2.6.36 kernel >>> > but it looks like this is going to take some effort (Thomas already >>> > spent half a day on it today). >>> > >>> > Maybe a bit of history: we started with the 2.6.34 kernel with no SMP. >>> > Here we observed issue in some scenarios. By playing with the >>> > configuration we found that these issues were resolved by switching to >>> > SMP (even on one core). >>> > After the activation of SMP we started to see BADNESS issues once we >>> > started to run our Xenomai based applications. >>> > Thomas found a patch on the Xenomai mailing list to solve the badness >>> > issue but this was based on 2.6.35.7. We ported the FreeScale patches >>> > to that releases and we observed the blocking application issue >>> > described above. >>> > >>> > We will try to port the FreeScale patches to 2.6.36 and see how the >>> > system behaves. More information on this later. >>> > Can you guys give us any hint of what would be the most stable version >>> > of Linux + Xenomai to base our application on? >>> >>> SMP-wise for powerpc, 2.6.36 + upcoming 2.5.6 >> >> s,upcoming,, >> >> I mean 2.5.6 stock. >> >>> >>> > >>> > Thanks >>> > Ronny >>> > >>> > ___ >>> > Adeos-main mailing list >>> > [email protected] >>> > https://mail.gna.org/listinfo/adeos-main >>> >> >> -- >> Philippe. >> >> >> > > Hello > > we have done the test on both 2.6.36 and 2.6.36.4 without I-PIPE. > In both cases the issue is seen. > > Any hints to helps us debugging this issue would be appreciated. > > Best regards > Ronny > Hello Today we have identified the issue. In the net/packet/af_packet.c file a spin_lock is used where a spin_lock_bh should be used. We observed that the spin_lock was entered recursively: by the application and also from the context of a softIRQ. Result was a deadlock. I will post a patch soon on the netdev mailing list. We will upgrade most probably to 2.6.36.4 and Xenomai 2.5.6 like you guys suggested. Thanks for your support. Best regards, Ronny ___ Adeos-main mailing list [email protected] https://mail.gna.org/listinfo/adeos-main
Re: [Adeos-main] Kernel blocked during send/receive raw ethernet packets.
On Mon, Jul 4, 2011 at 10:13 PM, Philippe Gerum wrote: > On Mon, 2011-07-04 at 22:09 +0200, Philippe Gerum wrote: >> On Mon, 2011-07-04 at 22:04 +0200, Ronny Meeus wrote: >> > On Mon, Jul 4, 2011 at 1:42 PM, Gilles Chanteperdrix >> > wrote: >> > > On 07/04/2011 10:20 AM, Philippe Gerum wrote: >> > >> Could you try 2.6.36-ipipe as well in case 2.6.35.7 without pipeline >> > >> does not exhibit the issue? A number of changes went in the IRQ replay >> > >> code during this time frame, and 2.6.35 was in a state of flux regarding >> > >> this. >> > > >> > > And please try Xenomai 2.5.6. >> > > >> > > -- >> > > Gilles. >> > > >> > >> > Hello >> > >> > today we tested tested: "Running the same test on 2.6.35.7 without IPIPE". >> > The result is also not OK. The problem can be reproduced on this >> > kernel, so it looks like the issue has nothing to do with the I-PIPE. >> > We are currently porting the FreeScale patches to the 2.6.36 kernel >> > but it looks like this is going to take some effort (Thomas already >> > spent half a day on it today). >> > >> > Maybe a bit of history: we started with the 2.6.34 kernel with no SMP. >> > Here we observed issue in some scenarios. By playing with the >> > configuration we found that these issues were resolved by switching to >> > SMP (even on one core). >> > After the activation of SMP we started to see BADNESS issues once we >> > started to run our Xenomai based applications. >> > Thomas found a patch on the Xenomai mailing list to solve the badness >> > issue but this was based on 2.6.35.7. We ported the FreeScale patches >> > to that releases and we observed the blocking application issue >> > described above. >> > >> > We will try to port the FreeScale patches to 2.6.36 and see how the >> > system behaves. More information on this later. >> > Can you guys give us any hint of what would be the most stable version >> > of Linux + Xenomai to base our application on? >> >> SMP-wise for powerpc, 2.6.36 + upcoming 2.5.6 > > s,upcoming,, > > I mean 2.5.6 stock. > >> >> > >> > Thanks >> > Ronny >> > >> > ___ >> > Adeos-main mailing list >> > [email protected] >> > https://mail.gna.org/listinfo/adeos-main >> > > -- > Philippe. > > > Hello we have done the test on both 2.6.36 and 2.6.36.4 without I-PIPE. In both cases the issue is seen. Any hints to helps us debugging this issue would be appreciated. Best regards Ronny ___ Adeos-main mailing list [email protected] https://mail.gna.org/listinfo/adeos-main
Re: [Adeos-main] Kernel blocked during send/receive raw ethernet packets.
On Mon, 2011-07-04 at 22:09 +0200, Philippe Gerum wrote: > On Mon, 2011-07-04 at 22:04 +0200, Ronny Meeus wrote: > > On Mon, Jul 4, 2011 at 1:42 PM, Gilles Chanteperdrix > > wrote: > > > On 07/04/2011 10:20 AM, Philippe Gerum wrote: > > >> Could you try 2.6.36-ipipe as well in case 2.6.35.7 without pipeline > > >> does not exhibit the issue? A number of changes went in the IRQ replay > > >> code during this time frame, and 2.6.35 was in a state of flux regarding > > >> this. > > > > > > And please try Xenomai 2.5.6. > > > > > > -- > > >Gilles. > > > > > > > Hello > > > > today we tested tested: "Running the same test on 2.6.35.7 without IPIPE". > > The result is also not OK. The problem can be reproduced on this > > kernel, so it looks like the issue has nothing to do with the I-PIPE. > > We are currently porting the FreeScale patches to the 2.6.36 kernel > > but it looks like this is going to take some effort (Thomas already > > spent half a day on it today). > > > > Maybe a bit of history: we started with the 2.6.34 kernel with no SMP. > > Here we observed issue in some scenarios. By playing with the > > configuration we found that these issues were resolved by switching to > > SMP (even on one core). > > After the activation of SMP we started to see BADNESS issues once we > > started to run our Xenomai based applications. > > Thomas found a patch on the Xenomai mailing list to solve the badness > > issue but this was based on 2.6.35.7. We ported the FreeScale patches > > to that releases and we observed the blocking application issue > > described above. > > > > We will try to port the FreeScale patches to 2.6.36 and see how the > > system behaves. More information on this later. > > Can you guys give us any hint of what would be the most stable version > > of Linux + Xenomai to base our application on? > > SMP-wise for powerpc, 2.6.36 + upcoming 2.5.6 s,upcoming,, I mean 2.5.6 stock. > > > > > Thanks > > Ronny > > > > ___ > > Adeos-main mailing list > > [email protected] > > https://mail.gna.org/listinfo/adeos-main > -- Philippe. ___ Adeos-main mailing list [email protected] https://mail.gna.org/listinfo/adeos-main
Re: [Adeos-main] Kernel blocked during send/receive raw ethernet packets.
On Mon, 2011-07-04 at 22:04 +0200, Ronny Meeus wrote: > On Mon, Jul 4, 2011 at 1:42 PM, Gilles Chanteperdrix > wrote: > > On 07/04/2011 10:20 AM, Philippe Gerum wrote: > >> Could you try 2.6.36-ipipe as well in case 2.6.35.7 without pipeline > >> does not exhibit the issue? A number of changes went in the IRQ replay > >> code during this time frame, and 2.6.35 was in a state of flux regarding > >> this. > > > > And please try Xenomai 2.5.6. > > > > -- > >Gilles. > > > > Hello > > today we tested tested: "Running the same test on 2.6.35.7 without IPIPE". > The result is also not OK. The problem can be reproduced on this > kernel, so it looks like the issue has nothing to do with the I-PIPE. > We are currently porting the FreeScale patches to the 2.6.36 kernel > but it looks like this is going to take some effort (Thomas already > spent half a day on it today). > > Maybe a bit of history: we started with the 2.6.34 kernel with no SMP. > Here we observed issue in some scenarios. By playing with the > configuration we found that these issues were resolved by switching to > SMP (even on one core). > After the activation of SMP we started to see BADNESS issues once we > started to run our Xenomai based applications. > Thomas found a patch on the Xenomai mailing list to solve the badness > issue but this was based on 2.6.35.7. We ported the FreeScale patches > to that releases and we observed the blocking application issue > described above. > > We will try to port the FreeScale patches to 2.6.36 and see how the > system behaves. More information on this later. > Can you guys give us any hint of what would be the most stable version > of Linux + Xenomai to base our application on? SMP-wise for powerpc, 2.6.36 + upcoming 2.5.6 > > Thanks > Ronny > > ___ > Adeos-main mailing list > [email protected] > https://mail.gna.org/listinfo/adeos-main -- Philippe. ___ Adeos-main mailing list [email protected] https://mail.gna.org/listinfo/adeos-main
Re: [Adeos-main] Kernel blocked during send/receive raw ethernet packets.
On Mon, Jul 4, 2011 at 1:42 PM, Gilles Chanteperdrix wrote: > On 07/04/2011 10:20 AM, Philippe Gerum wrote: >> Could you try 2.6.36-ipipe as well in case 2.6.35.7 without pipeline >> does not exhibit the issue? A number of changes went in the IRQ replay >> code during this time frame, and 2.6.35 was in a state of flux regarding >> this. > > And please try Xenomai 2.5.6. > > -- > Gilles. > Hello today we tested tested: "Running the same test on 2.6.35.7 without IPIPE". The result is also not OK. The problem can be reproduced on this kernel, so it looks like the issue has nothing to do with the I-PIPE. We are currently porting the FreeScale patches to the 2.6.36 kernel but it looks like this is going to take some effort (Thomas already spent half a day on it today). Maybe a bit of history: we started with the 2.6.34 kernel with no SMP. Here we observed issue in some scenarios. By playing with the configuration we found that these issues were resolved by switching to SMP (even on one core). After the activation of SMP we started to see BADNESS issues once we started to run our Xenomai based applications. Thomas found a patch on the Xenomai mailing list to solve the badness issue but this was based on 2.6.35.7. We ported the FreeScale patches to that releases and we observed the blocking application issue described above. We will try to port the FreeScale patches to 2.6.36 and see how the system behaves. More information on this later. Can you guys give us any hint of what would be the most stable version of Linux + Xenomai to base our application on? Thanks Ronny ___ Adeos-main mailing list [email protected] https://mail.gna.org/listinfo/adeos-main
Re: [Adeos-main] Kernel blocked during send/receive raw ethernet packets.
On 07/04/2011 10:20 AM, Philippe Gerum wrote: > Could you try 2.6.36-ipipe as well in case 2.6.35.7 without pipeline > does not exhibit the issue? A number of changes went in the IRQ replay > code during this time frame, and 2.6.35 was in a state of flux regarding > this. And please try Xenomai 2.5.6. -- Gilles. ___ Adeos-main mailing list [email protected] https://mail.gna.org/listinfo/adeos-main
Re: [Adeos-main] Kernel blocked during send/receive raw ethernet packets.
On Mon, 2011-07-04 at 10:06 +0200, Ronny Meeus wrote: > On Sat, Jul 2, 2011 at 11:33 PM, Ronny Meeus wrote: > > Hello > > > > we use have a FreeScale P4040 (powerpc) based board running Linux+Xenomai. > > I copy-paste here some information I found in the bootlog: > > > > [0.00] Using P4080 DS machine description > > [0.00] Memory CAM mapping: 256/256/256 Mb, residual: 1248Mb > > [0.00] Linux version 2.6.35.7-hg98224f47aa52-dirty > > (x@devws108) (gcc version 4.4.6 (Buildroot 2011.05-hg98224f47aa52) > > ) #1 SMP Fri Jul 1 08:42:30 CEST 2011 > > > > [0.00] clocksource: timebase mult[6aaaf09] shift[22] registered > > [0.00] I-pipe 2.12-01: pipeline enabled. > > [0.00] Console: colour dummy device 80x25 > > [0.181150] pid_max: default: 32768 minimum: 301 > > > > [2.093842] I-pipe: Domain Xenomai registered. > > [2.146016] Xenomai: hal/powerpc started. > > [2.193904] Xenomai: scheduling class idle registered. > > [2.255328] Xenomai: scheduling class rt registered. > > [2.319092] Xenomai: real-time nucleus v2.5.5 (Ghosts) loaded. > > [2.388207] Xenomai: starting native API services. > > [2.445249] Xenomai: starting pSOS+ services. > > [2.497478] highmem bounce pool size: 64 pages > > [2.550932] fuse init (API version 7.14) > > > > Although the P4040 has 4 cores, we are currently using only 1 core. > > This is specified in the device tree we are using. > > The kernel runs SMP enabled. > > > > I start 2 test applications on this board. > > The first application is sending raw Ethernet packets on a link that > > is put in loop. The result is that all packets we send are received > > (unmodified) back on the same interface. > > The second application is listening on the same Ethernet interface > > also via a raw Ethernet socket. > > Both application are plain Linux application so no Xenomai code is used. > > > > One side effect of using raw Ethernet sockets is that all packets sent > > on one socket will also be received by all other raw Ethernet sockets. > > This means that the listening application will receive each packet 2 > > times: once while sending and a second time when it is received via > > the loop. (A side question: can the behavior be disabled somehow? We > > basically do not want to receive all packets we send ...) > > > > After a very short time (sending something like 3 packets), both > > applications block completely and 60 seconds later an indication is > > displayed on the console that the kernel is locked. > > > > [ 805.307213] BUG: soft lockup - CPU#0 stuck for 61s! > > [send_eth_socket:1907] > > [ 805.389519] Modules linked in: reboot_helper dpll_si53xx crave > > ndps_a_cpld > > [ 805.471880] NIP: c000cc4c LR: CTR: > > [ 805.531274] REGS: c1f87040 TRAP: Not tainted > > (2.6.35.7-hg98224f47aa52-dirty) > > [ 805.623992] MSR: 00029002 CR: XER: > > [ 805.696972] TASK = ec7116d0[1907] 'send_eth_socket' THREAD: ec6aa000 > > CPU: 0 > > [ 805.778248] GPR00: > > > > [ 805.878359] GPR08: > > > > [ 805.978452] GPR16: > > > > [ 806.078571] GPR24: > > > > [ 806.180773] NIP [c000cc4c] udelay+0x24/0x30 > > [ 806.230782] LR [] (null) > > [ 806.269334] Call Trace: > > [ 806.298521] [efff3b50] [c00071b4] show_stack+0x78/0x18c (unreliable) > > [ 806.374600] [efff3b90] [c00078c4] show_regs+0x200/0x2ec > > [ 806.437125] [efff3bc0] [c00658d4] softlockup_tick+0x1dc/0x23c > > [ 806.505897] [efff3bf0] [c003cc50] run_local_timers+0x1c/0x2c > > [ 806.573626] [efff3c00] [c003cca4] update_process_times+0x44/0x80 > > [ 806.645528] [efff3c20] [c0059bc4] tick_sched_timer+0xd0/0x128 > > [ 806.714307] [efff3c50] [c004d8f0] __run_hrtimer+0x68/0x14c > > [ 806.779958] [efff3c70] [c004efa4] hrtimer_interrupt+0x1d8/0x41c > > [ 806.850812] [efff3cf0] [c000d8d8] timer_interrupt+0x1b4/0x238 > > [ 806.919586] [efff3d10] [c0009ac4] __ipipe_do_timer+0x44/0x54 > > [ 806.987315] [efff3d20] [c006d448] __ipipe_sync_stage+0x1d0/0x27c > > [ 807.059212] [efff3d60] [c0009728] __ipipe_grab_timer+0x104/0x12c > > [ 807.131112] [efff3d70] [c00129e0] __ipipe_ret_from_except+0x0/0xc > > [ 807.204063] --- Exception: 901 at _raw_spin_lock+0x30/0x3c > > [ 807.204068] LR = tpacket_rcv+0x264/0x570 > > [ 807.320754] [efff3e30] [c0325e48] tpacket_rcv+0xf4/0x570 (unreliable) > > [ 807.397875] [efff3e80] [c02c43b0] __netif_receive_skb+0x2b4/0x2f0 > > [ 807.470811] [efff3eb0] [c02c4fa0] netif_receive_skb+0x98/0xac > > [ 807.539583] [efff3ee0] [c0292838] ingress_rx_default_dqrr+0x428/0x4b4 > > [ 807.616693] [efff3f10] [c02ac2a8] qman_poll_dqrr+0x1e0/0x284 > > [ 807.684426] [efff3f50] [c02940
Re: [Adeos-main] Kernel blocked during send/receive raw ethernet packets.
On Sat, Jul 2, 2011 at 11:33 PM, Ronny Meeus wrote: > Hello > > we use have a FreeScale P4040 (powerpc) based board running Linux+Xenomai. > I copy-paste here some information I found in the bootlog: > > [ 0.00] Using P4080 DS machine description > [ 0.00] Memory CAM mapping: 256/256/256 Mb, residual: 1248Mb > [ 0.00] Linux version 2.6.35.7-hg98224f47aa52-dirty > (x@devws108) (gcc version 4.4.6 (Buildroot 2011.05-hg98224f47aa52) > ) #1 SMP Fri Jul 1 08:42:30 CEST 2011 > > [ 0.00] clocksource: timebase mult[6aaaf09] shift[22] registered > [ 0.00] I-pipe 2.12-01: pipeline enabled. > [ 0.00] Console: colour dummy device 80x25 > [ 0.181150] pid_max: default: 32768 minimum: 301 > > [ 2.093842] I-pipe: Domain Xenomai registered. > [ 2.146016] Xenomai: hal/powerpc started. > [ 2.193904] Xenomai: scheduling class idle registered. > [ 2.255328] Xenomai: scheduling class rt registered. > [ 2.319092] Xenomai: real-time nucleus v2.5.5 (Ghosts) loaded. > [ 2.388207] Xenomai: starting native API services. > [ 2.445249] Xenomai: starting pSOS+ services. > [ 2.497478] highmem bounce pool size: 64 pages > [ 2.550932] fuse init (API version 7.14) > > Although the P4040 has 4 cores, we are currently using only 1 core. > This is specified in the device tree we are using. > The kernel runs SMP enabled. > > I start 2 test applications on this board. > The first application is sending raw Ethernet packets on a link that > is put in loop. The result is that all packets we send are received > (unmodified) back on the same interface. > The second application is listening on the same Ethernet interface > also via a raw Ethernet socket. > Both application are plain Linux application so no Xenomai code is used. > > One side effect of using raw Ethernet sockets is that all packets sent > on one socket will also be received by all other raw Ethernet sockets. > This means that the listening application will receive each packet 2 > times: once while sending and a second time when it is received via > the loop. (A side question: can the behavior be disabled somehow? We > basically do not want to receive all packets we send ...) > > After a very short time (sending something like 3 packets), both > applications block completely and 60 seconds later an indication is > displayed on the console that the kernel is locked. > > [ 805.307213] BUG: soft lockup - CPU#0 stuck for 61s! [send_eth_socket:1907] > [ 805.389519] Modules linked in: reboot_helper dpll_si53xx crave ndps_a_cpld > [ 805.471880] NIP: c000cc4c LR: CTR: > [ 805.531274] REGS: c1f87040 TRAP: Not tainted > (2.6.35.7-hg98224f47aa52-dirty) > [ 805.623992] MSR: 00029002 CR: XER: > [ 805.696972] TASK = ec7116d0[1907] 'send_eth_socket' THREAD: ec6aa000 CPU: 0 > [ 805.778248] GPR00: > > [ 805.878359] GPR08: > > [ 805.978452] GPR16: > > [ 806.078571] GPR24: > > [ 806.180773] NIP [c000cc4c] udelay+0x24/0x30 > [ 806.230782] LR [] (null) > [ 806.269334] Call Trace: > [ 806.298521] [efff3b50] [c00071b4] show_stack+0x78/0x18c (unreliable) > [ 806.374600] [efff3b90] [c00078c4] show_regs+0x200/0x2ec > [ 806.437125] [efff3bc0] [c00658d4] softlockup_tick+0x1dc/0x23c > [ 806.505897] [efff3bf0] [c003cc50] run_local_timers+0x1c/0x2c > [ 806.573626] [efff3c00] [c003cca4] update_process_times+0x44/0x80 > [ 806.645528] [efff3c20] [c0059bc4] tick_sched_timer+0xd0/0x128 > [ 806.714307] [efff3c50] [c004d8f0] __run_hrtimer+0x68/0x14c > [ 806.779958] [efff3c70] [c004efa4] hrtimer_interrupt+0x1d8/0x41c > [ 806.850812] [efff3cf0] [c000d8d8] timer_interrupt+0x1b4/0x238 > [ 806.919586] [efff3d10] [c0009ac4] __ipipe_do_timer+0x44/0x54 > [ 806.987315] [efff3d20] [c006d448] __ipipe_sync_stage+0x1d0/0x27c > [ 807.059212] [efff3d60] [c0009728] __ipipe_grab_timer+0x104/0x12c > [ 807.131112] [efff3d70] [c00129e0] __ipipe_ret_from_except+0x0/0xc > [ 807.204063] --- Exception: 901 at _raw_spin_lock+0x30/0x3c > [ 807.204068] LR = tpacket_rcv+0x264/0x570 > [ 807.320754] [efff3e30] [c0325e48] tpacket_rcv+0xf4/0x570 (unreliable) > [ 807.397875] [efff3e80] [c02c43b0] __netif_receive_skb+0x2b4/0x2f0 > [ 807.470811] [efff3eb0] [c02c4fa0] netif_receive_skb+0x98/0xac > [ 807.539583] [efff3ee0] [c0292838] ingress_rx_default_dqrr+0x428/0x4b4 > [ 807.616693] [efff3f10] [c02ac2a8] qman_poll_dqrr+0x1e0/0x284 > [ 807.684426] [efff3f50] [c0294088] dpaa_eth_poll+0x34/0xd0 > [ 807.749031] [efff3f70] [c02c5280] net_rx_action+0xc0/0x1e8 > [ 807.814683] [efff3fa0] [c0035ab0] __do_softirq+0x138/0x210 > [ 807.880333] [efff3ff0] [c00115e8] call_do_softirq+0x14/0x24 > [ 807.947022] [ec6abab0
