Re: [BUG][AX25] spinlock lockup
Applied. Thanks! Regards, Jann -Ursprüngliche Nachricht- Von: Jarek Poplawski [mailto:[EMAIL PROTECTED] Gesendet: Sonntag, 24. Februar 2008 20:51 An: Jann Traschewski Cc: netdev@vger.kernel.org; 'Ralf Baechle' Betreff: Re: [BUG][AX25] spinlock lockup On Sun, Feb 24, 2008 at 04:10:29AM +0100, Jann Traschewski wrote: Hello, Hi! I got a spinlock lockup using the latest Kernel 2.6.24.2 with recent patches from Jarek Poplawski applied. ... ppp_deflate nf_nat zlib_deflateBUG: unable to handle kernel NULL pointer dereference zlib_inflate nf_conntrack_ipv4 bsd_comp slhc ppp_async xt_state ... EIP is at skb_append+0x1b/0x30 ... 0010 BUG: spinlock lockup on CPU#1, bcm/12213, f40846b8 Looks like 2 in 1: NULL pointer dereference and (later?) lockup. There is only one function in AX25 calling skb_append(), and it really looks suspicious: appends skb after previously enqueued one, but in the meantime this previous skb could be removed from the queue. Here is a patch for testing: it fixes this simple way, so this is not fully compatible with the current method, but let's check if this could be a problem? Regards, Jarek P. (testing patch #1) --- net/ax25/ax25_subr.c | 11 +++ 1 files changed, 3 insertions(+), 8 deletions(-) diff --git a/net/ax25/ax25_subr.c b/net/ax25/ax25_subr.c index d8f2157..034aa10 100644 --- a/net/ax25/ax25_subr.c +++ b/net/ax25/ax25_subr.c @@ -64,20 +64,15 @@ void ax25_frames_acked(ax25_cb *ax25, unsigned short nr) void ax25_requeue_frames(ax25_cb *ax25) { - struct sk_buff *skb, *skb_prev = NULL; + struct sk_buff *skb; /* * Requeue all the un-ack-ed frames on the output queue to be picked * up by ax25_kick called from the timer. This arrangement handles the * possibility of an empty output queue. */ - while ((skb = skb_dequeue(ax25-ack_queue)) != NULL) { - if (skb_prev == NULL) - skb_queue_head(ax25-write_queue, skb); - else - skb_append(skb_prev, skb, ax25-write_queue); - skb_prev = skb; - } + while ((skb = skb_dequeue_tail(ax25-ack_queue)) != NULL) + skb_queue_head(ax25-write_queue, skb); } /* -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[BUG][AX25] spinlock lockup
Hello, I got a spinlock lockup using the latest Kernel 2.6.24.2 with recent patches from Jarek Poplawski applied. ppp_deflate nf_nat zlib_deflateBUG: unable to handle kernel NULL pointer dereference zlib_inflate nf_conntrack_ipv4 bsd_comp slhc ppp_async xt_state tun ppp_genericprinting eip: c02338c2 nf_conntrack bitrev ipt_REJECT*pde = iptable_filter crc32 iptable_mangle mkiss xt_MARK at virtual address 0004 ax25 ipv6Oops: 0002 [#1] SMP ipip crc16 tunnel4Modules linked in: iptable_nat ide_cd cdrom aic7xxx scsi_transport_spi parport_serial parport_pc parport i2c_piix4 netconsole genrtc Pid: 3032, comm: linuxnet Not tainted (2.6.24.2-dg8ngn-p02 #1) EIP: 0060:[c02338c2] EFLAGS: 00010092 CPU: 0 EIP is at skb_append+0x1b/0x30 EAX: 0292 EBX: c9318980 ECX: c0466a80 EDX: ESI: c9e085c0 EDI: f40846ac EBP: f40846b8 ESP: f72cbd04 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Process linuxnet (pid: 3032, ti=f72ca000 task=f7d08030 task.ti=f72ca000) Stack: c9318980 c9e085c0 f4084600 f8a0e2a3 f4084600 0005 0010 BUG: spinlock lockup on CPU#1, bcm/12213, f40846b8 Pid: 12213, comm: bcm Tainted: G D 2.6.24.2-dg8ngn-p02 #1 [c01cb96f] _raw_spin_lock+0xbb/0xdc [c0299fdc] _spin_lock_irqsave+0x39/0x41 [c02339bf] skb_dequeue+0xf/0x3f [c02339bf] skb_dequeue+0xf/0x3f [f8a0c998] ax25_kick+0x14f/0x18c [ax25] [f8a10a26] ax25_sendmsg+0x35f/0x49a [ax25] [c01473d1] file_read_actor+0x7d/0xd5 [c0147c35] do_generic_mapping_read+0x1d5/0x3b2 [c0147354] file_read_actor+0x0/0xd5 [c022e404] sock_aio_write+0xbf/0xcb [c01494ba] generic_file_aio_read+0x161/0x19c [c01627bb] do_sync_write+0xc7/0x10a [c0130ea9] autoremove_wake_function+0x0/0x35 [c0162f48] vfs_write+0x9e/0x10c [c01634b0] sys_write+0x41/0x67 [c0103ea2] syscall_call+0x7/0xb === -- Jann Traschewski, Drosselstr.1, D-90513 Zirndorf, Germany Tel.: +49-911-696971, Mobile: +49-170-1045937, EMail: [EMAIL PROTECTED] Ham: DG8NGN / DB0VOX, http://www.qsl.net/db0fhn, ICQ UIN: 4130182 -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH][AX25] ax25_out: check skb for NULL in ax25_kick()
Applied and stable with Kernel 2.6.24.2 since 12 hours. Regards, Jann -Ursprüngliche Nachricht- Von: Jarek Poplawski [mailto:[EMAIL PROTECTED] Gesendet: Mittwoch, 13. Februar 2008 12:56 An: David Miller Cc: Jann Traschewski; Bernard Pidoux F6BVP; Ralf Baechle; netdev@vger.kernel.org Betreff: [PATCH][AX25] ax25_out: check skb for NULL in ax25_kick() Hi, Here is an official version of testing patch #2 from this thread. The only difference: ax25-vs is changed only after checking skb is not NULL (plus a comment). IMHO it could be applied. Thanks, Jarek P. Subject: [AX25] ax25_out: check skb for NULL in ax25_kick() According to some OOPS reports ax25_kick tries to clone NULL skbs sometimes. It looks like a race with ax25_clear_queues(). Probably there is no need to add more than a simple check for this yet. Another report suggested there are probably also cases where ax25 -paclen == 0 can happen in ax25_output(); this wasn't confirmed during testing but let's leave this debugging check for some time. Reported-and-tested-by: Jann Traschewski [EMAIL PROTECTED] Signed-off-by: Jarek Poplawski [EMAIL PROTECTED] --- diff -Nurp 2.6.24-mm1-/net/ax25/ax25_out.c 2.6.24-mm1+/net/ax25/ax25_out.c --- 2.6.24-mm1-/net/ax25/ax25_out.c 2008-01-24 22:58:37.0 + +++ 2.6.24-mm1+/net/ax25/ax25_out.c 2008-02-13 10:43:50.0 + @@ -117,6 +117,12 @@ void ax25_output(ax25_cb *ax25, int pacl unsigned char *p; int frontlen, len, fragno, ka9qfrag, first = 1; + if (paclen 16) { + WARN_ON_ONCE(1); + kfree_skb(skb); + return; + } + if ((skb-len - 1) paclen) { if (*skb-data == AX25_P_TEXT) { skb_pull(skb, 1); /* skip PID */ @@ -251,8 +257,6 @@ void ax25_kick(ax25_cb *ax25) if (start == end) return; - ax25-vs = start; - /* * Transmit data until either we're out of data to send or * the window is full. Send a poll on the final I frame if @@ -261,8 +265,13 @@ void ax25_kick(ax25_cb *ax25) /* * Dequeue the frame and copy it. + * Check for race with ax25_clear_queues(). */ skb = skb_dequeue(ax25-write_queue); + if (!skb) + return; + + ax25-vs = start; do { if ((skbn = skb_clone(skb, GFP_ATOMIC)) == NULL) { -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH][AX25] ax25_route: make ax25_route_lock BH safe
Applied on 2.6.24.2 and up without any problems/warnings since 12 hours. Thanks, Jann -Ursprüngliche Nachricht- Von: Jarek Poplawski [mailto:[EMAIL PROTECTED] Gesendet: Montag, 11. Februar 2008 13:43 An: David Miller Cc: Jann Traschewski; Bernard Pidoux F6BVP; Ralf Baechle DL5RB; netdev@vger.kernel.org Betreff: [PATCH][AX25] ax25_route: make ax25_route_lock BH safe Subject: [AX25] ax25_route: make ax25_route_lock BH safe = [ INFO: inconsistent lock state ] 2.6.24-dg8ngn-p02 #1 - inconsistent {softirq-on-W} - {in-softirq-R} usage. linuxnet/3046 [HC0[0]:SC1[2]:HE1:SE0] takes: (ax25_route_lock){--.+}, at: [f8a0cfb7] ax25_get_route+0x18/0xb7 [ax25] {softirq-on-W} state was registered at: ... This lockdep report shows that ax25_route_lock is taken for reading in softirq context, and for writing in process context with BHs enabled. So, to make this safe, all write_locks in ax25_route.c are changed to _bh versions. Reported-by: Jann Traschewski [EMAIL PROTECTED], Signed-off-by: Jarek Poplawski [EMAIL PROTECTED] --- diff -Nurp 2.6.24-mm1-/net/ax25/ax25_route.c 2.6.24-mm1+/net/ax25/ax25_route.c --- 2.6.24-mm1-/net/ax25/ax25_route.c 2008-02-05 07:45:38.0 + +++ 2.6.24-mm1+/net/ax25/ax25_route.c 2008-02-11 11:58:47.0 + @@ -45,7 +45,7 @@ void ax25_rt_device_down(struct net_devi { ax25_route *s, *t, *ax25_rt; - write_lock(ax25_route_lock); + write_lock_bh(ax25_route_lock); ax25_rt = ax25_route_list; while (ax25_rt != NULL) { s = ax25_rt; @@ -68,7 +68,7 @@ void ax25_rt_device_down(struct net_devi } } } - write_unlock(ax25_route_lock); + write_unlock_bh(ax25_route_lock); } static int __must_check ax25_rt_add(struct ax25_routes_struct *route) @@ -82,7 +82,7 @@ static int __must_check ax25_rt_add(stru if (route-digi_count AX25_MAX_DIGIS) return -EINVAL; - write_lock(ax25_route_lock); + write_lock_bh(ax25_route_lock); ax25_rt = ax25_route_list; while (ax25_rt != NULL) { @@ -92,7 +92,7 @@ static int __must_check ax25_rt_add(stru ax25_rt-digipeat = NULL; if (route-digi_count != 0) { if ((ax25_rt-digipeat = kmalloc(sizeof(ax25_digi), GFP_ATOMIC)) == NULL) { - write_unlock(ax25_route_lock); + write_unlock_bh(ax25_route_lock); return -ENOMEM; } ax25_rt-digipeat-lastrepeat = -1; @@ -102,14 +102,14 @@ static int __must_check ax25_rt_add(stru ax25_rt-digipeat-calls[i]= route-digi_addr[i]; } } - write_unlock(ax25_route_lock); + write_unlock_bh(ax25_route_lock); return 0; } ax25_rt = ax25_rt-next; } if ((ax25_rt = kmalloc(sizeof(ax25_route), GFP_ATOMIC)) == NULL) { - write_unlock(ax25_route_lock); + write_unlock_bh(ax25_route_lock); return -ENOMEM; } @@ -120,7 +120,7 @@ static int __must_check ax25_rt_add(stru ax25_rt-ip_mode = ' '; if (route-digi_count != 0) { if ((ax25_rt-digipeat = kmalloc(sizeof(ax25_digi), GFP_ATOMIC)) == NULL) { - write_unlock(ax25_route_lock); + write_unlock_bh(ax25_route_lock); kfree(ax25_rt); return -ENOMEM; } @@ -133,7 +133,7 @@ static int __must_check ax25_rt_add(stru } ax25_rt-next = ax25_route_list; ax25_route_list = ax25_rt; - write_unlock(ax25_route_lock); + write_unlock_bh(ax25_route_lock); return 0; } @@ -152,7 +152,7 @@ static int ax25_rt_del(struct ax25_route if ((ax25_dev = ax25_addr_ax25dev(route-port_addr)) == NULL) return -EINVAL; - write_lock(ax25_route_lock); + write_lock_bh(ax25_route_lock); ax25_rt = ax25_route_list; while (ax25_rt != NULL) { @@ -174,7 +174,7 @@ static int ax25_rt_del(struct ax25_route } } } - write_unlock(ax25_route_lock); + write_unlock_bh(ax25_route_lock); return 0; } @@ -188,7 +188,7 @@ static int ax25_rt_opt(struct ax25_route if ((ax25_dev = ax25_addr_ax25dev(rt_option-port_addr)) == NULL) return -EINVAL; - write_lock(ax25_route_lock); + write_lock_bh(ax25_route_lock); ax25_rt = ax25_route_list; while (ax25_rt != NULL) { @@ -216,7 +216,7 @@ static int ax25_rt_opt
[BUG][AX25] mkiss and ax25_route lockdep warning
Hello, After using Lock debugging: prove locking correctness with the Kernel I got this warning: = [ INFO: inconsistent lock state ] 2.6.24-dg8ngn-p02 #1 - inconsistent {softirq-on-W} - {in-softirq-R} usage. linuxnet/3046 [HC0[0]:SC1[2]:HE1:SE0] takes: (ax25_route_lock){--.+}, at: [f8a0cfb7] ax25_get_route+0x18/0xb7 [ax25] {softirq-on-W} state was registered at: [c013a340] __lock_acquire+0x464/0xb7b [c01398d8] mark_held_locks+0x39/0x53 [c0124d35] local_bh_enable_ip+0xcd/0xd5 [c0139aaf] trace_hardirqs_on+0x11a/0x13d [c013ae58] lock_acquire+0x5f/0x77 [f8a0d381] ax25_rt_ioctl+0x66/0x325 [ax25] [c0299ce7] _write_lock+0x29/0x34 [f8a0d381] ax25_rt_ioctl+0x66/0x325 [ax25] [f8a0d381] ax25_rt_ioctl+0x66/0x325 [ax25] [f8a0f699] ax25_ioctl+0x1b6/0x5b8 [ax25] [c0161280] fd_install+0x1e/0x46 [c0299c7a] _spin_lock+0x29/0x34 [c022e75d] sock_ioctl+0x1bb/0x1e0 [c022e5a2] sock_ioctl+0x0/0x1e0 [c016c76f] do_ioctl+0x1f/0x62 [c016c9d2] vfs_ioctl+0x220/0x232 [c0139aaf] trace_hardirqs_on+0x11a/0x13d [c016ca17] sys_ioctl+0x33/0x4c [c0103ea2] syscall_call+0x7/0xb [] 0x irq event stamp: 12 hardirqs last enabled at (12): [c0124d35] local_bh_enable_ip+0xcd/0xd5 hardirqs last disabled at (11): [c0124cc3] local_bh_enable_ip+0x5b/0xd5 softirqs last enabled at (119892): [f89fd53d] mkiss_receive_buf+0x2a5/0x394 [mkiss] softirqs last disabled at (119893): [c01249f0] do_softirq+0x37/0x4d other info that might help us debug this: 5 locks held by linuxnet/3046: #0: (tty-atomic_write_lock){--..}, at: [c01ded5e] tty_write_lock+0x11/0x37 #1: (rcu_read_lock){..--}, at: [c023a8d2] net_rx_action+0x4e/0x1c4 #2: (rcu_read_lock){..--}, at: [c0238579] netif_receive_skb+0xe6/0x3d6 #3: (rcu_read_lock){..--}, at: [c025549d] ip_local_deliver_finish+0x2d/0x1f7 #4: (slock-AF_INET){-+..}, at: [c0274e43] icmp_send+0x10e/0x37a stack backtrace: Pid: 3046, comm: linuxnet Not tainted 2.6.24-dg8ngn-p02 #1 [c0138e13] print_usage_bug+0x138/0x142 [c013961f] mark_lock+0x1ca/0x44a [c013a2d9] __lock_acquire+0x3fd/0xb7b [c01398d8] mark_held_locks+0x39/0x53 [c0124d35] local_bh_enable_ip+0xcd/0xd5 [c013ae58] lock_acquire+0x5f/0x77 [f8a0cfb7] ax25_get_route+0x18/0xb7 [ax25] [c0299dc7] _read_lock+0x29/0x34 [f8a0cfb7] ax25_get_route+0x18/0xb7 [ax25] [f8a0cfb7] ax25_get_route+0x18/0xb7 [ax25] [f89fda29] ax_header+0x0/0x1a [mkiss] [f8a0c3f6] ax25_rebuild_header+0x30/0x202 [ax25] [f89fda29] ax_header+0x0/0x1a [mkiss] [c023f90c] neigh_compat_output+0x7b/0x97 [c0258bd6] ip_finish_output+0x1da/0x204 [c0259a26] ip_output+0x74/0x89 [c0257831] ip_push_pending_frames+0x2d8/0x33a [c025742c] dst_output+0x0/0x7 [c0275039] icmp_send+0x304/0x37a [c013a353] __lock_acquire+0x477/0xb7b [c0270c05] __udp4_lib_lookup+0xec/0xf6 [c0271a48] __udp4_lib_rcv+0x586/0x771 [c02555ae] ip_local_deliver_finish+0x13e/0x1f7 [c025549d] ip_local_deliver_finish+0x2d/0x1f7 [c0255451] ip_rcv_finish+0x2c1/0x2e0 [c025591c] ip_rcv+0x1f0/0x22b [c0255190] ip_rcv_finish+0x0/0x2e0 [c0238808] netif_receive_skb+0x375/0x3d6 [c0238579] netif_receive_skb+0xe6/0x3d6 [c023adc4] process_backlog+0x6c/0xcd [c023a940] net_rx_action+0xbc/0x1c4 [c023a8d2] net_rx_action+0x4e/0x1c4 [c0124944] __do_softirq+0x69/0xde [f89fd53d] mkiss_receive_buf+0x2a5/0x394 [mkiss] [c01249f0] do_softirq+0x37/0x4d [c0124d15] local_bh_enable_ip+0xad/0xd5 [f89fd53d] mkiss_receive_buf+0x2a5/0x394 [mkiss] [c029a042] _spin_unlock_irqrestore+0x34/0x39 [c01e3233] pty_write+0x2f/0x39 [c01e1233] write_chan+0x22d/0x2a1 [c01191e7] default_wake_function+0x0/0x8 [c01def37] tty_write+0x14d/0x1c2 [c01e1006] write_chan+0x0/0x2a1 [c01dedea] tty_write+0x0/0x1c2 [c0162f28] vfs_write+0x8a/0x10c [c01634a4] sys_write+0x41/0x67 [c0103ea2] syscall_call+0x7/0xb === -- Jann Traschewski, Drosselstr.1, D-90513 Zirndorf, Germany Tel.: +49-911-696971, Mobile: +49-170-1045937, EMail: [EMAIL PROTECTED] Ham: DG8NGN / DB0VOX, http://www.qsl.net/db0fhn, ICQ UIN: 4130182 -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH][AX25] ax25_ds_timer: use mod_timer instead of add_timer
Patches from Jarek applied (incl. both testing patches). Machine is stable since 2 days now. Regards, Jann -Ursprüngliche Nachricht- Von: Jarek Poplawski [mailto:[EMAIL PROTECTED] Gesendet: Mittwoch, 6. Februar 2008 10:14 An: netdev@vger.kernel.org Cc: Ralf Baechle; Jann Traschewski; David Miller Betreff: [PATCH][AX25] ax25_ds_timer: use mod_timer instead of add_timer On Wed, Feb 06, 2008 at 08:15:09AM +, Jarek Poplawski wrote: On Wed, Feb 06, 2008 at 07:45:29AM +, Jarek Poplawski wrote: ... From: Jann Traschewski [EMAIL PROTECTED] Subject: SMP with AX.25 ... [AX25] ax25_ds_timer: use mod_timer instead of add_timer This patch changes current use of: init_timer(), add_timer() and del_timer() to setup_timer() with mod_timer(), which should be safer anyway. Reported-by: Jann Traschewski [EMAIL PROTECTED] Signed-off-by: Jarek Poplawski [EMAIL PROTECTED] --- include/net/ax25.h |1 + net/ax25/ax25_dev.c |2 +- net/ax25/ax25_ds_timer.c | 12 3 files changed, 6 insertions(+), 9 deletions(-) diff --git a/include/net/ax25.h b/include/net/ax25.h index 3f0236f..717e219 100644 --- a/include/net/ax25.h +++ b/include/net/ax25.h @@ -324,6 +324,7 @@ extern void ax25_dama_on(ax25_cb *); extern void ax25_dama_off(ax25_cb *); /* ax25_ds_timer.c */ +extern void ax25_ds_setup_timer(ax25_dev *); extern void ax25_ds_set_timer(ax25_dev *); extern void ax25_ds_del_timer(ax25_dev *); extern void ax25_ds_timer(ax25_cb *); diff --git a/net/ax25/ax25_dev.c b/net/ax25/ax25_dev.c index 528c874..a7a0e0c 100644 --- a/net/ax25/ax25_dev.c +++ b/net/ax25/ax25_dev.c @@ -82,7 +82,7 @@ void ax25_dev_device_up(struct net_device *dev) ax25_dev-values[AX25_VALUES_DS_TIMEOUT]= AX25_DEF_DS_TIMEOUT; #if defined(CONFIG_AX25_DAMA_SLAVE) || defined(CONFIG_AX25_DAMA_MASTER) - init_timer(ax25_dev-dama.slave_timer); + ax25_ds_setup_timer(ax25_dev); #endif spin_lock_bh(ax25_dev_lock); diff --git a/net/ax25/ax25_ds_timer.c b/net/ax25/ax25_ds_timer.c index c4e3b02..2ce79df 100644 --- a/net/ax25/ax25_ds_timer.c +++ b/net/ax25/ax25_ds_timer.c @@ -40,13 +40,10 @@ static void ax25_ds_timeout(unsigned long); * 1/10th of a second. */ -static void ax25_ds_add_timer(ax25_dev *ax25_dev) +void ax25_ds_setup_timer(ax25_dev *ax25_dev) { - struct timer_list *t = ax25_dev-dama.slave_timer; - t-data = (unsigned long) ax25_dev; - t-function = ax25_ds_timeout; - t-expires = jiffies + HZ; - add_timer(t); + setup_timer(ax25_dev-dama.slave_timer, ax25_ds_timeout, + (unsigned long)ax25_dev); } void ax25_ds_del_timer(ax25_dev *ax25_dev) @@ -60,10 +57,9 @@ void ax25_ds_set_timer(ax25_dev *ax25_dev) if (ax25_dev == NULL) /* paranoia */ return; - del_timer(ax25_dev-dama.slave_timer); ax25_dev-dama.slave_timeout = msecs_to_jiffies(ax25_dev-values[AX25_VALUES_DS_TIMEOUT]) / 10; - ax25_ds_add_timer(ax25_dev); + mod_timer(ax25_dev-dama.slave_timer, jiffies + HZ); } /* -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html