Hi Holger,
We will look into this. It is a new setup for us, so it has to be
prepared first.
///jon
On 11/19/20 9:33 AM, Holger Brunck wrote:
Hi all,
we have currently problems with TIPC in combination with a TAP interface and
fragmented
messages. We have a Kirkwood board getting TIPC ethernet packets from a third
party stack
in user space which we fill into a TAP device to pass this packets to the
kernel. On the
remote side we have another board with a physical ethernet device. After
migrating to
kernel 5.4 we saw kernel crashes as soon as we send fragmented TIPC messages via
this link. Before the kernel crash we see "refcount_t: underflow;
use-after-free"
warnings coming either from the TIPC or the driver/net/tun.
For smaller packets everything work fine. To track this down we wrote a small
test program which receives raw ethernet packets in user space and send them
back to
kernel space via a TAP interface, to be able to reproduce it with a mainline
kernel only.
So instead of a link e.g. eth0:eth0 we have a link from eth0:tap0.
I now tested this on latest mainline kernel and still see warnings coming from
the kernel, but
not the crash.
So in the testcase we have TIPC server running on a remote board having a
ethernet bearer
on a interface. The ethernet interface is connected to the ethernet interface
of the
kirkwood board. The incoming messages are passed to user space and filled into
the TAP device.
So we have a TIPC link between tap0 and eth0. If I send now back and forth
TIPC connectionless
messages to the server via the TAP device the kernel print the following
messages as soon as
the packets are larger then the MTU and TIPC need to reassemble the fragmented
messages coming
from the TAP driver:
root@km_kirkwood:~#
------------[ cut here ]------------
WARNING: CPU: 0 PID: 94 at lib/refcount.c:28 sock_wfree+0xd8/0xe0
refcount_t: underflow; use-after-free.
Modules linked in:
CPU: 0 PID: 94 Comm: tipc_ping Tainted: G W
5.10.0-rc4-17824-g0fa8ee0d9ab9 #112
Hardware name: Marvell Kirkwood (Flattened Device Tree)
[<c000f924>] (unwind_backtrace) from [<c000c980>] (show_stack+0x10/0x14)
[<c000c980>] (show_stack) from [<c001c27c>] (__warn+0xe8/0xfc)
[<c001c27c>] (__warn) from [<c001c2fc>] (warn_slowpath_fmt+0x6c/0xb8)
[<c001c2fc>] (warn_slowpath_fmt) from [<c0564b78>] (sock_wfree+0xd8/0xe0)
[<c0564b78>] (sock_wfree) from [<c056b730>] (skb_release_all+0xc/0x28)
[<c056b730>] (skb_release_all) from [<c056b784>] (kfree_skb+0x20/0x6c)
[<c056b784>] (kfree_skb) from [<c056be44>] (skb_release_data+0xc0/0x140)
[<c056be44>] (skb_release_data) from [<c056b784>] (kfree_skb+0x20/0x6c)
[<c056b784>] (kfree_skb) from [<c0720ea4>] (tipc_recvmsg+0x270/0x3c4)
[<c0720ea4>] (tipc_recvmsg) from [<c0560428>] (__sys_recvfrom+0xb0/0x128)
[<c0560428>] (__sys_recvfrom) from [<c05604bc>] (sys_recv+0x18/0x20)
[<c05604bc>] (sys_recv) from [<c0008240>] (ret_fast_syscall+0x0/0x50)
Exception stack(0xc1615fa8 to 0xc1615ff0)
5fa0: 00000001 00000000 00000003 00025010 0000fa00 00000000
5fc0: 00000001 00000000 00010888 00000123 00000000 00000000 49cb0000 be939bec
5fe0: 00000000 be93997c 00011114 49d9666c
---[ end trace e7c3a1366c544d79 ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 74 at lib/refcount.c:22 sock_alloc_send_pskb+0x240/0x258
refcount_t: saturated; leaking memory.
Modules linked in:
CPU: 0 PID: 74 Comm: main Tainted: G W
5.10.0-rc4-17824-g0fa8ee0d9ab9 #112
Hardware name: Marvell Kirkwood (Flattened Device Tree)
[<c000f924>] (unwind_backtrace) from [<c000c980>] (show_stack+0x10/0x14)
[<c000c980>] (show_stack) from [<c001c27c>] (__warn+0xe8/0xfc)
[<c001c27c>] (__warn) from [<c001c2fc>] (warn_slowpath_fmt+0x6c/0xb8)
[<c001c2fc>] (warn_slowpath_fmt) from [<c0563b40>]
(sock_alloc_send_pskb+0x240/0x258)
[<c0563b40>] (sock_alloc_send_pskb) from [<c043d500>]
(tun_get_user+0x224/0x13f8)
[<c043d500>] (tun_get_user) from [<c043e718>] (tun_chr_write_iter+0x44/0x78)
[<c043e718>] (tun_chr_write_iter) from [<c012f1bc>] (vfs_write+0x1fc/0x448)
[<c012f1bc>] (vfs_write) from [<c012f52c>] (ksys_write+0x54/0xcc)
[<c012f52c>] (ksys_write) from [<c0008240>] (ret_fast_syscall+0x0/0x50)
Exception stack(0xc158bfa8 to 0xc158bff0)
bfa0: 00011244 00000000 00000004 00022098 000005c2 00000004
bfc0: 00011244 00000000 000108bc 00000004 00000000 00000000 49cb0000 becf3c94
bfe0: 00000000 becf3bb0 49e21ce0 49e223e4
---[ end trace e7c3a1366c544d7a ]---
Kernel version:
root@km_kirkwood:~# cat /proc/version
Linux version 5.10.0-rc4-17824-g0fa8ee0d9ab9
If I simply use ethernet bearer on both sides everything works fine, so I need
the TAP interface to be involved to see the problem.
It seems that we have a problem in the skb_buf allocation or handling between
the TAP driver and the TIPC layer as soon it comes to fragmentation.
Does anyone have an idea what could cause the problem and how to track this
down? Any help is appreciated.
Best regards
Holger Brunck
_______________________________________________
tipc-discussion mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tipc-discussion
