This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:
apport-collect 1987269 and then change the status of the bug to 'Confirmed'. If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'. This change has been made by an automated script, maintained by the Ubuntu Kernel Team. ** Changed in: linux (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987269 Title: [5.18.0 mainline] mlx5 deadlock Status in linux package in Ubuntu: Incomplete Bug description: $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.1 LTS Release: 22.04 Codename: jammy $ uname -a Linux pc1-rb3-n3 5.18.0-051800-generic #202205222030 SMP PREEMPT_DYNAMIC Sun May 22 20:33:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.735461] INFO: task ovn-controller:30439 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.742835] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.749051] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757010] task:ovn-controller state:D stack: 0 pid:30439 ppid: 1 flags:0x00000002 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757019] Call Trace: Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757024] <TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757031] __schedule+0x246/0x5c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757045] schedule+0x55/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757050] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757057] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757064] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757069] mutex_lock+0x35/0x40 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757073] rtnetlink_rcv_msg+0x114/0x3f0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757082] ? rtnl_calcit.isra.0+0x140/0x140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757087] netlink_rcv_skb+0x56/0x100 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757095] rtnetlink_rcv+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757099] netlink_unicast+0x236/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757104] netlink_sendmsg+0x25e/0x4d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757109] sock_sendmsg+0x66/0x70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757117] ____sys_sendmsg+0x24f/0x290 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757123] ? import_iovec+0x1b/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757130] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757137] ___sys_sendmsg+0x81/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757143] ? ___sys_recvmsg+0x99/0x110 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757148] ? netdev_name_node_lookup_rcu+0x6b/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757155] ? fput+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757161] ? __sys_sendmsg+0x98/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757166] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757174] __sys_sendmsg+0x62/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757180] __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757186] do_syscall_64+0x5c/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757191] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757194] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757200] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757203] ? exit_to_user_mode_prepare+0x37/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757210] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757214] ? __x64_sys_write+0x19/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757220] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757224] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757227] entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757232] RIP: 0033:0x7f571f927b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757239] RSP: 002b:00007ffd50fa8ec0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757246] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f571f927b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757252] RDX: 0000000000000000 RSI: 00007ffd50fa8f50 RDI: 0000000000000020 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757255] RBP: 00007ffd50fa9c40 R08: 0000000000000000 R09: 000055e1a6a66f60 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757260] R10: 0000000000000001 R11: 0000000000000293 R12: 0000000000000000 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757264] R13: 000055e1a53bc510 R14: 0000000000074f30 R15: 00007ffd50fa8f50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757271] </TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757646] INFO: task ovs-vswitchd:45997 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.764817] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.771036] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.778998] task:ovs-vswitchd state:D stack: 0 pid:45997 ppid: 1 flags:0x00000002 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779005] Call Trace: Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779007] <TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779009] __schedule+0x246/0x5c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779017] schedule+0x55/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779023] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779029] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779036] ? netlink_unicast+0x305/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779041] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779046] mutex_lock+0x35/0x40 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779050] rtnetlink_rcv_msg+0x114/0x3f0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779056] ? rtnl_calcit.isra.0+0x140/0x140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779060] netlink_rcv_skb+0x56/0x100 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779065] rtnetlink_rcv+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779069] netlink_unicast+0x236/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779077] netlink_sendmsg+0x25e/0x4d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779083] sock_sendmsg+0x66/0x70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779090] ____sys_sendmsg+0x24f/0x290 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779096] ? import_iovec+0x1b/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779101] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779107] ___sys_sendmsg+0x81/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779112] ? _copy_from_user+0x3e/0x60 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779117] ? iovec_from_user.part.0+0x59/0x190 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779121] ? __import_iovec+0x47/0x150 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779126] ? ___sys_recvmsg+0x99/0x110 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779131] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779136] __sys_sendmsg+0x62/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779143] __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779149] do_syscall_64+0x5c/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779154] ? exit_to_user_mode_prepare+0x37/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779159] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779163] ? __x64_sys_recvmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779168] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779172] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779177] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779181] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779185] ? asm_sysvec_apic_timer_interrupt+0xa/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779192] entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779197] RIP: 0033:0x7f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779201] RSP: 002b:00007ffcefe1f410 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779208] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779213] RDX: 0000000000000000 RSI: 00007ffcefe1f4a0 RDI: 0000000000000141 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779217] RBP: 00007ffcefe201d8 R08: 0000000000000000 R09: 0000000000000020 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779220] R10: 0000000000000001 R11: 0000000000000293 R12: 00007f4014006030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779223] R13: 0000000000000001 R14: 0000000000000003 R15: 00007ffcefe1f4a0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779229] </TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779247] INFO: task handler98:46136 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.786162] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.792382] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800327] task:handler98 state:D stack: 0 pid:46136 ppid: 1 flags:0x00004002 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800331] Call Trace: Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800333] <TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800334] __schedule+0x246/0x5c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800340] ? mlx5dr_send_postsend_action+0x70/0x90 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800429] schedule+0x55/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800433] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800436] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800441] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800444] mutex_lock+0x35/0x40 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800448] mlx5_tc_ct_flow_offload+0x36/0x60 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800526] mlx5e_tc_rule_offload+0xa3/0x110 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800600] mlx5e_tc_offload_fdb_rules+0x36/0xb0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800668] mlx5e_tc_add_fdb_flow+0x3a2/0x410 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800740] __mlx5e_add_fdb_flow+0x2be/0x3e0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800806] mlx5e_add_fdb_flow+0x39/0x220 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800871] mlx5e_configure_flower+0x3af/0x3e0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800936] mlx5e_rep_setup_tc_cls_flower+0x3a/0x70 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801001] mlx5e_rep_setup_tc_cb+0x58/0x80 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801065] tc_setup_cb_add+0xcf/0x200 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801071] fl_hw_replace_filter+0x15b/0x240 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801077] fl_change+0x8b1/0xbf0 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801082] tc_new_tfilter+0x3e5/0xb70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801086] ? skb_queue_tail+0x48/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801091] ? tcf_exts_terse_dump+0xc0/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801094] rtnetlink_rcv_msg+0x2fd/0x3f0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801097] ? skb_free_head+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801101] ? rtnl_calcit.isra.0+0x140/0x140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801103] netlink_rcv_skb+0x56/0x100 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801107] rtnetlink_rcv+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801109] netlink_unicast+0x236/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801112] netlink_sendmsg+0x25e/0x4d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801116] sock_sendmsg+0x66/0x70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801119] ____sys_sendmsg+0x24f/0x290 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801122] ? import_iovec+0x1b/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801125] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801129] ___sys_sendmsg+0x81/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801132] ? ___sys_sendmsg+0x8e/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801135] ? __import_iovec+0x47/0x150 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801138] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801141] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801144] __sys_sendmsg+0x62/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801148] __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801151] do_syscall_64+0x5c/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801154] ? exit_to_user_mode_prepare+0x37/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801157] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801160] ? __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801163] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801166] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801169] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801171] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801174] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801176] entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801178] RIP: 0033:0x7f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801179] RSP: 002b:00007f3fcffb2490 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801182] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801183] RDX: 0000000000000000 RSI: 00007f3fcffb2520 RDI: 000000000000013e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801185] RBP: 00007f3fcffb3258 R08: 0000000000000000 R09: 0000000000000134 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801186] R10: 0000000000007463 R11: 0000000000000293 R12: 00007f3ed804cda0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801188] R13: 0000000000000001 R14: 000000000000000f R15: 00007f3fcffb2520 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801191] </TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801193] INFO: task handler105:46143 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.808173] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.814381] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822319] task:handler105 state:D stack: 0 pid:46143 ppid: 1 flags:0x00004002 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822324] Call Trace: Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822326] <TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822328] __schedule+0x246/0x5c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822332] ? mlx5dr_send_postsend_action+0x70/0x90 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822408] schedule+0x55/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822412] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822415] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822419] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822423] mutex_lock+0x35/0x40 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822426] mlx5_tc_ct_flow_offload+0x36/0x60 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822500] mlx5e_tc_rule_offload+0xa3/0x110 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822569] mlx5e_tc_offload_fdb_rules+0x36/0xb0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822635] mlx5e_tc_add_fdb_flow+0x3a2/0x410 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822702] __mlx5e_add_fdb_flow+0x2be/0x3e0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822771] mlx5e_add_fdb_flow+0x15c/0x220 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822834] mlx5e_configure_flower+0x3af/0x3e0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822898] mlx5e_rep_setup_tc_cls_flower+0x3a/0x70 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822989] mlx5e_rep_setup_tc_cb+0x58/0x80 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823054] tc_setup_cb_add+0xcf/0x200 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823058] fl_hw_replace_filter+0x15b/0x240 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823064] fl_change+0x8b1/0xbf0 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823070] tc_new_tfilter+0x3e5/0xb70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823077] ? tcf_exts_terse_dump+0xc0/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823080] rtnetlink_rcv_msg+0x2fd/0x3f0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823084] ? skb_free_head+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823089] ? rtnl_calcit.isra.0+0x140/0x140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823094] netlink_rcv_skb+0x56/0x100 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823098] rtnetlink_rcv+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823102] netlink_unicast+0x236/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823106] netlink_sendmsg+0x25e/0x4d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823112] sock_sendmsg+0x66/0x70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823116] ____sys_sendmsg+0x24f/0x290 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823119] ? import_iovec+0x1b/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823122] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823127] ___sys_sendmsg+0x81/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823131] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823135] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823139] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823143] __sys_sendmsg+0x62/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823148] __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823152] do_syscall_64+0x5c/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823155] ? pebs_update_state+0x40/0x2e0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823162] ? exit_to_user_mode_prepare+0x37/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823166] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823170] ? __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823174] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823178] entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823182] RIP: 0033:0x7f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823186] RSP: 002b:00007f3faffb2490 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823191] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823195] RDX: 0000000000000000 RSI: 00007f3faffb2520 RDI: 0000000000000144 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823198] RBP: 00007f3faffb3258 R08: 0000000000000000 R09: 0000000000000134 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823201] R10: 0000000000007463 R11: 0000000000000293 R12: 00007f3fbc015dd0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823204] R13: 0000000000000001 R14: 0000000000000003 R15: 00007f3faffb2520 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823208] </TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823213] INFO: task revalidator130:46169 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.830553] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.836758] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844692] task:revalidator130 state:D stack: 0 pid:46169 ppid: 1 flags:0x00004002 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844696] Call Trace: Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844697] <TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844699] __schedule+0x246/0x5c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844702] schedule+0x55/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844705] schedule_timeout+0x115/0x150 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844707] ? __free_pages+0xa1/0xf0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844713] __wait_for_common+0x92/0x180 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844715] ? usleep_range_state+0x90/0x90 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844717] wait_for_completion+0x24/0x30 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844719] flush_workqueue+0x133/0x3e0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844725] mlx5_tc_ct_del_ft_cb.part.0+0x120/0x180 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844774] mlx5_tc_ct_delete_flow+0xb7/0xd0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844819] mlx5e_tc_rule_unoffload+0x8f/0xf0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844865] mlx5e_tc_unoffload_fdb_rules+0x36/0x60 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844909] mlx5e_tc_del_fdb_flow+0x227/0x3e0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844952] __mlx5e_tc_del_fdb_peer_flow+0xcc/0xf0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844994] mlx5e_flow_put+0x63/0xc0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845035] mlx5e_delete_flower+0x202/0x270 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845077] mlx5e_rep_setup_tc_cls_flower+0x6a/0x70 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845119] mlx5e_rep_setup_tc_cb+0x58/0x80 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845160] tc_setup_cb_destroy+0xb5/0x190 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845162] fl_hw_destroy_filter+0x95/0xc0 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845166] __fl_delete+0x1d0/0x1f0 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845169] fl_delete+0x39/0x80 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845172] tc_del_tfilter+0x385/0x910 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845176] ? tc_setup_cb_add+0x200/0x200 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845178] rtnetlink_rcv_msg+0x2fd/0x3f0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845181] ? rtnl_calcit.isra.0+0x140/0x140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845183] netlink_rcv_skb+0x56/0x100 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845185] rtnetlink_rcv+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845187] netlink_unicast+0x236/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845189] netlink_sendmsg+0x25e/0x4d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845191] sock_sendmsg+0x66/0x70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845193] ____sys_sendmsg+0x24f/0x290 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845195] ? import_iovec+0x1b/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845197] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845199] ___sys_sendmsg+0x81/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845201] ? __import_iovec+0x47/0x150 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845203] ? ___sys_recvmsg+0x99/0x110 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845205] ? move_addr_to_user+0x4d/0xe0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845207] ? __sys_getsockname+0xcb/0xe0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845209] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845211] __sys_sendmsg+0x62/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845213] ? pebs_update_state+0x40/0x2e0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845216] __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845218] do_syscall_64+0x5c/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845220] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845222] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845224] ? exit_to_user_mode_prepare+0x37/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845226] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845228] ? __x64_sys_recvmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845230] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845231] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845233] ? __x64_sys_socket+0x17/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845235] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845236] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845237] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845239] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845240] ? sysvec_apic_timer_interrupt+0x4e/0x90 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845242] ? asm_sysvec_apic_timer_interrupt+0xa/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845243] entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845245] RIP: 0033:0x7f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845246] RSP: 002b:00007f3f4dff05a0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845247] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845249] RDX: 0000000000000000 RSI: 00007f3f4dff0630 RDI: 0000000000000140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845250] RBP: 00007f3f4dff1368 R08: 0000000000000000 R09: 0000000000000010 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845251] R10: 0000000000000001 R11: 0000000000000293 R12: 00007f3f30019bd0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845251] R13: 0000000000000001 R14: 0000000000000002 R15: 00007f3f4dff0630 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845254] </TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845536] INFO: task revalidator133:46172 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.852972] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.859174] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867104] task:revalidator133 state:D stack: 0 pid:46172 ppid: 1 flags:0x00000002 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867109] Call Trace: Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867111] <TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867113] __schedule+0x246/0x5c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867120] schedule+0x55/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867124] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867129] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867133] ? __nla_validate_parse+0x14c/0x1a0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867140] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867143] mutex_lock+0x35/0x40 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867145] rtnl_lock+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867148] tc_get_tfilter+0x11a/0x5d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867151] ? _copy_to_iter+0xcc/0x6c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867154] ? tc_new_tfilter+0xb70/0xb70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867158] rtnetlink_rcv_msg+0x2fd/0x3f0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867161] ? rtnl_calcit.isra.0+0x140/0x140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867163] netlink_rcv_skb+0x56/0x100 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867166] rtnetlink_rcv+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867167] netlink_unicast+0x236/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867169] netlink_sendmsg+0x25e/0x4d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867172] sock_sendmsg+0x66/0x70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867174] ____sys_sendmsg+0x24f/0x290 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867176] ? import_iovec+0x1b/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867178] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867181] ? iovec_from_user.part.0+0x59/0x190 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867183] ___sys_sendmsg+0x81/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867185] ? fput+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867188] ? __sys_recvmsg+0x95/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867191] ? 0xffffffffb5000000 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867193] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867198] __sys_sendmsg+0x62/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867201] ? pebs_update_state+0x40/0x2e0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867204] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867207] __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867210] do_syscall_64+0x5c/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867213] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867216] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867218] entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867221] RIP: 0033:0x7f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867224] RSP: 002b:00007f3f2fff44b0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867228] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867231] RDX: 0000000000000000 RSI: 00007f3f2fff4540 RDI: 000000000000013f Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867234] RBP: 00007f3f2fff5278 R08: 0000000000000000 R09: 0000000000000024 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867238] R10: 0000000000000000 R11: 0000000000000293 R12: 00007f3f2004b130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867240] R13: 0000000000000001 R14: 0000000000000005 R15: 00007f3f2fff4540 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867244] </TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867385] INFO: task kworker/u256:0:1812497 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.874888] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.881104] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889038] task:kworker/u256:0 state:D stack: 0 pid:1812497 ppid: 2 flags:0x00004000 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889043] Workqueue: mlx5e mlx5e_rep_neigh_update [mlx5_core] Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889103] Call Trace: Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889105] <TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889106] __schedule+0x246/0x5c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889108] ? sched_clock_local+0x13/0x90 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889113] schedule+0x55/0xc0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889116] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889118] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889121] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889124] mutex_lock+0x35/0x40 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889126] rtnl_lock+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889129] mlx5e_rep_neigh_update+0x35/0x1a0 [mlx5_core] Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889175] process_one_work+0x21f/0x3f0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889178] worker_thread+0x50/0x3e0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889181] ? rescuer_thread+0x390/0x390 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889183] kthread+0xee/0x120 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889187] ? kthread_complete_and_exit+0x20/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889190] ret_from_fork+0x22/0x30 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889195] </TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889200] INFO: task kworker/56:0:2429317 blocked for more than 120 seconds. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.896544] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.902740] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910672] task:kworker/56:0 state:D stack: 0 pid:2429317 ppid: 2 flags:0x00004000 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910677] Workqueue: ipv6_addrconf addrconf_verify_work Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910684] Call Trace: Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910685] <TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910686] __schedule+0x246/0x5c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910689] ? newidle_balance+0x30d/0x480 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910694] schedule+0x55/0xc0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910695] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910698] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910701] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910703] mutex_lock+0x35/0x40 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910705] rtnl_lock+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910707] addrconf_verify_work+0x12/0x30 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910711] process_one_work+0x21f/0x3f0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910715] worker_thread+0x50/0x3e0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910718] ? rescuer_thread+0x390/0x390 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910722] kthread+0xee/0x120 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910725] ? kthread_complete_and_exit+0x20/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910728] ret_from_fork+0x22/0x30 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910732] </TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910740] INFO: task kworker/u256:51:2462708 blocked for more than 121 seconds. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.918334] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.924539] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932489] task:kworker/u256:51 state:D stack: 0 pid:2462708 ppid: 2 flags:0x00004000 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932493] Workqueue: mlx5_fc mlx5e_rep_neigh_stats_work [mlx5_core] Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932552] Call Trace: Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932553] <TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932555] __schedule+0x246/0x5c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932558] ? default_send_IPI_single_phys+0x4c/0x80 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932563] schedule+0x55/0xc0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932565] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932568] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932572] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932575] mutex_lock+0x35/0x40 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932577] rtnl_lock+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932580] mlx5e_rep_neigh_stats_work+0x20/0x120 [mlx5_core] Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932632] process_one_work+0x21f/0x3f0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932635] worker_thread+0x50/0x3e0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932638] ? rescuer_thread+0x390/0x390 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932640] kthread+0xee/0x120 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932642] ? kthread_complete_and_exit+0x20/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932645] ret_from_fork+0x22/0x30 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932648] </TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932654] INFO: task kworker/u256:29:2732306 blocked for more than 121 seconds. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.940241] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.946494] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954437] task:kworker/u256:29 state:D stack: 0 pid:2732306 ppid: 2 flags:0x00004000 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954441] Workqueue: mlx5_fc mlx5e_rep_neigh_stats_work [mlx5_core] Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954486] Call Trace: Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954487] <TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954488] __schedule+0x246/0x5c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954491] ? default_send_IPI_single_phys+0x4c/0x80 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954494] schedule+0x55/0xc0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954495] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954498] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954500] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954504] mutex_lock+0x35/0x40 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954507] rtnl_lock+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954511] mlx5e_rep_neigh_stats_work+0x20/0x120 [mlx5_core] Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954554] process_one_work+0x21f/0x3f0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954557] worker_thread+0x50/0x3e0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954560] ? rescuer_thread+0x390/0x390 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954564] kthread+0xee/0x120 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954567] ? kthread_complete_and_exit+0x20/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954570] ret_from_fork+0x22/0x30 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954574] </TASK> To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1987269/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp