Checking On Wed, Aug 24, 2022 at 7:50 PM Jeff Lane <1971...@bugs.launchpad.net> wrote:
> So this is actually in 5.19, and has been pulled into 22.04 via our 5.15 > kernel. Is there anything more to do here? > > commit eb2f403f098fedb4c58283e9532df2ff0d2a36e9 > Author: James Smart <jsmart2...@gmail.com> > Date: Tue Apr 12 15:19:44 2022 -0700 > > scsi: lpfc: Move cfg_log_verbose check before calling lpfc_dmp_dbg() > > BugLink: https://bugs.launchpad.net/bugs/1981864 > > [ Upstream commit e294647b1aed4247fe52851f3a3b2b19ae906228 ] > > In an attempt to log message 0126 with LOG_TRACE_EVENT, the following > hard > lockup call trace hangs the system. > > Call Trace: > _raw_spin_lock_irqsave+0x32/0x40 > lpfc_dmp_dbg.part.32+0x28/0x220 [lpfc] > lpfc_cmpl_els_fdisc+0x145/0x460 [lpfc] > lpfc_sli_cancel_jobs+0x92/0xd0 [lpfc] > lpfc_els_flush_cmd+0x43c/0x670 [lpfc] > lpfc_els_flush_all_cmd+0x37/0x60 [lpfc] > lpfc_sli4_async_event_proc+0x956/0x1720 [lpfc] > lpfc_do_work+0x1485/0x1d70 [lpfc] > kthread+0x112/0x130 > ret_from_fork+0x1f/0x40 > Kernel panic - not syncing: Hard LOCKUP > > The same CPU tries to claim the phba->port_list_lock twice. > > Move the cfg_log_verbose checks as part of the lpfc_printf_vlog() and > lpfc_printf_log() macros before calling lpfc_dmp_dbg(). There is no > need > to take the phba->port_list_lock within lpfc_dmp_dbg(). > > Link: > https://lore.kernel.org/r/20220412222008.126521-3-jsmart2...@gmail.com > Co-developed-by: Justin Tee <justin....@broadcom.com> > Signed-off-by: Justin Tee <justin....@broadcom.com> > Signed-off-by: James Smart <jsmart2...@gmail.com> > Signed-off-by: Martin K. Petersen <martin.peter...@oracle.com> > Signed-off-by: Sasha Levin <sas...@kernel.org> > Signed-off-by: Kamal Mostafa <ka...@canonical.com> > Signed-off-by: Stefan Bader <stefan.ba...@canonical.com> > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/1971193 > > Title: > Server Crash while running IO and switch port bounce test with 2K > login session > > Status in linux package in Ubuntu: > Incomplete > > Bug description: > [Impact] > Server crash and Call trace reported on one of the servers running IO and > switch port bounce test from the 2K login session configuration. > > Call Trace: > [56048.470488] Call Trace: > [56048.470489] _raw_spin_lock_irqsave+0x32/0x40 > [56048.470489] lpfc_dmp_dbg.part.32+0x28/0x220 [lpfc] > [56048.470490] lpfc_cmpl_els_fdisc+0x145/0x460 [lpfc] > [56048.470490] lpfc_sli_cancel_jobs+0x92/0xd0 [lpfc] > [56048.470490] lpfc_els_flush_cmd+0x43c/0x670 [lpfc] > [56048.470491] lpfc_els_flush_all_cmd+0x37/0x60 [lpfc] > [56048.470491] lpfc_sli4_async_event_proc+0x956/0x1720 [lpfc] > [56048.470492] lpfc_do_work+0x1485/0x1d70 [lpfc] > [56048.470492] ? __schedule+0x280/0x700 > [56048.470492] ? finish_wait+0x80/0x80 > [56048.470493] ? lpfc_unregister_unused_fcf+0x80/0x80 [lpfc] > [56048.470493] kthread+0x112/0x130 > [56048.470493] ? kthread_flush_work_fn+0x10/0x10 > [56048.470494] ret_from_fork+0x1f/0x40 > [56048.470494] Kernel panic - not syncing: Hard LOCKUP > [56048.470495] CPU: 0 PID: 682 Comm: lpfc_worker_0 Kdump: loaded > Tainted: G > IOE --------- - - 4.18.0-240.el8.x86_64 #1 > [56048.470496] Hardware name: Dell Inc. PowerEdge R740/0DY2X0, BIOS > 2.11.2 > 004/21/2021 > [56048.470496] Call Trace: > [56048.470496] <NMI> > [56048.470496] dump_stack+0x5c/0x80 > [56048.470497] panic+0xe7/0x2a9 > [56048.470497] ? __switch_to_asm+0x51/0x70 > [56048.470497] nmi_panic.cold.9+0xc/0xc > [56048.470498] watchdog_overflow_callback.cold.7+0x5c/0x70 > [56048.470498] __perf_event_overflow+0x52/0xf0 > [56048.470499] handle_pmi_common+0x1db/0x270 > [56048.470499] ? __set_pte_vaddr+0x32/0x50 > [56048.470499] ? __native_set_fixmap+0x24/0x30 > [56048.470500] ? ghes_copy_tofrom_phys+0xd3/0x1c0 > [56048.470500] ? __ghes_peek_estatus.isra.12+0x49/0xa0 > [56048.470500] intel_pmu_handle_irq+0xbf/0x160 > [56048.470501] perf_event_nmi_handler+0x2d/0x50 > [56048.470501] nmi_handle+0x63/0x110 > [56048.470501] default_do_nmi+0x4e/0x100 > [56048.470502] do_nmi+0x128/0x190 > [56048.470502] end_repeat_nmi+0x16/0x6a > [56048.470503] RIP: 0010:native_queued_spin_lock_slowpath+0x5d/0x1d0 > [56048.470504] Code: 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 > 30 e4 > 09 d0 a9 00 01 ff ff 75 47 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 8b 07 > <84> c0 75 > f8 b8 01 00 00 00 66 89 07 c3 8b 37 81 fe 00 01 00 00 75 > [56048.470504] RSP: 0018:ffffacebc7877ca8 EFLAGS: 00000002 > [56048.470505] RAX: 0000000000000101 RBX: 0000000000000246 RCX: > 000000000000001f > [56048.470505] RDX: 0000000000000000 RSI: 0000000000000000 RDI: > ffff94dcf5341dc0 > [56048.470506] RBP: ffff94dcf5340000 R08: 0000000000000002 R09: > 0000000000029600 > [56048.470506] R10: 000060d29656a45c R11: ffff94dcf534fd12 R12: > ffff94dcf5341db0 > [56048.470507] R13: ffff94dcf5341dc0 R14: ffff94dcc4ae8a00 R15: > 0000000000000003 > [56048.470507] ? native_queued_spin_lock_slowpath+0x5d/0x1d0 > [56048.470507] ? native_queued_spin_lock_slowpath+0x5d/0x1d0 > [56048.470508] </NMI> > [56048.470508] _raw_spin_lock_irqsave+0x32/0x40 > [56048.470509] lpfc_dmp_dbg.part.32+0x28/0x220 [lpfc] > [56048.470509] lpfc_cmpl_els_fdisc+0x145/0x460 [lpfc] > [56048.470509] lpfc_sli_cancel_jobs+0x92/0xd0 [lpfc] > [56048.470510] lpfc_els_flush_cmd+0x43c/0x670 [lpfc] > [56048.470510] lpfc_els_flush_all_cmd+0x37/0x60 [lpfc] > [56048.470510] lpfc_sli4_async_event_proc+0x956/0x1720 [lpfc] > [56048.470511] lpfc_do_work+0x1485/0x1d70 [lpfc] > [56048.470511] ? __schedule+0x280/0x700 > [56048.470511] ? finish_wait+0x80/0x80 > [56048.470512] ? lpfc_unregister_unused_fcf+0x80/0x80 [lpfc] > [56048.470512] kthread+0x112/0x130 > [56048.470513] ? kthread_flush_work_fn+0x10/0x10 > [56048.470513] ret_from_fork+0x1f/0x40 > [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# > > [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# cat > /etc/redhat-release > Red Hat Enterprise Linux release 8.3 (Ootpa) > > [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# cat > /sys/module/lpfc/version > 0:14.0.390.2 > > [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# cat > /sys/class/scsi_host/host*/modeldesc > Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter > Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter > > [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# cat > /sys/class/scsi_host/host*/fwrev > 14.0.390.1, sli-4:2:c > 14.0.390.1, sli-4:2:c > > [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# cat > /sys/class/fc_host/host*/port_name > 0x10000090faf09459 > 0x10000090faf0945a > [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# > > HBA Attributes for 10:00:00:90:fa:f0:94:59 > > Host Name : ms-svr3-10-231-131-160 > Manufacturer : Emulex Corporation > Serial Number : FC70793283 > Model : LPe32002-M2 > Model Desc : Emulex LightPulse LPe32002-M2 2-Port > 32Gb Fibre > Channel Adapter > Node WWN : 20 00 00 90 fa f0 94 59 > Node Symname : > HW Version : 0000000c 00000001 00000000 > FW Version : 14.0.390.1 > Vendor Spec ID : 10DF > Number of Ports : 1 > Driver Name : lpfc > Driver Version : 14.0.390.2; HBAAPI(I) v2.3.d, 07-12-10 > Device ID : E300 > HBA Type : LPe32002-M2 > Operational FW : 14.0.390.1 > IEEE Address : 00 90 fa f0 94 59 > Boot Code : Enabled > Boot Version : 14.0.390.1 > Board Temperature : Normal > Function Type : FC > Sub Device ID : E300 > PCI Bus Number : 94 > PCI Func Number : 0 > Sub Vendor ID : 10DF > IPL Filename : H62LEX1 > Service Processor FW Name : 14.0.390.1 > ULP FW Name : 14.0.390.1 > FC Universal BIOS Version : 14.0.390.1 > FC x86 BIOS Version : 14.0.390.1 > FC EFI BIOS Version : 14.0.388.0 > FC FCODE Version : 14.0.386.0 > Flash Firmware Version : 14.0.390.1 > Secure Firmware : Enabled > > [root@ms-svr3-10-231-131-160 log]# hbacmd portattrib > 10:00:00:90:fa:f0:94:59 > > Port Attributes for 10:00:00:90:fa:f0:94:59 > > Node WWN : 20 00 00 90 fa f0 94 59 > Port WWN : 10 00 00 90 fa f0 94 59 > Port Symname : > Port FCID : 0000 > Port Type : Unknown > Port State : Link Down > Port Service Type : 8 > Port Supported FC4 : 00 00 01 00 00 00 00 01 > 00 00 00 00 00 00 00 00 > 00 00 00 00 00 00 00 00 > 00 00 00 00 00 00 00 00 > Port Active FC4 : 00 00 01 00 00 00 00 01 > 00 00 00 00 00 00 00 00 > 00 00 00 00 00 00 00 00 > 00 00 00 00 00 00 00 00 > Port Supported Speed : 8 16 32 Gbit/sec > Configured Port Speed : Auto Detect > Port Speed : Not Available > Max Frame Size : 2048 > OS Device Name : /sys/class/scsi_host/host15 > Num Discovered Ports : 0 > Fabric Name : 00 00 00 00 00 00 00 00 > Function Type : FC > FEC : Enabled > > [Fixes] > The following patch will resolve the issue: > scsi: lpfc: Move cfg_log_verbose check before calling lpfc_dmp_dbg() > In an attempt to log message 0126 with LOG_TRACE_EVENT, the following > hard > lockup call trace hangs the system. > > [Testcase] > > > [root@ms-svr3-10-231-131-160 log]# > [reply] [-]Comment 3James Smart 2022-04-13 09:12:37 PDT > Patches pushed upstream 4/12/22: > > https://lore.kernel.org/linux- > scsi/20220412222008.126521-1-jsmart2...@gmail.com/T/#t > > To manage notifications about this bug go to: > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1971193/+subscriptions > > -- *Laurie Barry* Software Program Manager | Emulex Connectivity Division office/soft phone: 978-719-1503 | mobile: 603-930-1771 200 Brickstone Square | Andover, MA 01810 laurie.ba...@broadcom.com | broadcom.com -- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1971193 Title: Server Crash while running IO and switch port bounce test with 2K login session Status in linux package in Ubuntu: Incomplete Bug description: [Impact] Server crash and Call trace reported on one of the servers running IO and switch port bounce test from the 2K login session configuration. Call Trace: [56048.470488] Call Trace: [56048.470489] _raw_spin_lock_irqsave+0x32/0x40 [56048.470489] lpfc_dmp_dbg.part.32+0x28/0x220 [lpfc] [56048.470490] lpfc_cmpl_els_fdisc+0x145/0x460 [lpfc] [56048.470490] lpfc_sli_cancel_jobs+0x92/0xd0 [lpfc] [56048.470490] lpfc_els_flush_cmd+0x43c/0x670 [lpfc] [56048.470491] lpfc_els_flush_all_cmd+0x37/0x60 [lpfc] [56048.470491] lpfc_sli4_async_event_proc+0x956/0x1720 [lpfc] [56048.470492] lpfc_do_work+0x1485/0x1d70 [lpfc] [56048.470492] ? __schedule+0x280/0x700 [56048.470492] ? finish_wait+0x80/0x80 [56048.470493] ? lpfc_unregister_unused_fcf+0x80/0x80 [lpfc] [56048.470493] kthread+0x112/0x130 [56048.470493] ? kthread_flush_work_fn+0x10/0x10 [56048.470494] ret_from_fork+0x1f/0x40 [56048.470494] Kernel panic - not syncing: Hard LOCKUP [56048.470495] CPU: 0 PID: 682 Comm: lpfc_worker_0 Kdump: loaded Tainted: G IOE --------- - - 4.18.0-240.el8.x86_64 #1 [56048.470496] Hardware name: Dell Inc. PowerEdge R740/0DY2X0, BIOS 2.11.2 004/21/2021 [56048.470496] Call Trace: [56048.470496] <NMI> [56048.470496] dump_stack+0x5c/0x80 [56048.470497] panic+0xe7/0x2a9 [56048.470497] ? __switch_to_asm+0x51/0x70 [56048.470497] nmi_panic.cold.9+0xc/0xc [56048.470498] watchdog_overflow_callback.cold.7+0x5c/0x70 [56048.470498] __perf_event_overflow+0x52/0xf0 [56048.470499] handle_pmi_common+0x1db/0x270 [56048.470499] ? __set_pte_vaddr+0x32/0x50 [56048.470499] ? __native_set_fixmap+0x24/0x30 [56048.470500] ? ghes_copy_tofrom_phys+0xd3/0x1c0 [56048.470500] ? __ghes_peek_estatus.isra.12+0x49/0xa0 [56048.470500] intel_pmu_handle_irq+0xbf/0x160 [56048.470501] perf_event_nmi_handler+0x2d/0x50 [56048.470501] nmi_handle+0x63/0x110 [56048.470501] default_do_nmi+0x4e/0x100 [56048.470502] do_nmi+0x128/0x190 [56048.470502] end_repeat_nmi+0x16/0x6a [56048.470503] RIP: 0010:native_queued_spin_lock_slowpath+0x5d/0x1d0 [56048.470504] Code: 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 47 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 8b 07 <84> c0 75 f8 b8 01 00 00 00 66 89 07 c3 8b 37 81 fe 00 01 00 00 75 [56048.470504] RSP: 0018:ffffacebc7877ca8 EFLAGS: 00000002 [56048.470505] RAX: 0000000000000101 RBX: 0000000000000246 RCX: 000000000000001f [56048.470505] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff94dcf5341dc0 [56048.470506] RBP: ffff94dcf5340000 R08: 0000000000000002 R09: 0000000000029600 [56048.470506] R10: 000060d29656a45c R11: ffff94dcf534fd12 R12: ffff94dcf5341db0 [56048.470507] R13: ffff94dcf5341dc0 R14: ffff94dcc4ae8a00 R15: 0000000000000003 [56048.470507] ? native_queued_spin_lock_slowpath+0x5d/0x1d0 [56048.470507] ? native_queued_spin_lock_slowpath+0x5d/0x1d0 [56048.470508] </NMI> [56048.470508] _raw_spin_lock_irqsave+0x32/0x40 [56048.470509] lpfc_dmp_dbg.part.32+0x28/0x220 [lpfc] [56048.470509] lpfc_cmpl_els_fdisc+0x145/0x460 [lpfc] [56048.470509] lpfc_sli_cancel_jobs+0x92/0xd0 [lpfc] [56048.470510] lpfc_els_flush_cmd+0x43c/0x670 [lpfc] [56048.470510] lpfc_els_flush_all_cmd+0x37/0x60 [lpfc] [56048.470510] lpfc_sli4_async_event_proc+0x956/0x1720 [lpfc] [56048.470511] lpfc_do_work+0x1485/0x1d70 [lpfc] [56048.470511] ? __schedule+0x280/0x700 [56048.470511] ? finish_wait+0x80/0x80 [56048.470512] ? lpfc_unregister_unused_fcf+0x80/0x80 [lpfc] [56048.470512] kthread+0x112/0x130 [56048.470513] ? kthread_flush_work_fn+0x10/0x10 [56048.470513] ret_from_fork+0x1f/0x40 [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.3 (Ootpa) [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# cat /sys/module/lpfc/version 0:14.0.390.2 [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# cat /sys/class/scsi_host/host*/modeldesc Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# cat /sys/class/scsi_host/host*/fwrev 14.0.390.1, sli-4:2:c 14.0.390.1, sli-4:2:c [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# cat /sys/class/fc_host/host*/port_name 0x10000090faf09459 0x10000090faf0945a [root@ms-svr3-10-231-131-160 127.0.0.1-2021-11-20-05:14:30]# HBA Attributes for 10:00:00:90:fa:f0:94:59 Host Name : ms-svr3-10-231-131-160 Manufacturer : Emulex Corporation Serial Number : FC70793283 Model : LPe32002-M2 Model Desc : Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter Node WWN : 20 00 00 90 fa f0 94 59 Node Symname : HW Version : 0000000c 00000001 00000000 FW Version : 14.0.390.1 Vendor Spec ID : 10DF Number of Ports : 1 Driver Name : lpfc Driver Version : 14.0.390.2; HBAAPI(I) v2.3.d, 07-12-10 Device ID : E300 HBA Type : LPe32002-M2 Operational FW : 14.0.390.1 IEEE Address : 00 90 fa f0 94 59 Boot Code : Enabled Boot Version : 14.0.390.1 Board Temperature : Normal Function Type : FC Sub Device ID : E300 PCI Bus Number : 94 PCI Func Number : 0 Sub Vendor ID : 10DF IPL Filename : H62LEX1 Service Processor FW Name : 14.0.390.1 ULP FW Name : 14.0.390.1 FC Universal BIOS Version : 14.0.390.1 FC x86 BIOS Version : 14.0.390.1 FC EFI BIOS Version : 14.0.388.0 FC FCODE Version : 14.0.386.0 Flash Firmware Version : 14.0.390.1 Secure Firmware : Enabled [root@ms-svr3-10-231-131-160 log]# hbacmd portattrib 10:00:00:90:fa:f0:94:59 Port Attributes for 10:00:00:90:fa:f0:94:59 Node WWN : 20 00 00 90 fa f0 94 59 Port WWN : 10 00 00 90 fa f0 94 59 Port Symname : Port FCID : 0000 Port Type : Unknown Port State : Link Down Port Service Type : 8 Port Supported FC4 : 00 00 01 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Port Active FC4 : 00 00 01 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Port Supported Speed : 8 16 32 Gbit/sec Configured Port Speed : Auto Detect Port Speed : Not Available Max Frame Size : 2048 OS Device Name : /sys/class/scsi_host/host15 Num Discovered Ports : 0 Fabric Name : 00 00 00 00 00 00 00 00 Function Type : FC FEC : Enabled [Fixes] The following patch will resolve the issue: scsi: lpfc: Move cfg_log_verbose check before calling lpfc_dmp_dbg() In an attempt to log message 0126 with LOG_TRACE_EVENT, the following hard lockup call trace hangs the system. [Testcase] [root@ms-svr3-10-231-131-160 log]# [reply] [-]Comment 3James Smart 2022-04-13 09:12:37 PDT Patches pushed upstream 4/12/22: https://lore.kernel.org/linux- scsi/20220412222008.126521-1-jsmart2...@gmail.com/T/#t To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1971193/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
