[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From sthou...@in.ibm.com 2019-10-01 02:41 EDT--- Kalpana, Please help here opening new bug. Add mranw...@us.ibm.com The lockup issue reported later by Michael will be tracked with the new bug and Michael will be able to test workaround mentioned in comment #75. We no longer tracking lockup issue with this bug. > Hi Seeteena, > > Shriya is no longer with IBM. So, who should be the contact person to open a > new bug? > But before that, we need to check, whether the workaround I mentioned in > comment #75 > addresses the issue or not. > > Thanks, > Anju -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1752002 Title: [P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is seen while running perf fuzzer (perf:) Status in The Ubuntu-power-systems project: Incomplete Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: Incomplete Bug description: == Comment: #0 - Shriya R. Kulkarni - 2018-02-02 01:21:36 == Problem Description : = Warn on message is seen while running perf fuzzer tests. Machine details : == Hardware : Witherspoon (wsp12) + DD2.1 OS : Ubuntu 1804 uname -a : 4.13.0-32-generic #35~lp1746225 ( Kernel from the bug : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 ) Steps to reproduce : Build Kernel : To avoid the kernel crash due to Perf fuzzer , use the kernel mentioned in the link : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 #! /bin/bash set -x git clone https://github.com/deater/perf_event_tests.git cd perf_event_tests/include mkdir asm cd asm wget http://9.114.13.132/repo/shriya/perf_regs.h cd ../../lib make sleep 10 cd ../fuzzer make sleep 10 echo 0 > /proc/sys/kernel/nmi_watchdog echo 2 > /proc/sys/kernel/perf_event_paranoid echo 10 > /proc/sys/kernel/perf_event_max_sample_rate ./perf_fuzzer -r 1492143527 Call trace : === [ 329.228031] [ cut here ] [ 329.228039] WARNING: CPU: 43 PID: 9088 at /home/jsalisbury/bugs/lp1746225/ubuntu-artful/kernel/events/core.c:3038 perf_pmu_sched_task+0x170/0x180 [ 329.228040] Modules linked in: ofpart at24 uio_pdrv_genirq uio cmdlinepart powernv_flash mtd ipmi_powernv vmx_crypto ipmi_devintf ipmi_msghandler ibmpowernv opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum lpfc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm tg3 nvmet_fc drm ahci nvmet nvme_fc libahci nvme_fabrics mlxfw nvme_core devlink scsi_transport_fc [ 329.228068] CPU: 43 PID: 9088 Comm: perf_fuzzer Not tainted 4.13.0-32-generic #35~lp1746225 [ 329.228070] task: c03f776ac900 task.stack: c03f77728000 [ 329.228071] NIP: c0299b70 LR: c02a4534 CTR: c029bb80 [ 329.228073] REGS: c03f7772b760 TRAP: 0700 Not tainted (4.13.0-32-generic) [ 329.228073] MSR: 9282b033 [ 329.228079] CR: 24008822 XER: [ 329.228080] CFAR: c0299a70 SOFTE: 0 GPR00: c02a4534 c03f7772b9e0 c1606200 c03fef858908 GPR04: c03f776ac900 0001 003fee73 GPR08: c11220d8 0002 GPR12: c029bb80 c7a3d900 GPR16: GPR20: c03f776ad090 c0c71354 GPR24: c03fef716780 003fee73 c03fe69d4200 c03f776ad330 GPR28: c11220d8 0001 c14c6108 c03fef858900 [ 329.228098] NIP [c0299b70] perf_pmu_sched_task+0x170/0x180 [ 329.228100] LR [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228101] Call Trace: [ 329.228102] [c03f7772b9e0] [c02a0678] perf_iterate_sb+0x158/0x2a0 (unreliable) [ 329.228105] [c03f7772ba30] [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228107] [c03f7772bab0] [c01396dc] finish_task_switch+0x21c/0x310 [ 329.228109] [c03f7772bb60] [c0c71354] __schedule+0x304/0xb80 [ 329.228111] [c03f7772bc40] [c0c71c10] schedule+0x40/0xc0 [ 329.228113] [c03f7772bc60] [c01033f4] do_wait+0x254/0x2e0 [ 329.228115] [c03f7772bcd0] [c0104ac0] kernel_wait4+0xa0/0x1a0 [ 329.228117] [c03f7772bd70] [c0104c24] SyS_wait4+0x64/0xc0 [ 329.228121] [c03f7772be30] [c000b184] system_call+0x58/0x6c [ 329.228121] Instruction dump: [ 329.228123] 3beafea0 7faa4800 409eff18 e801006
[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From kalsh...@in.ibm.com 2019-10-01 02:45 EDT--- (In reply to comment #83) > (In reply to comment #81) > > (In reply to comment #80) > > > > > Would it be possible for you to raise a new bug for the lockup issue > > > reported later by Michael? > > > > Shriya, > > > > Please open new bug based on the above > > > > > Thanks again, Andy. > > Hi Seeteena, > > Shriya is no longer with IBM. So, who should be the contact person to open a > new bug? > But before that, we need to check, whether the workaround I mentioned in > comment #75 > addresses the issue or not. > > Thanks, > Anju Sastry should be able to create a new defect based on devel suggestion. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1752002 Title: [P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is seen while running perf fuzzer (perf:) Status in The Ubuntu-power-systems project: Incomplete Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: Incomplete Bug description: == Comment: #0 - Shriya R. Kulkarni - 2018-02-02 01:21:36 == Problem Description : = Warn on message is seen while running perf fuzzer tests. Machine details : == Hardware : Witherspoon (wsp12) + DD2.1 OS : Ubuntu 1804 uname -a : 4.13.0-32-generic #35~lp1746225 ( Kernel from the bug : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 ) Steps to reproduce : Build Kernel : To avoid the kernel crash due to Perf fuzzer , use the kernel mentioned in the link : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 #! /bin/bash set -x git clone https://github.com/deater/perf_event_tests.git cd perf_event_tests/include mkdir asm cd asm wget http://9.114.13.132/repo/shriya/perf_regs.h cd ../../lib make sleep 10 cd ../fuzzer make sleep 10 echo 0 > /proc/sys/kernel/nmi_watchdog echo 2 > /proc/sys/kernel/perf_event_paranoid echo 10 > /proc/sys/kernel/perf_event_max_sample_rate ./perf_fuzzer -r 1492143527 Call trace : === [ 329.228031] [ cut here ] [ 329.228039] WARNING: CPU: 43 PID: 9088 at /home/jsalisbury/bugs/lp1746225/ubuntu-artful/kernel/events/core.c:3038 perf_pmu_sched_task+0x170/0x180 [ 329.228040] Modules linked in: ofpart at24 uio_pdrv_genirq uio cmdlinepart powernv_flash mtd ipmi_powernv vmx_crypto ipmi_devintf ipmi_msghandler ibmpowernv opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum lpfc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm tg3 nvmet_fc drm ahci nvmet nvme_fc libahci nvme_fabrics mlxfw nvme_core devlink scsi_transport_fc [ 329.228068] CPU: 43 PID: 9088 Comm: perf_fuzzer Not tainted 4.13.0-32-generic #35~lp1746225 [ 329.228070] task: c03f776ac900 task.stack: c03f77728000 [ 329.228071] NIP: c0299b70 LR: c02a4534 CTR: c029bb80 [ 329.228073] REGS: c03f7772b760 TRAP: 0700 Not tainted (4.13.0-32-generic) [ 329.228073] MSR: 9282b033 [ 329.228079] CR: 24008822 XER: [ 329.228080] CFAR: c0299a70 SOFTE: 0 GPR00: c02a4534 c03f7772b9e0 c1606200 c03fef858908 GPR04: c03f776ac900 0001 003fee73 GPR08: c11220d8 0002 GPR12: c029bb80 c7a3d900 GPR16: GPR20: c03f776ad090 c0c71354 GPR24: c03fef716780 003fee73 c03fe69d4200 c03f776ad330 GPR28: c11220d8 0001 c14c6108 c03fef858900 [ 329.228098] NIP [c0299b70] perf_pmu_sched_task+0x170/0x180 [ 329.228100] LR [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228101] Call Trace: [ 329.228102] [c03f7772b9e0] [c02a0678] perf_iterate_sb+0x158/0x2a0 (unreliable) [ 329.228105] [c03f7772ba30] [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228107] [c03f7772bab0] [c01396dc] finish_task_switch+0x21c/0x310 [ 329.228109] [c03f7772bb60] [c0c71354] __schedule+0x304/0xb80 [ 329.228111] [c03f7772bc40] [c0c71c10] schedule+0x40/0xc0 [ 329.228113] [c03f7772bc60] [c01033f4] do_wait+0x254/0x2e0 [ 329.228115] [c03f7772bcd0] [c0104ac0] kernel_wait4+0xa0/0x1a0 [ 329.228117] [c03f7772bd70] [c0104c24] SyS_wait4+0x64/0xc0 [ 329.228121] [c03f7772be30] [c000b184] system_call+
[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From anjutsudha...@in.ibm.com 2019-10-01 01:43 EDT--- (In reply to comment #81) > (In reply to comment #80) > > > Would it be possible for you to raise a new bug for the lockup issue > > reported later by Michael? > > Shriya, > > Please open new bug based on the above > > > Thanks again, Andy. Hi Seeteena, Shriya is no longer with IBM. So, who should be the contact person to open a new bug? But before that, we need to check, whether the workaround I mentioned in comment #75 addresses the issue or not. Thanks, Anju -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1752002 Title: [P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is seen while running perf fuzzer (perf:) Status in The Ubuntu-power-systems project: Incomplete Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: Incomplete Bug description: == Comment: #0 - Shriya R. Kulkarni - 2018-02-02 01:21:36 == Problem Description : = Warn on message is seen while running perf fuzzer tests. Machine details : == Hardware : Witherspoon (wsp12) + DD2.1 OS : Ubuntu 1804 uname -a : 4.13.0-32-generic #35~lp1746225 ( Kernel from the bug : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 ) Steps to reproduce : Build Kernel : To avoid the kernel crash due to Perf fuzzer , use the kernel mentioned in the link : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 #! /bin/bash set -x git clone https://github.com/deater/perf_event_tests.git cd perf_event_tests/include mkdir asm cd asm wget http://9.114.13.132/repo/shriya/perf_regs.h cd ../../lib make sleep 10 cd ../fuzzer make sleep 10 echo 0 > /proc/sys/kernel/nmi_watchdog echo 2 > /proc/sys/kernel/perf_event_paranoid echo 10 > /proc/sys/kernel/perf_event_max_sample_rate ./perf_fuzzer -r 1492143527 Call trace : === [ 329.228031] [ cut here ] [ 329.228039] WARNING: CPU: 43 PID: 9088 at /home/jsalisbury/bugs/lp1746225/ubuntu-artful/kernel/events/core.c:3038 perf_pmu_sched_task+0x170/0x180 [ 329.228040] Modules linked in: ofpart at24 uio_pdrv_genirq uio cmdlinepart powernv_flash mtd ipmi_powernv vmx_crypto ipmi_devintf ipmi_msghandler ibmpowernv opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum lpfc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm tg3 nvmet_fc drm ahci nvmet nvme_fc libahci nvme_fabrics mlxfw nvme_core devlink scsi_transport_fc [ 329.228068] CPU: 43 PID: 9088 Comm: perf_fuzzer Not tainted 4.13.0-32-generic #35~lp1746225 [ 329.228070] task: c03f776ac900 task.stack: c03f77728000 [ 329.228071] NIP: c0299b70 LR: c02a4534 CTR: c029bb80 [ 329.228073] REGS: c03f7772b760 TRAP: 0700 Not tainted (4.13.0-32-generic) [ 329.228073] MSR: 9282b033 [ 329.228079] CR: 24008822 XER: [ 329.228080] CFAR: c0299a70 SOFTE: 0 GPR00: c02a4534 c03f7772b9e0 c1606200 c03fef858908 GPR04: c03f776ac900 0001 003fee73 GPR08: c11220d8 0002 GPR12: c029bb80 c7a3d900 GPR16: GPR20: c03f776ad090 c0c71354 GPR24: c03fef716780 003fee73 c03fe69d4200 c03f776ad330 GPR28: c11220d8 0001 c14c6108 c03fef858900 [ 329.228098] NIP [c0299b70] perf_pmu_sched_task+0x170/0x180 [ 329.228100] LR [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228101] Call Trace: [ 329.228102] [c03f7772b9e0] [c02a0678] perf_iterate_sb+0x158/0x2a0 (unreliable) [ 329.228105] [c03f7772ba30] [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228107] [c03f7772bab0] [c01396dc] finish_task_switch+0x21c/0x310 [ 329.228109] [c03f7772bb60] [c0c71354] __schedule+0x304/0xb80 [ 329.228111] [c03f7772bc40] [c0c71c10] schedule+0x40/0xc0 [ 329.228113] [c03f7772bc60] [c01033f4] do_wait+0x254/0x2e0 [ 329.228115] [c03f7772bcd0] [c0104ac0] kernel_wait4+0xa0/0x1a0 [ 329.228117] [c03f7772bd70] [c0104c24] SyS_wait4+0x64/0xc0 [ 329.228121] [c03f7772be30] [c000b184] system_call+0x58/0x6c [ 329.228121] Instruction dump: [ 329.228123] 3beafea0 7faa4800 409eff18 e8010060 eb610028 ebc10040 7c0803a6 3821005
[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From anjutsudha...@in.ibm.com 2019-09-20 02:46 EDT--- (In reply to comment #78) > Anju , please comment on distro's question. Hi Seeteena, As per comment #58 and comment #59, the initial issue reported here can be resolved by adding those two patches. The lockup issue reported later by Michael, is not related to IMC. I have provided some workaround patches in comment #75, but I did not see any response for that. The lock up issue reported is something similar to https://bugzilla.linux.ibm.com/show_bug.cgi?id=161854 So in my opinion that should be tracked separately. The initial perf fuzzer call-trace with IMC is resolved here with the patches mentioned in comment #59. Thanks, Anju -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1752002 Title: [P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is seen while running perf fuzzer (perf:) Status in The Ubuntu-power-systems project: Incomplete Status in linux package in Ubuntu: In Progress Status in linux source package in Bionic: In Progress Bug description: == Comment: #0 - Shriya R. Kulkarni - 2018-02-02 01:21:36 == Problem Description : = Warn on message is seen while running perf fuzzer tests. Machine details : == Hardware : Witherspoon (wsp12) + DD2.1 OS : Ubuntu 1804 uname -a : 4.13.0-32-generic #35~lp1746225 ( Kernel from the bug : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 ) Steps to reproduce : Build Kernel : To avoid the kernel crash due to Perf fuzzer , use the kernel mentioned in the link : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 #! /bin/bash set -x git clone https://github.com/deater/perf_event_tests.git cd perf_event_tests/include mkdir asm cd asm wget http://9.114.13.132/repo/shriya/perf_regs.h cd ../../lib make sleep 10 cd ../fuzzer make sleep 10 echo 0 > /proc/sys/kernel/nmi_watchdog echo 2 > /proc/sys/kernel/perf_event_paranoid echo 10 > /proc/sys/kernel/perf_event_max_sample_rate ./perf_fuzzer -r 1492143527 Call trace : === [ 329.228031] [ cut here ] [ 329.228039] WARNING: CPU: 43 PID: 9088 at /home/jsalisbury/bugs/lp1746225/ubuntu-artful/kernel/events/core.c:3038 perf_pmu_sched_task+0x170/0x180 [ 329.228040] Modules linked in: ofpart at24 uio_pdrv_genirq uio cmdlinepart powernv_flash mtd ipmi_powernv vmx_crypto ipmi_devintf ipmi_msghandler ibmpowernv opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum lpfc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm tg3 nvmet_fc drm ahci nvmet nvme_fc libahci nvme_fabrics mlxfw nvme_core devlink scsi_transport_fc [ 329.228068] CPU: 43 PID: 9088 Comm: perf_fuzzer Not tainted 4.13.0-32-generic #35~lp1746225 [ 329.228070] task: c03f776ac900 task.stack: c03f77728000 [ 329.228071] NIP: c0299b70 LR: c02a4534 CTR: c029bb80 [ 329.228073] REGS: c03f7772b760 TRAP: 0700 Not tainted (4.13.0-32-generic) [ 329.228073] MSR: 9282b033 [ 329.228079] CR: 24008822 XER: [ 329.228080] CFAR: c0299a70 SOFTE: 0 GPR00: c02a4534 c03f7772b9e0 c1606200 c03fef858908 GPR04: c03f776ac900 0001 003fee73 GPR08: c11220d8 0002 GPR12: c029bb80 c7a3d900 GPR16: GPR20: c03f776ad090 c0c71354 GPR24: c03fef716780 003fee73 c03fe69d4200 c03f776ad330 GPR28: c11220d8 0001 c14c6108 c03fef858900 [ 329.228098] NIP [c0299b70] perf_pmu_sched_task+0x170/0x180 [ 329.228100] LR [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228101] Call Trace: [ 329.228102] [c03f7772b9e0] [c02a0678] perf_iterate_sb+0x158/0x2a0 (unreliable) [ 329.228105] [c03f7772ba30] [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228107] [c03f7772bab0] [c01396dc] finish_task_switch+0x21c/0x310 [ 329.228109] [c03f7772bb60] [c0c71354] __schedule+0x304/0xb80 [ 329.228111] [c03f7772bc40] [c0c71c10] schedule+0x40/0xc0 [ 329.228113] [c03f7772bc60] [c01033f4] do_wait+0x254/0x2e0 [ 329.228115] [c03f7772bcd0] [c0104ac0] kernel_wait4+0xa0/0x1a0 [ 329.228117] [c03f7772bd70] [c0104c24] SyS_wait4+0x64/0xc0 [ 329.228121] [c03
[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From mranw...@us.ibm.com 2019-07-15 09:59 EDT--- I added: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c9108b083706330cd5484d121fbb0ad67e8f647 in addition to: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?h=next&id=7ccc4fe5ff9e3a134e863beed0dba18a5e511659 It ran a lot longer - hour instead of minutes, but then ended up with this: [18734.191331] perf: interrupt took too long (1054 > 1051), lowering kernel.perf_event_max_sample_rate to 7500 [18736.617855] perf: Dynamic interrupt throttling disabled, can hang your system! [18751.191062] perf: interrupt took too long (2317 > 1333), lowering kernel.perf_event_max_sample_rate to 3250 [18753.006339] perf: interrupt took too long (2218 > 1), lowering kernel.perf_event_max_sample_rate to 3500 [18754.156398] perf: Dynamic interrupt throttling disabled, can hang your system! [18775.067223] perf: interrupt took too long (2227 > 1), lowering kernel.perf_event_max_sample_rate to 3500 [18779.532549] perf: Dynamic interrupt throttling disabled, can hang your system! [18834.315583] perf: Dynamic interrupt throttling disabled, can hang your system! [18851.090933] Watchdog CPU:102 Hard LOCKUP [18851.090936] Modules linked in: kvm_hv kvm vmx_crypto crct10dif_vpmsum ast drm_kms_helper ttm ofpart cmdlinepart drm fb_sys_fops ipmi_powernv at24 syscopyarea ipmi_devintf powernv_flash sysfillrect ipmi_msghandler opal_prd mtd ibmpowernv sysimgblt i2c_algo_bit uio_pdrv_genirq uio sch_fq_codel ip_tables x_tables autofs4 mlx5_core ahci mlxfw crc32c_vpmsum tg3 libahci devlink [18851.090995] CPU: 102 PID: 0 Comm: swapper/102 Tainted: G L 4.15.0-54-generic #58 [18851.090997] NIP: c0100740 LR: c010058c CTR: c00fe770 [18851.091000] REGS: c7ad3d80 TRAP: 0900 Tainted: G L (4.15.0-54-generic) [18851.091001] MSR: 90009033 CR: 28002882 XER: [18851.091014] CFAR: c000deb8 SOFTE: 0 GPR00: c0100584 c00020397b423850 c170b800 c03f80925000 GPR04: 7fff c00020397b4238b0 c000203994486958 c0002039944868b8 GPR08: 0004 0001 GPR12: c0002039944868f8 c7a56200 c00020397b423f90 GPR16: c004ad60 c004ad30 c11d5380 GPR20: 0800 c1742494 0066 0001 GPR24: 0198 8000 0006 GPR28: c18d1808 06004010 c0002039944868a0 [18851.091058] NIP [c0100740] power_pmu_enable+0x4f0/0x600 [18851.091060] LR [c010058c] power_pmu_enable+0x33c/0x600 [18851.091061] Call Trace: [18851.091065] [c00020397b423850] [c0100584] power_pmu_enable+0x334/0x600 (unreliable) [18851.091071] [c00020397b423930] [c02c9dbc] ctx_resched+0xec/0x150 [18851.091075] [c00020397b423970] [c02ca014] __perf_install_in_context+0x1f4/0x280 [18851.091079] [c00020397b4239c0] [c02bf7d0] remote_function+0x40/0x90 [18851.091083] [c00020397b4239f0] [c01db9dc] flush_smp_call_function_queue+0xac/0x1d0 [18851.091087] [c00020397b423a70] [c004bf7c] smp_ipi_demux_relaxed+0x9c/0x110 [18851.091092] [c00020397b423ab0] [c0047948] doorbell_exception+0xa8/0xe0 [18851.091096] [c00020397b423ae0] [c0009ad4] h_doorbell_common+0x114/0x120 [18851.091102] --- interrupt: e81 at replay_interrupt_return+0x0/0x4 LR = arch_local_irq_restore+0x74/0x90 [18851.091106] [c00020397b423dd0] [0040] 0x40 (unreliable) [18851.09] [c00020397b423df0] [c0acea80] cpuidle_enter_state+0xf0/0x450 [18851.091116] [c00020397b423e50] [c017852c] call_cpuidle+0x4c/0x90 [18851.091119] [c00020397b423e70] [c0178940] do_idle+0x2b0/0x330 [18851.091122] [c00020397b423ec0] [c0178bf8] cpu_startup_entry+0x38/0x40 [18851.091125] [c00020397b423ef0] [c004d280] start_secondary+0x4f0/0x510 [18851.091129] [c00020397b423f90] [c000ab6c] start_secondary_prolog+0x10/0x14 [18851.091131] Instruction dump: [18851.091135] 7fa9f000 419d001c 7d29c850 7d244b78 f93801b8 4bfffe44 6000 6000 [18851.091147] 3920 3880 f93801b8 4bfffe2c 4bfffb98 6000 6000 [18854.550395] Watchdog CPU:102 became unstuck [18873.055963] perf: interrupt took too long (329 > 1), lowering kernel.perf_event_max_sample_rate to 48500 [18873.965492] perf: interrupt took too long (420 > 411), lowering kernel.perf_event_max_sample_rate to 38000 [18898.926809] perf: Dynamic interrupt throttling disabled, can hang your system! [18933.376330] perf: interrupt took too long (1534 > 1481), lowering kernel.perf_event_max_sample_rate to 5000 [18942.748893] perf: interrupt took too long (1589 > 1), lowering kernel.perf_event_max_sample_rate to 5000 [18961.230616] perf: interrupt took too long (1190 > 1188), lowering kernel.perf_event_max_s
[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From mranw...@us.ibm.com 2019-06-24 04:52 EDT--- Anju, I'm trying this and seeing: == Starting fuzzing at 2019-06-24 03:26:26 == Cannot open /sys/kernel/tracing/kprobe_events SIGIO due to RT queue overflow Signal from invalid fd 10 Bad file descriptor Iteration 1, 139369 syscalls in 25.77 s (5.407 k syscalls/s) Open attempts: 131443 Successful: 918 Currently open: 5 ENOENT : 370 E2BIG : 11571 EBADF : 7775 EINVAL : 110627 ENOSPC : 31 EOVERFLOW : 1 EOPNOTSUPP : 150 Trinity Type (Normal 142/32892)(Sampling 11/32823)(Global 693/32727)(Random 72/33001) Type (Hardware 268/18287)(software 318/17277)(tracepoint 60/16952)(Cache 55/16464)(cpu 151/16831)(breakpoint 17/16793)(nest_alink0_imc 1/387)(nest_alink1_imc 2/353)(nest_alink2_imc 2/455)(nest_alink3_imc 0/380)(nest_capp0_imc 0/352)(nest_capp1_imc 0/385)(nest_centaur0_imc 0/381)(nest_centaur1_imc 0/384)(nest_centaur2_imc 2/339)(nest_centaur3_imc 1/380)(nest_centaur4_imc 0/482)(nest_centaur5_imc 2/378)(nest_centaur6_imc 0/344)(nest_centaur7_imc 39/23839) Close: 913/913 Successful Read: 821/892 Successful Write: 0/837 Successful Ioctl: 276/883 Successful: (ENABLE 60/60)(DISABLE 60/60)(REFRESH 4/57)(RESET 66/66)(PERIOD 9/58)(SET_OUTPUT 5/53)(SET_FILTER 0/68)(ID 63/63)(SET_BPF 0/57)(PAUSE_OUTPUT 9/63)(QUERY_BPF 0/71)(MOD_ATTR 0/71)(#12 0/0)(#13 0/0)(#14 0/0)(>14 0/136) Mmap: 661/1075 Successful: (MMAP 661/1075)(TRASH 119/144)(READ 127/134)(UNMAP 661/1050)(AUX 0/155)(AUX_READ 0/0) Prctl: 900/900 Successful Fork: 466/466 Successful Poll: 831/893 Successful Access: 329/918 Successful Overflows: 1972959 Recursive: 0 SIGIOs due to RT signal queue full: 1 Iteration 2, 140897 syscalls in 7.04 s (20.024 k syscalls/s) Open attempts: 133151 Successful: 901 Currently open: 24 ENOENT : 402 E2BIG : 11642 EBADF : 8002 EINVAL : 112005 ENOSPC : 33 EOVERFLOW : 1 EOPNOTSUPP : 165 Trinity Type (Normal 110/33454)(Sampling 15/33233)(Global 700/33200)(Random 76/33264) Type (Hardware 276/18590)(software 303/17115)(tracepoint 54/17217)(Cache 56/16759)(cpu 138/17185)(breakpoint 15/17011)(nest_alink0_imc 2/352)(nest_alink1_imc 3/370)(nest_alink2_imc 0/498)(nest_alink3_imc 3/382)(nest_capp0_imc 1/378)(nest_capp1_imc 1/359)(nest_centaur0_imc 1/364)(nest_centaur1_imc 0/363)(nest_centaur2_imc 1/376)(nest_centaur3_imc 2/356)(nest_centaur4_imc 1/483)(nest_centaur5_imc 1/369)(nest_centaur6_imc 0/385)(nest_centaur7_imc 43/24239) Close: 882/882 Successful Read: 790/870 Successful Write: 0/857 Successful Ioctl: 266/896 Successful: (ENABLE 63/63)(DISABLE 59/59)(REFRESH 6/74)(RESET 69/69)(PERIOD 5/68)(SET_OUTPUT 5/76)(SET_FILTER 0/68)(ID 49/49)(SET_BPF 0/67)(PAUSE_OUTPUT 10/60)(QUERY_BPF 0/64)(MOD_ATTR 0/59)(#12 0/0)(#13 0/0)(#14 0/0)(>14 0/120) Mmap: 623/1028 Successful: (MMAP 623/1028)(TRASH 121/147)(READ 118/124)(UNMAP 618/1004)(AUX 0/135)(AUX_READ 0/0) Prctl: 868/868 Successful Fork: 452/452 Successful Poll: 791/865 Successful Access: 332/926 Successful Overflows: 0 Recursive: 0 SIGIOs due to RT signal queue full: 0 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 0 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 1 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 2 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 3 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 4 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 5 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 6 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 7 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 8 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 9 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 10 Stuck in a signal storm w/o forward progress; Max throttle count hit, giving up And the end of dmesg and the (-proposed) kernel version: [ 374.873933] perf: Dynamic interrupt throttling disabled, can hang your system! [ 398.775635] perf: Dynamic interrupt throttling disabled, can hang your system! [ 401.495532] perf: Dynamic interrupt throttling disabled, can hang your system! ubuntu@ltc-wspoon5:~$ cat /proc/version Linux version 4.15.0-51-generic (arighi@kathleen) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #55 SMP Wed Jun 5 14:15:11 UTC 2019 ubuntu@ltc-wspoon5:~$ I don't see similar on the -50 kernel. This might be unrelated, but you have ideas? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1752002 Title: [P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is seen while running perf fuzzer (perf:) Status in The Ubuntu-power-systems project: In Progress
[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From mranw...@us.ibm.com 2019-06-10 17:45 EDT--- Anju - I tried this with both the 4.15.0-50 kernel (that has the first patch) and the test kernel in the above comment and I see the same behavior. No call trace, but on both the test ends: == Starting fuzzing at 2019-06-10 16:35:58 == Watchdog triggered; failed to progress for 60 seconds; killing Trying to shut ourselves down: 4986, last child 0 This is on a dd 2.3 witherspoon. Any ideas? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1752002 Title: [P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is seen while running perf fuzzer (perf:) Status in The Ubuntu-power-systems project: Incomplete Status in linux package in Ubuntu: In Progress Status in linux source package in Bionic: In Progress Bug description: == Comment: #0 - Shriya R. Kulkarni - 2018-02-02 01:21:36 == Problem Description : = Warn on message is seen while running perf fuzzer tests. Machine details : == Hardware : Witherspoon (wsp12) + DD2.1 OS : Ubuntu 1804 uname -a : 4.13.0-32-generic #35~lp1746225 ( Kernel from the bug : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 ) Steps to reproduce : Build Kernel : To avoid the kernel crash due to Perf fuzzer , use the kernel mentioned in the link : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 #! /bin/bash set -x git clone https://github.com/deater/perf_event_tests.git cd perf_event_tests/include mkdir asm cd asm wget http://9.114.13.132/repo/shriya/perf_regs.h cd ../../lib make sleep 10 cd ../fuzzer make sleep 10 echo 0 > /proc/sys/kernel/nmi_watchdog echo 2 > /proc/sys/kernel/perf_event_paranoid echo 10 > /proc/sys/kernel/perf_event_max_sample_rate ./perf_fuzzer -r 1492143527 Call trace : === [ 329.228031] [ cut here ] [ 329.228039] WARNING: CPU: 43 PID: 9088 at /home/jsalisbury/bugs/lp1746225/ubuntu-artful/kernel/events/core.c:3038 perf_pmu_sched_task+0x170/0x180 [ 329.228040] Modules linked in: ofpart at24 uio_pdrv_genirq uio cmdlinepart powernv_flash mtd ipmi_powernv vmx_crypto ipmi_devintf ipmi_msghandler ibmpowernv opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum lpfc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm tg3 nvmet_fc drm ahci nvmet nvme_fc libahci nvme_fabrics mlxfw nvme_core devlink scsi_transport_fc [ 329.228068] CPU: 43 PID: 9088 Comm: perf_fuzzer Not tainted 4.13.0-32-generic #35~lp1746225 [ 329.228070] task: c03f776ac900 task.stack: c03f77728000 [ 329.228071] NIP: c0299b70 LR: c02a4534 CTR: c029bb80 [ 329.228073] REGS: c03f7772b760 TRAP: 0700 Not tainted (4.13.0-32-generic) [ 329.228073] MSR: 9282b033 [ 329.228079] CR: 24008822 XER: [ 329.228080] CFAR: c0299a70 SOFTE: 0 GPR00: c02a4534 c03f7772b9e0 c1606200 c03fef858908 GPR04: c03f776ac900 0001 003fee73 GPR08: c11220d8 0002 GPR12: c029bb80 c7a3d900 GPR16: GPR20: c03f776ad090 c0c71354 GPR24: c03fef716780 003fee73 c03fe69d4200 c03f776ad330 GPR28: c11220d8 0001 c14c6108 c03fef858900 [ 329.228098] NIP [c0299b70] perf_pmu_sched_task+0x170/0x180 [ 329.228100] LR [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228101] Call Trace: [ 329.228102] [c03f7772b9e0] [c02a0678] perf_iterate_sb+0x158/0x2a0 (unreliable) [ 329.228105] [c03f7772ba30] [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228107] [c03f7772bab0] [c01396dc] finish_task_switch+0x21c/0x310 [ 329.228109] [c03f7772bb60] [c0c71354] __schedule+0x304/0xb80 [ 329.228111] [c03f7772bc40] [c0c71c10] schedule+0x40/0xc0 [ 329.228113] [c03f7772bc60] [c01033f4] do_wait+0x254/0x2e0 [ 329.228115] [c03f7772bcd0] [c0104ac0] kernel_wait4+0xa0/0x1a0 [ 329.228117] [c03f7772bd70] [c0104c24] SyS_wait4+0x64/0xc0 [ 329.228121] [c03f7772be30] [c000b184] system_call+0x58/0x6c [ 329.228121] Instruction dump: [ 329.228123] 3beafea0 7faa4800 409eff18 e8010060 eb610028 ebc100
[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From sthou...@in.ibm.com 2019-06-04 06:40 EDT--- Canonical, Any update -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1752002 Title: [P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is seen while running perf fuzzer (perf:) Status in The Ubuntu-power-systems project: Fix Released Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Released Bug description: == Comment: #0 - Shriya R. Kulkarni - 2018-02-02 01:21:36 == Problem Description : = Warn on message is seen while running perf fuzzer tests. Machine details : == Hardware : Witherspoon (wsp12) + DD2.1 OS : Ubuntu 1804 uname -a : 4.13.0-32-generic #35~lp1746225 ( Kernel from the bug : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 ) Steps to reproduce : Build Kernel : To avoid the kernel crash due to Perf fuzzer , use the kernel mentioned in the link : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 #! /bin/bash set -x git clone https://github.com/deater/perf_event_tests.git cd perf_event_tests/include mkdir asm cd asm wget http://9.114.13.132/repo/shriya/perf_regs.h cd ../../lib make sleep 10 cd ../fuzzer make sleep 10 echo 0 > /proc/sys/kernel/nmi_watchdog echo 2 > /proc/sys/kernel/perf_event_paranoid echo 10 > /proc/sys/kernel/perf_event_max_sample_rate ./perf_fuzzer -r 1492143527 Call trace : === [ 329.228031] [ cut here ] [ 329.228039] WARNING: CPU: 43 PID: 9088 at /home/jsalisbury/bugs/lp1746225/ubuntu-artful/kernel/events/core.c:3038 perf_pmu_sched_task+0x170/0x180 [ 329.228040] Modules linked in: ofpart at24 uio_pdrv_genirq uio cmdlinepart powernv_flash mtd ipmi_powernv vmx_crypto ipmi_devintf ipmi_msghandler ibmpowernv opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum lpfc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm tg3 nvmet_fc drm ahci nvmet nvme_fc libahci nvme_fabrics mlxfw nvme_core devlink scsi_transport_fc [ 329.228068] CPU: 43 PID: 9088 Comm: perf_fuzzer Not tainted 4.13.0-32-generic #35~lp1746225 [ 329.228070] task: c03f776ac900 task.stack: c03f77728000 [ 329.228071] NIP: c0299b70 LR: c02a4534 CTR: c029bb80 [ 329.228073] REGS: c03f7772b760 TRAP: 0700 Not tainted (4.13.0-32-generic) [ 329.228073] MSR: 9282b033 [ 329.228079] CR: 24008822 XER: [ 329.228080] CFAR: c0299a70 SOFTE: 0 GPR00: c02a4534 c03f7772b9e0 c1606200 c03fef858908 GPR04: c03f776ac900 0001 003fee73 GPR08: c11220d8 0002 GPR12: c029bb80 c7a3d900 GPR16: GPR20: c03f776ad090 c0c71354 GPR24: c03fef716780 003fee73 c03fe69d4200 c03f776ad330 GPR28: c11220d8 0001 c14c6108 c03fef858900 [ 329.228098] NIP [c0299b70] perf_pmu_sched_task+0x170/0x180 [ 329.228100] LR [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228101] Call Trace: [ 329.228102] [c03f7772b9e0] [c02a0678] perf_iterate_sb+0x158/0x2a0 (unreliable) [ 329.228105] [c03f7772ba30] [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228107] [c03f7772bab0] [c01396dc] finish_task_switch+0x21c/0x310 [ 329.228109] [c03f7772bb60] [c0c71354] __schedule+0x304/0xb80 [ 329.228111] [c03f7772bc40] [c0c71c10] schedule+0x40/0xc0 [ 329.228113] [c03f7772bc60] [c01033f4] do_wait+0x254/0x2e0 [ 329.228115] [c03f7772bcd0] [c0104ac0] kernel_wait4+0xa0/0x1a0 [ 329.228117] [c03f7772bd70] [c0104c24] SyS_wait4+0x64/0xc0 [ 329.228121] [c03f7772be30] [c000b184] system_call+0x58/0x6c [ 329.228121] Instruction dump: [ 329.228123] 3beafea0 7faa4800 409eff18 e8010060 eb610028 ebc10040 7c0803a6 38210050 [ 329.228127] eb81ffe0 eba1ffe8 ebe1fff8 4e800020 <0fe0> 4bbc 6000 6042 [ 329.228131] ---[ end trace 8c46856d314c1811 ]--- [ 375.755943] hrtimer: interrupt took 31601 ns == Comment: #4 - SEETEENA THOUFEEK - 2018-02-05 06:34:09 == == Comment: #5 - SEETEENA THOUFEEK - 2018-02-05 06:36:12 == We have similar issue reported on different distro where Anju Provided the patch. Patch attached above. .
[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From sthou...@in.ibm.com 2019-05-07 02:33 EDT--- Anju , please check if both patches are needed to fix this issue. --- Comment From sthou...@in.ibm.com 2019-05-07 04:59 EDT--- (In reply to comment #58) > (In reply to comment #57) > > Anju , please check if both patches are needed to fix this issue. > > yes, both may be needed, as the grouping of events with perf-fuzzer can > happen in any way. Based on the developer response both the patches are needed to fix this issue. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5aa04b3eb6fca63d2e9827be656dcadc26d54e11 https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?h=next&id=7ccc4fe5ff9e3a134e863beed0dba18a5e511659 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1752002 Title: [P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is seen while running perf fuzzer (perf:) Status in The Ubuntu-power-systems project: Fix Released Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Released Bug description: == Comment: #0 - Shriya R. Kulkarni - 2018-02-02 01:21:36 == Problem Description : = Warn on message is seen while running perf fuzzer tests. Machine details : == Hardware : Witherspoon (wsp12) + DD2.1 OS : Ubuntu 1804 uname -a : 4.13.0-32-generic #35~lp1746225 ( Kernel from the bug : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 ) Steps to reproduce : Build Kernel : To avoid the kernel crash due to Perf fuzzer , use the kernel mentioned in the link : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 #! /bin/bash set -x git clone https://github.com/deater/perf_event_tests.git cd perf_event_tests/include mkdir asm cd asm wget http://9.114.13.132/repo/shriya/perf_regs.h cd ../../lib make sleep 10 cd ../fuzzer make sleep 10 echo 0 > /proc/sys/kernel/nmi_watchdog echo 2 > /proc/sys/kernel/perf_event_paranoid echo 10 > /proc/sys/kernel/perf_event_max_sample_rate ./perf_fuzzer -r 1492143527 Call trace : === [ 329.228031] [ cut here ] [ 329.228039] WARNING: CPU: 43 PID: 9088 at /home/jsalisbury/bugs/lp1746225/ubuntu-artful/kernel/events/core.c:3038 perf_pmu_sched_task+0x170/0x180 [ 329.228040] Modules linked in: ofpart at24 uio_pdrv_genirq uio cmdlinepart powernv_flash mtd ipmi_powernv vmx_crypto ipmi_devintf ipmi_msghandler ibmpowernv opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum lpfc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm tg3 nvmet_fc drm ahci nvmet nvme_fc libahci nvme_fabrics mlxfw nvme_core devlink scsi_transport_fc [ 329.228068] CPU: 43 PID: 9088 Comm: perf_fuzzer Not tainted 4.13.0-32-generic #35~lp1746225 [ 329.228070] task: c03f776ac900 task.stack: c03f77728000 [ 329.228071] NIP: c0299b70 LR: c02a4534 CTR: c029bb80 [ 329.228073] REGS: c03f7772b760 TRAP: 0700 Not tainted (4.13.0-32-generic) [ 329.228073] MSR: 9282b033 [ 329.228079] CR: 24008822 XER: [ 329.228080] CFAR: c0299a70 SOFTE: 0 GPR00: c02a4534 c03f7772b9e0 c1606200 c03fef858908 GPR04: c03f776ac900 0001 003fee73 GPR08: c11220d8 0002 GPR12: c029bb80 c7a3d900 GPR16: GPR20: c03f776ad090 c0c71354 GPR24: c03fef716780 003fee73 c03fe69d4200 c03f776ad330 GPR28: c11220d8 0001 c14c6108 c03fef858900 [ 329.228098] NIP [c0299b70] perf_pmu_sched_task+0x170/0x180 [ 329.228100] LR [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228101] Call Trace: [ 329.228102] [c03f7772b9e0] [c02a0678] perf_iterate_sb+0x158/0x2a0 (unreliable) [ 329.228105] [c03f7772ba30] [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228107] [c03f7772bab0] [c01396dc] finish_task_switch+0x21c/0x310 [ 329.228109] [c03f7772bb60] [c0c71354] __schedule+0x304/0xb80 [ 329.228111] [c03f7772bc40] [c0c71c10] schedule+0x40/0xc0 [ 329.228113] [c03f7772bc60] [c01033f4] do_wait+0x254/0x2e0 [ 329.228115] [c03f7772bcd0] [c0104ac0] kernel_wait4+0xa0/0x1a0 [ 329.228117] [c03f7772bd70] [c0104c24] SyS_wait
[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From anjutsudha...@in.ibm.com 2018-09-27 07:16 EDT--- > Anju: What is the status on upstream review on your fixed code? Hi, Patch is now upstream, https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?h=next&id=7ccc4fe5ff9e3a134e863beed0dba18a5e511659 Thanks, Anju. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1752002 Title: [P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is seen while running perf fuzzer (perf:) Status in The Ubuntu-power-systems project: Fix Released Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Released Bug description: == Comment: #0 - Shriya R. Kulkarni - 2018-02-02 01:21:36 == Problem Description : = Warn on message is seen while running perf fuzzer tests. Machine details : == Hardware : Witherspoon (wsp12) + DD2.1 OS : Ubuntu 1804 uname -a : 4.13.0-32-generic #35~lp1746225 ( Kernel from the bug : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 ) Steps to reproduce : Build Kernel : To avoid the kernel crash due to Perf fuzzer , use the kernel mentioned in the link : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 #! /bin/bash set -x git clone https://github.com/deater/perf_event_tests.git cd perf_event_tests/include mkdir asm cd asm wget http://9.114.13.132/repo/shriya/perf_regs.h cd ../../lib make sleep 10 cd ../fuzzer make sleep 10 echo 0 > /proc/sys/kernel/nmi_watchdog echo 2 > /proc/sys/kernel/perf_event_paranoid echo 10 > /proc/sys/kernel/perf_event_max_sample_rate ./perf_fuzzer -r 1492143527 Call trace : === [ 329.228031] [ cut here ] [ 329.228039] WARNING: CPU: 43 PID: 9088 at /home/jsalisbury/bugs/lp1746225/ubuntu-artful/kernel/events/core.c:3038 perf_pmu_sched_task+0x170/0x180 [ 329.228040] Modules linked in: ofpart at24 uio_pdrv_genirq uio cmdlinepart powernv_flash mtd ipmi_powernv vmx_crypto ipmi_devintf ipmi_msghandler ibmpowernv opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum lpfc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm tg3 nvmet_fc drm ahci nvmet nvme_fc libahci nvme_fabrics mlxfw nvme_core devlink scsi_transport_fc [ 329.228068] CPU: 43 PID: 9088 Comm: perf_fuzzer Not tainted 4.13.0-32-generic #35~lp1746225 [ 329.228070] task: c03f776ac900 task.stack: c03f77728000 [ 329.228071] NIP: c0299b70 LR: c02a4534 CTR: c029bb80 [ 329.228073] REGS: c03f7772b760 TRAP: 0700 Not tainted (4.13.0-32-generic) [ 329.228073] MSR: 9282b033 [ 329.228079] CR: 24008822 XER: [ 329.228080] CFAR: c0299a70 SOFTE: 0 GPR00: c02a4534 c03f7772b9e0 c1606200 c03fef858908 GPR04: c03f776ac900 0001 003fee73 GPR08: c11220d8 0002 GPR12: c029bb80 c7a3d900 GPR16: GPR20: c03f776ad090 c0c71354 GPR24: c03fef716780 003fee73 c03fe69d4200 c03f776ad330 GPR28: c11220d8 0001 c14c6108 c03fef858900 [ 329.228098] NIP [c0299b70] perf_pmu_sched_task+0x170/0x180 [ 329.228100] LR [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228101] Call Trace: [ 329.228102] [c03f7772b9e0] [c02a0678] perf_iterate_sb+0x158/0x2a0 (unreliable) [ 329.228105] [c03f7772ba30] [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228107] [c03f7772bab0] [c01396dc] finish_task_switch+0x21c/0x310 [ 329.228109] [c03f7772bb60] [c0c71354] __schedule+0x304/0xb80 [ 329.228111] [c03f7772bc40] [c0c71c10] schedule+0x40/0xc0 [ 329.228113] [c03f7772bc60] [c01033f4] do_wait+0x254/0x2e0 [ 329.228115] [c03f7772bcd0] [c0104ac0] kernel_wait4+0xa0/0x1a0 [ 329.228117] [c03f7772bd70] [c0104c24] SyS_wait4+0x64/0xc0 [ 329.228121] [c03f7772be30] [c000b184] system_call+0x58/0x6c [ 329.228121] Instruction dump: [ 329.228123] 3beafea0 7faa4800 409eff18 e8010060 eb610028 ebc10040 7c0803a6 38210050 [ 329.228127] eb81ffe0 eba1ffe8 ebe1fff8 4e800020 <0fe0> 4bbc 6000 6042 [ 329.228131] ---[ end trace 8c46856d314c1811 ]--- [ 375.755943] hrtimer: interrupt took 31601 ns == Comment: #4 - SEETEENA TH
[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From shriy...@in.ibm.com 2018-05-17 03:04 EDT---
Hi Anju ,
The above test kernel add issue while setting up, so rebuilt the
upstream kernel with the patches mentioned above.
=> Perf fuzzer works fine and issue is not seen.
=> Thread IMC also works fine.
root@ltc-wspoon4:/usr/lib/linux-tools-4.15.0-20# ./perf stat -e
'{emulation-faults, thread_imc/CPM_CCYC/, thread_imc/CPM_CS_32MHZ_CYC/,
thread_imc/CPM_CS_BRU_CMPL_KERNEL/}'yes > /dev/null
^Cyes: Interrupt
Performance counter stats for 'yes':
0 emulation-faults
1,509,903,360 thread_imc/CPM_CCYC/
1,493,126,656 thread_imc/CPM_CS_32MHZ_CYC/
0 thread_imc/CPM_CS_BRU_CMPL_KERNEL/
35.697528468 seconds time elapsed
Machine : Witherspoon + DD2.2
uname -r : 4.17.0-rc5
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1752002
Title:
[P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is
seen while running perf fuzzer (perf:)
Status in The Ubuntu-power-systems project:
Incomplete
Status in linux package in Ubuntu:
Triaged
Status in linux source package in Bionic:
Triaged
Bug description:
== Comment: #0 - Shriya R. Kulkarni - 2018-02-02
01:21:36 ==
Problem Description :
=
Warn on message is seen while running perf fuzzer tests.
Machine details :
==
Hardware : Witherspoon (wsp12) + DD2.1
OS : Ubuntu 1804
uname -a : 4.13.0-32-generic #35~lp1746225 ( Kernel from the bug :
https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 )
Steps to reproduce :
Build Kernel :
To avoid the kernel crash due to Perf fuzzer , use the kernel mentioned in
the link : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7
#! /bin/bash
set -x
git clone https://github.com/deater/perf_event_tests.git
cd perf_event_tests/include
mkdir asm
cd asm
wget http://9.114.13.132/repo/shriya/perf_regs.h
cd ../../lib
make
sleep 10
cd ../fuzzer
make
sleep 10
echo 0 > /proc/sys/kernel/nmi_watchdog
echo 2 > /proc/sys/kernel/perf_event_paranoid
echo 10 > /proc/sys/kernel/perf_event_max_sample_rate
./perf_fuzzer -r 1492143527
Call trace :
===
[ 329.228031] [ cut here ]
[ 329.228039] WARNING: CPU: 43 PID: 9088 at
/home/jsalisbury/bugs/lp1746225/ubuntu-artful/kernel/events/core.c:3038
perf_pmu_sched_task+0x170/0x180
[ 329.228040] Modules linked in: ofpart at24 uio_pdrv_genirq uio cmdlinepart
powernv_flash mtd ipmi_powernv vmx_crypto ipmi_devintf ipmi_msghandler
ibmpowernv opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4
crc32c_vpmsum lpfc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect
sysimgblt mlx5_core fb_sys_fops ttm tg3 nvmet_fc drm ahci nvmet nvme_fc libahci
nvme_fabrics mlxfw nvme_core devlink scsi_transport_fc
[ 329.228068] CPU: 43 PID: 9088 Comm: perf_fuzzer Not tainted
4.13.0-32-generic #35~lp1746225
[ 329.228070] task: c03f776ac900 task.stack: c03f77728000
[ 329.228071] NIP: c0299b70 LR: c02a4534 CTR:
c029bb80
[ 329.228073] REGS: c03f7772b760 TRAP: 0700 Not tainted
(4.13.0-32-generic)
[ 329.228073] MSR: 9282b033
[ 329.228079] CR: 24008822 XER:
[ 329.228080] CFAR: c0299a70 SOFTE: 0
GPR00: c02a4534 c03f7772b9e0 c1606200
c03fef858908
GPR04: c03f776ac900 0001
003fee73
GPR08: c11220d8
0002
GPR12: c029bb80 c7a3d900
GPR16:
GPR20: c03f776ad090
c0c71354
GPR24: c03fef716780 003fee73 c03fe69d4200
c03f776ad330
GPR28: c11220d8 0001 c14c6108
c03fef858900
[ 329.228098] NIP [c0299b70] perf_pmu_sched_task+0x170/0x180
[ 329.228100] LR [c02a4534] __perf_event_task_sched_in+0xc4/0x230
[ 329.228101] Call Trace:
[ 329.228102] [c03f7772b9e0] [c02a0678]
perf_iterate_sb+0x158/0x2a0 (unreliable)
[ 329.228105] [c03f7772ba30] [c02a4534]
__perf_event_task_sched_in+0xc4/0x230
[ 329.228107] [c03f7772bab0] [c01396dc]
finish_task_switch+0x21c/0x310
[ 329.228109] [c03f7772bb60] [c0c71354] __schedule+0x304/0xb80
[ 329.228111] [c03f7772bc40] [c0c71c10] schedule+0x40/0xc0
[ 329.228113] [c03f7772bc60] [c01033f4] do_wait+0x254/0x2e0
[ 329.228115] [c03f7772bcd0] [c0104ac0] kernel_wait4+0xa0/0x1a0
[ 329.228117] [c03f7772bd70] [c0104c24]
[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From shriy...@in.ibm.com 2018-04-30 04:24 EDT--- Verified with test kernel : Issue is resolved. === But hitting another trace : [ 294.764782] perf: Dynamic interrupt throttling disabled, can hang your system! [ 315.685952] perf: Dynamic interrupt throttling disabled, can hang your system! [ 317.385747] perf: Dynamic interrupt throttling disabled, can hang your system! [ 335.030061] hrtimer: interrupt took 1494725987 ns [ 386.576484] perf: Dynamic interrupt throttling disabled, can hang your system! [ 403.964295] perf: Dynamic interrupt throttling disabled, can hang your system! [ 414.884012] perf: Dynamic interrupt throttling disabled, can hang your system! [ 431.700329] perf: Dynamic interrupt throttling disabled, can hang your system! [ 471.108095] INFO: rcu_sched self-detected stall on CPU [ 471.108214] 116-: (5250 ticks this GP) idle=c9a/142/0 softirq=6343/6344 fqs=2625 [ 471.108351] (t=5251 jiffies g=8835 c=8834 q=1160) [ 471.108508] Task dump for CPU 116: [ 471.108518] perf_fuzzer R running task0 5428 5267 0x0004a006 [ 471.108549] Call Trace: [ 471.108582] [c0002038e74231c0] [c0149e98] sched_show_task.part.16+0xd8/0x110 (unreliable) [ 471.108627] [c0002038e7423230] [c01a9e5c] rcu_dump_cpu_stacks+0xd4/0x138 [ 471.108664] [c0002038e7423280] [c01a8f28] rcu_check_callbacks+0x8e8/0xb40 [ 471.108698] [c0002038e74233b0] [c01b71c8] update_process_times+0x48/0x90 [ 471.108731] [c0002038e74233e0] [c01cef14] tick_sched_handle.isra.5+0x34/0xd0 [ 471.108760] [c0002038e7423410] [c01cf010] tick_sched_timer+0x60/0xe0 [ 471.108795] [c0002038e7423450] [c01b7d74] __hrtimer_run_queues+0x144/0x370 [ 471.108830] [c0002038e74234d0] [c01b8ccc] hrtimer_interrupt+0xfc/0x350 [ 471.108867] [c0002038e74235a0] [c00248f0] __timer_interrupt+0x90/0x260 [ 471.108903] [c0002038e74235f0] [c0024d08] timer_interrupt+0x98/0xe0 [ 471.108943] [c0002038e7423620] [c000b998] fast_exception_return+0x148/0x16c [ 471.108990] --- interrupt: 901 at arch_local_irq_restore+0x84/0x90 LR = __do_softirq+0xd8/0x3e4 [ 471.109017] [c0002038e7423910] [c01b8d60] hrtimer_interrupt+0x190/0x350 (unreliable) [ 471.109054] [c0002038e7423930] [c0cffbc8] __do_softirq+0xd8/0x3e4 [ 471.109089] [c0002038e7423a10] [c0115928] irq_exit+0xe8/0x120 [ 471.109124] [c0002038e7423a30] [c0024d0c] timer_interrupt+0x9c/0xe0 [ 471.109164] [c0002038e7423a60] [c000b998] fast_exception_return+0x148/0x16c [ 471.109211] --- interrupt: 901 at mutex_unlock+0x18/0x50 LR = perf_event_for_each_child+0xb0/0xf0 [ 471.109236] [c0002038e7423d50] [c02b9e70] perf_event_for_each_child+0x60/0xf0 (unreliable) [ 471.109279] [c0002038e7423d90] [c02c4da8] perf_event_task_enable+0x78/0xe0 [ 471.109309] [c0002038e7423dd0] [c012d4e4] SyS_prctl+0x364/0x6a0 [ 471.109345] [c0002038e7423e30] [c000b184] system_call+0x58/0x6c [ 477.935937] watchdog: BUG: soft lockup - CPU#116 stuck for 23s! [perf_fuzzer:5428] [ 477.936042] Modules linked in: xt_CHECKSUM(E) iptable_mangle(E) ipt_MASQUERADE(E) nf_nat_masquerade_ipv4(E) iptable_nat(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack_ipv4(E) nf_defrag_ipv4(E) xt_conntrack(E) nf_conntrack(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_tcpudp(E) bridge(E) stp(E) llc(E) ebtable_filter(E) ebtables(E) ip6table_filter(E) ip6_tables(E) iptable_filter(E) kvm_hv(E) kvm(E) at24(E) ofpart(E) ipmi_powernv(E) ipmi_devintf(E) ipmi_msghandler(E) uio_pdrv_genirq(E) uio(E) cmdlinepart(E) powernv_flash(E) mtd(E) ibmpowernv(E) opal_prd(E) vmx_crypto(E) sch_fq_codel(E) ib_iser(E) rdma_cm(E) iw_cm(E) ib_cm(E) ib_core(E) iscsi_tcp(E) libiscsi_tcp(E) libiscsi(E) scsi_transport_iscsi(E) ip_tables(E) x_tables(E) autofs4(E) btrfs(E) zstd_compress(E) raid10(E) raid456(E) async_raid6_recov(E) async_memcpy(E) [ 477.936571] async_pq(E) async_xor(E) async_tx(E) xor(E) raid6_pq(E) libcrc32c(E) raid1(E) raid0(E) multipath(E) linear(E) ast(E) mlx5_core(E) i2c_algo_bit(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) crct10dif_vpmsum(E) ahci(E) mlxfw(E) crc32c_vpmsum(E) drm(E) tg3(E) libahci(E) devlink(E) [ 477.936833] CPU: 116 PID: 5428 Comm: perf_fuzzer Tainted: GE 4.15.0-20-generic #21 [ 477.936850] NIP: c0016e84 LR: c0cffbc8 CTR: c0024480 [ 477.936870] REGS: c0002038e7423690 TRAP: 0901 Tainted: GE (4.15.0-20-generic) [ 477.936879] MSR: 90009033 CR: 48000244 XER: 2004 [ 477.936970] CFAR: c0016e30 SOFTE: 1 GPR00: c0cffbc8 c0002038e7423910 c16eae00 0001 GPR04: c000203994800400 01f3f3f9 c0002038e7388a00 GPR08: 20399365 b0001033 0008 0005 GPR12: c0024480 c7a6fc00 [ 477.937161] NIP [c
[Kernel-packages] [Bug 1752002] Comment bridged from LTC Bugzilla
--- Comment From kalsh...@in.ibm.com 2018-03-19 10:34 EDT--- What's latest on this bug? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1752002 Title: [P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is seen while running perf fuzzer (perf:) Status in The Ubuntu-power-systems project: Incomplete Status in linux package in Ubuntu: Triaged Status in linux source package in Bionic: Triaged Bug description: == Comment: #0 - Shriya R. Kulkarni - 2018-02-02 01:21:36 == Problem Description : = Warn on message is seen while running perf fuzzer tests. Machine details : == Hardware : Witherspoon (wsp12) + DD2.1 OS : Ubuntu 1804 uname -a : 4.13.0-32-generic #35~lp1746225 ( Kernel from the bug : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 ) Steps to reproduce : Build Kernel : To avoid the kernel crash due to Perf fuzzer , use the kernel mentioned in the link : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 #! /bin/bash set -x git clone https://github.com/deater/perf_event_tests.git cd perf_event_tests/include mkdir asm cd asm wget http://9.114.13.132/repo/shriya/perf_regs.h cd ../../lib make sleep 10 cd ../fuzzer make sleep 10 echo 0 > /proc/sys/kernel/nmi_watchdog echo 2 > /proc/sys/kernel/perf_event_paranoid echo 10 > /proc/sys/kernel/perf_event_max_sample_rate ./perf_fuzzer -r 1492143527 Call trace : === [ 329.228031] [ cut here ] [ 329.228039] WARNING: CPU: 43 PID: 9088 at /home/jsalisbury/bugs/lp1746225/ubuntu-artful/kernel/events/core.c:3038 perf_pmu_sched_task+0x170/0x180 [ 329.228040] Modules linked in: ofpart at24 uio_pdrv_genirq uio cmdlinepart powernv_flash mtd ipmi_powernv vmx_crypto ipmi_devintf ipmi_msghandler ibmpowernv opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum lpfc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm tg3 nvmet_fc drm ahci nvmet nvme_fc libahci nvme_fabrics mlxfw nvme_core devlink scsi_transport_fc [ 329.228068] CPU: 43 PID: 9088 Comm: perf_fuzzer Not tainted 4.13.0-32-generic #35~lp1746225 [ 329.228070] task: c03f776ac900 task.stack: c03f77728000 [ 329.228071] NIP: c0299b70 LR: c02a4534 CTR: c029bb80 [ 329.228073] REGS: c03f7772b760 TRAP: 0700 Not tainted (4.13.0-32-generic) [ 329.228073] MSR: 9282b033 [ 329.228079] CR: 24008822 XER: [ 329.228080] CFAR: c0299a70 SOFTE: 0 GPR00: c02a4534 c03f7772b9e0 c1606200 c03fef858908 GPR04: c03f776ac900 0001 003fee73 GPR08: c11220d8 0002 GPR12: c029bb80 c7a3d900 GPR16: GPR20: c03f776ad090 c0c71354 GPR24: c03fef716780 003fee73 c03fe69d4200 c03f776ad330 GPR28: c11220d8 0001 c14c6108 c03fef858900 [ 329.228098] NIP [c0299b70] perf_pmu_sched_task+0x170/0x180 [ 329.228100] LR [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228101] Call Trace: [ 329.228102] [c03f7772b9e0] [c02a0678] perf_iterate_sb+0x158/0x2a0 (unreliable) [ 329.228105] [c03f7772ba30] [c02a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228107] [c03f7772bab0] [c01396dc] finish_task_switch+0x21c/0x310 [ 329.228109] [c03f7772bb60] [c0c71354] __schedule+0x304/0xb80 [ 329.228111] [c03f7772bc40] [c0c71c10] schedule+0x40/0xc0 [ 329.228113] [c03f7772bc60] [c01033f4] do_wait+0x254/0x2e0 [ 329.228115] [c03f7772bcd0] [c0104ac0] kernel_wait4+0xa0/0x1a0 [ 329.228117] [c03f7772bd70] [c0104c24] SyS_wait4+0x64/0xc0 [ 329.228121] [c03f7772be30] [c000b184] system_call+0x58/0x6c [ 329.228121] Instruction dump: [ 329.228123] 3beafea0 7faa4800 409eff18 e8010060 eb610028 ebc10040 7c0803a6 38210050 [ 329.228127] eb81ffe0 eba1ffe8 ebe1fff8 4e800020 <0fe0> 4bbc 6000 6042 [ 329.228131] ---[ end trace 8c46856d314c1811 ]--- [ 375.755943] hrtimer: interrupt took 31601 ns == Comment: #4 - SEETEENA THOUFEEK - 2018-02-05 06:34:09 == == Comment: #5 - SEETEENA THOUFEEK - 2018-02-05 06:36:12 == We have similar issue reported on different distro where Anju Provided the patch. Patch attached above. . Will ch
