------- Comment From [email protected] 2019-06-24 04:52 EDT------- Anju, I'm trying this and seeing: ================================================== Starting fuzzing at 2019-06-24 03:26:26 ================================================== Cannot open /sys/kernel/tracing/kprobe_events SIGIO due to RT queue overflow Signal from invalid fd 10 Bad file descriptor Iteration 10000, 139369 syscalls in 25.77 s (5.407 k syscalls/s) Open attempts: 131443 Successful: 918 Currently open: 5 ENOENT : 370 E2BIG : 11571 EBADF : 7775 EINVAL : 110627 ENOSPC : 31 EOVERFLOW : 1 EOPNOTSUPP : 150 Trinity Type (Normal 142/32892)(Sampling 11/32823)(Global 693/32727)(Random 72/33001) Type (Hardware 268/18287)(software 318/17277)(tracepoint 60/16952)(Cache 55/16464)(cpu 151/16831)(breakpoint 17/16793)(nest_alink0_imc 1/387)(nest_alink1_imc 2/353)(nest_alink2_imc 2/455)(nest_alink3_imc 0/380)(nest_capp0_imc 0/352)(nest_capp1_imc 0/385)(nest_centaur0_imc 0/381)(nest_centaur1_imc 0/384)(nest_centaur2_imc 2/339)(nest_centaur3_imc 1/380)(nest_centaur4_imc 0/482)(nest_centaur5_imc 2/378)(nest_centaur6_imc 0/344)(nest_centaur7_imc 39/23839) Close: 913/913 Successful Read: 821/892 Successful Write: 0/837 Successful Ioctl: 276/883 Successful: (ENABLE 60/60)(DISABLE 60/60)(REFRESH 4/57)(RESET 66/66)(PERIOD 9/58)(SET_OUTPUT 5/53)(SET_FILTER 0/68)(ID 63/63)(SET_BPF 0/57)(PAUSE_OUTPUT 9/63)(QUERY_BPF 0/71)(MOD_ATTR 0/71)(#12 0/0)(#13 0/0)(#14 0/0)(>14 0/136) Mmap: 661/1075 Successful: (MMAP 661/1075)(TRASH 119/144)(READ 127/134)(UNMAP 661/1050)(AUX 0/155)(AUX_READ 0/0) Prctl: 900/900 Successful Fork: 466/466 Successful Poll: 831/893 Successful Access: 329/918 Successful Overflows: 1972959 Recursive: 0 SIGIOs due to RT signal queue full: 1 Iteration 20000, 140897 syscalls in 7.04 s (20.024 k syscalls/s) Open attempts: 133151 Successful: 901 Currently open: 24 ENOENT : 402 E2BIG : 11642 EBADF : 8002 EINVAL : 112005 ENOSPC : 33 EOVERFLOW : 1 EOPNOTSUPP : 165 Trinity Type (Normal 110/33454)(Sampling 15/33233)(Global 700/33200)(Random 76/33264) Type (Hardware 276/18590)(software 303/17115)(tracepoint 54/17217)(Cache 56/16759)(cpu 138/17185)(breakpoint 15/17011)(nest_alink0_imc 2/352)(nest_alink1_imc 3/370)(nest_alink2_imc 0/498)(nest_alink3_imc 3/382)(nest_capp0_imc 1/378)(nest_capp1_imc 1/359)(nest_centaur0_imc 1/364)(nest_centaur1_imc 0/363)(nest_centaur2_imc 1/376)(nest_centaur3_imc 2/356)(nest_centaur4_imc 1/483)(nest_centaur5_imc 1/369)(nest_centaur6_imc 0/385)(nest_centaur7_imc 43/24239) Close: 882/882 Successful Read: 790/870 Successful Write: 0/857 Successful Ioctl: 266/896 Successful: (ENABLE 63/63)(DISABLE 59/59)(REFRESH 6/74)(RESET 69/69)(PERIOD 5/68)(SET_OUTPUT 5/76)(SET_FILTER 0/68)(ID 49/49)(SET_BPF 0/67)(PAUSE_OUTPUT 10/60)(QUERY_BPF 0/64)(MOD_ATTR 0/59)(#12 0/0)(#13 0/0)(#14 0/0)(>14 0/120) Mmap: 623/1028 Successful: (MMAP 623/1028)(TRASH 121/147)(READ 118/124)(UNMAP 618/1004)(AUX 0/135)(AUX_READ 0/0) Prctl: 868/868 Successful Fork: 452/452 Successful Poll: 791/865 Successful Access: 332/926 Successful Overflows: 0 Recursive: 0 SIGIOs due to RT signal queue full: 0 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 0 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 1 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 2 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 3 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 4 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 5 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 6 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 7 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 8 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 9 Throttling event 1 fd 6, last_refresh=0, period=8816262, type=1 throttles 10 Stuck in a signal storm w/o forward progress; Max throttle count hit, giving up
And the end of dmesg and the (-proposed) kernel version: [ 374.873933] perf: Dynamic interrupt throttling disabled, can hang your system! [ 398.775635] perf: Dynamic interrupt throttling disabled, can hang your system! [ 401.495532] perf: Dynamic interrupt throttling disabled, can hang your system! ubuntu@ltc-wspoon5:~$ cat /proc/version Linux version 4.15.0-51-generic (arighi@kathleen) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #55 SMP Wed Jun 5 14:15:11 UTC 2019 ubuntu@ltc-wspoon5:~$ I don't see similar on the -50 kernel. This might be unrelated, but you have ideas? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1752002 Title: [P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is seen while running perf fuzzer (perf:) Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: In Progress Status in linux source package in Bionic: In Progress Bug description: == Comment: #0 - Shriya R. Kulkarni <[email protected]> - 2018-02-02 01:21:36 == Problem Description : ============= Warn on message is seen while running perf fuzzer tests. Machine details : ========== Hardware : Witherspoon (wsp12) + DD2.1 OS : Ubuntu 1804 uname -a : 4.13.0-32-generic #35~lp1746225 ( Kernel from the bug : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 ) Steps to reproduce : ============ Build Kernel : -------------------- To avoid the kernel crash due to Perf fuzzer , use the kernel mentioned in the link : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 #! /bin/bash set -x git clone https://github.com/deater/perf_event_tests.git cd perf_event_tests/include mkdir asm cd asm wget http://9.114.13.132/repo/shriya/perf_regs.h cd ../../lib make sleep 10 cd ../fuzzer make sleep 10 echo 0 > /proc/sys/kernel/nmi_watchdog echo 2 > /proc/sys/kernel/perf_event_paranoid echo 100000 > /proc/sys/kernel/perf_event_max_sample_rate ./perf_fuzzer -r 1492143527 Call trace : ======= [ 329.228031] ------------[ cut here ]------------ [ 329.228039] WARNING: CPU: 43 PID: 9088 at /home/jsalisbury/bugs/lp1746225/ubuntu-artful/kernel/events/core.c:3038 perf_pmu_sched_task+0x170/0x180 [ 329.228040] Modules linked in: ofpart at24 uio_pdrv_genirq uio cmdlinepart powernv_flash mtd ipmi_powernv vmx_crypto ipmi_devintf ipmi_msghandler ibmpowernv opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum lpfc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm tg3 nvmet_fc drm ahci nvmet nvme_fc libahci nvme_fabrics mlxfw nvme_core devlink scsi_transport_fc [ 329.228068] CPU: 43 PID: 9088 Comm: perf_fuzzer Not tainted 4.13.0-32-generic #35~lp1746225 [ 329.228070] task: c000003f776ac900 task.stack: c000003f77728000 [ 329.228071] NIP: c000000000299b70 LR: c0000000002a4534 CTR: c00000000029bb80 [ 329.228073] REGS: c000003f7772b760 TRAP: 0700 Not tainted (4.13.0-32-generic) [ 329.228073] MSR: 900000000282b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> [ 329.228079] CR: 24008822 XER: 00000000 [ 329.228080] CFAR: c000000000299a70 SOFTE: 0 GPR00: c0000000002a4534 c000003f7772b9e0 c000000001606200 c000003fef858908 GPR04: c000003f776ac900 0000000000000001 ffffffffffffffff 0000003fee730000 GPR08: 0000000000000000 0000000000000000 c0000000011220d8 0000000000000002 GPR12: c00000000029bb80 c000000007a3d900 0000000000000000 0000000000000000 GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR20: 0000000000000000 0000000000000000 c000003f776ad090 c000000000c71354 GPR24: c000003fef716780 0000003fee730000 c000003fe69d4200 c000003f776ad330 GPR28: c0000000011220d8 0000000000000001 c0000000014c6108 c000003fef858900 [ 329.228098] NIP [c000000000299b70] perf_pmu_sched_task+0x170/0x180 [ 329.228100] LR [c0000000002a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228101] Call Trace: [ 329.228102] [c000003f7772b9e0] [c0000000002a0678] perf_iterate_sb+0x158/0x2a0 (unreliable) [ 329.228105] [c000003f7772ba30] [c0000000002a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228107] [c000003f7772bab0] [c0000000001396dc] finish_task_switch+0x21c/0x310 [ 329.228109] [c000003f7772bb60] [c000000000c71354] __schedule+0x304/0xb80 [ 329.228111] [c000003f7772bc40] [c000000000c71c10] schedule+0x40/0xc0 [ 329.228113] [c000003f7772bc60] [c0000000001033f4] do_wait+0x254/0x2e0 [ 329.228115] [c000003f7772bcd0] [c000000000104ac0] kernel_wait4+0xa0/0x1a0 [ 329.228117] [c000003f7772bd70] [c000000000104c24] SyS_wait4+0x64/0xc0 [ 329.228121] [c000003f7772be30] [c00000000000b184] system_call+0x58/0x6c [ 329.228121] Instruction dump: [ 329.228123] 3beafea0 7faa4800 409eff18 e8010060 eb610028 ebc10040 7c0803a6 38210050 [ 329.228127] eb81ffe0 eba1ffe8 ebe1fff8 4e800020 <0fe00000> 4bffffbc 60000000 60420000 [ 329.228131] ---[ end trace 8c46856d314c1811 ]--- [ 375.755943] hrtimer: interrupt took 31601 ns == Comment: #4 - SEETEENA THOUFEEK <[email protected]> - 2018-02-05 06:34:09 == == Comment: #5 - SEETEENA THOUFEEK <[email protected]> - 2018-02-05 06:36:12 == We have similar issue reported on different distro where Anju Provided the patch. Patch attached above. . Will check with her if that patch got accepted upstream == Comment: #14 - SEETEENA THOUFEEK <[email protected]> - 2018-02-23 01:52:50 == To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1752002/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : [email protected] Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
