[Bug 1777194] Comment bridged from LTC Bugzilla
--- Comment From hasri...@in.ibm.com 2018-07-31 03:30 EDT--- (In reply to comment #51) > (In reply to comment #50) > > Yes, it would probably be best to open a new bug, so we can track the new > > lockup separately. > > Harish: please open a new defect and let us close this as working. Closing this bug as this issue is not seen anymore. Will open a new bug for the other issues. Thanks. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1777194 Title: Hard LOCKUP observed on stressing Ubuntu 18 04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1777194/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1777194] Comment bridged from LTC Bugzilla
--- Comment From kalsh...@in.ibm.com 2018-07-29 08:54 EDT--- (In reply to comment #50) > Yes, it would probably be best to open a new bug, so we can track the new > lockup separately. Harish: please open a new defect and let us close this as working. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1777194 Title: Hard LOCKUP observed on stressing Ubuntu 18 04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1777194/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1777194] Comment bridged from LTC Bugzilla
--- Comment From hasri...@in.ibm.com 2018-07-26 04:12 EDT--- (In reply to comment #46) > According to our understanding of the comments and logs provided, the > original hard lockup is different to the one latest reported. So it seems > safe to say testing shows the original problem is resolved and a new problem > which is much less serious is observed. > > So I will mark the verification of this bug as "done". > > Thank you. Does this mean we need to open another Bug to track the issue observed in comment 15(LP: 1777194) ? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1777194 Title: Hard LOCKUP observed on stressing Ubuntu 18 04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1777194/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1777194] Comment bridged from LTC Bugzilla
--- Comment From hasri...@in.ibm.com 2018-07-20 02:36 EDT--- Following lockups are observed Jul 20 01:15:35 lep8d kernel: [ 483.899693] Watchdog CPU:126 Hard LOCKUP Jul 20 01:15:35 lep8d kernel: [ 483.899700] Modules linked in: serpent_generic cuse twofish_generic twofish_common vhost_net tap vhost_vsock vmw_vsock_virtio_transport_common vhost vsock lrw algif_skcipher tgr192 unix_diag wp512 sctp rmd320 rmd256 snd_seq snd_seq_device snd_timer rmd160 snd soundcore rmd128 dccp_ipv4 md4 dccp algif_hash af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc devlink ip6table_filter ip6_tables iptable_filter uio_pdrv_genirq uio kvm_hv kvm leds_powernv powernv_rng vmx_crypto ipmi_powernv ipmi_devintf ibmpowernv ipmi_msghandler binfmt_misc sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables Jul 20 01:15:35 lep8d kernel: [ 483.900118] x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear uas usb_storage crct10dif_vpmsum crc32c_vpmsum ipr tg3 [last unloaded: ebtables] Jul 20 01:15:35 lep8d kernel: [ 483.900292] CPU: 126 PID: 70582 Comm: stress-ng-fstat Not tainted 4.15.0-29-generic #32~lp1777194 Jul 20 01:15:35 lep8d kernel: [ 483.900300] NIP: c0d02364 LR: c01145ac CTR: c0162300 Jul 20 01:15:35 lep8d kernel: [ 483.900305] REGS: c0003fa17d80 TRAP: 0900 Not tainted (4.15.0-29-generic) Jul 20 01:15:35 lep8d kernel: [ 483.900312] MSR: 90009033 CR: 22004244 XER: Jul 20 01:15:35 lep8d kernel: [ 483.900360] CFAR: c0d0236c SOFTE: 0 Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR00: c01145ac c00fa8147d20 c16eaf00 c1712180 Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR04: 000d 0040 0842 Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR08: 002ffbad 807e c00fa81bc830 Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR12: 4400 cfad6a00 026380759318 74efb46d Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR16: 026380759210 74efbae24410 74efbae24420 74efb4edf240 Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR20: 7fffe93cbe40 7fffe93cbe78 Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR24: c00fa81bc780 c1712180 Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR28: c02d55ace160 c00fa8147db0 c1712180 Jul 20 01:15:35 lep8d kernel: [ 483.900551] NIP [c0d02364] _raw_write_lock_irq+0x74/0xe0 Jul 20 01:15:35 lep8d kernel: [ 483.900557] LR [c01145ac] do_exit+0x41c/0xc80 Jul 20 01:15:35 lep8d kernel: [ 483.900565] Call Trace: Jul 20 01:15:35 lep8d kernel: [ 483.900574] [c00fa8147d20] [c00fa8147d50] 0xc00fa8147d50 (unreliable) Jul 20 01:15:35 lep8d kernel: [ 483.900590] [c00fa8147d50] [c0114590] do_exit+0x400/0xc80 Jul 20 01:15:35 lep8d kernel: [ 483.900619] [c00fa8147e10] [c0114e74] SyS_exit+0x24/0x30 Jul 20 01:15:35 lep8d kernel: [ 483.900632] [c00fa8147e30] [c000b284] system_call+0x58/0x6c Jul 20 01:15:35 lep8d kernel: [ 483.900641] Instruction dump: Jul 20 01:15:35 lep8d kernel: [ 483.900648] 2fa9 409e0018 38210030 ebe1fff8 4e800020 6000 6042 7c210b78 Jul 20 01:15:35 lep8d kernel: [ 483.900692] e92d 89290009 71290002 40820050 <813f> 2f89 409effe4 7c421378 Jul 20 01:15:35 lep8d kernel: [ 483.975703] Watchdog CPU:142 Hard LOCKUP Jul 20 01:15:35 lep8d kernel: [ 483.975723] Modules linked in: serpent_generic cuse twofish_generic twofish_common vhost_net tap vhost_vsock vmw_vsock_virtio_transport_common vhost vsock lrw algif_skcipher tgr192 unix_diag wp512 sctp rmd320 rmd256 snd_seq snd_seq_device snd_timer rmd160 snd soundcore rmd128 dccp_ipv4 md4 dccp algif_hash af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc devlink ip6table_filter ip6_tables iptable_filter uio_pdrv_genirq uio kvm_hv kvm leds_powernv powernv_rng vmx_crypto ipmi_powernv ipmi_devintf ibmpowernv ipmi_msghandler binfmt_misc sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables Jul 20 01:15:35 lep8d kernel: [ 483.976470] x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear uas usb_storage crct10dif_vpmsum crc32c_vpmsum ipr
[Bug 1777194] Comment bridged from LTC Bugzilla
--- Comment From hasri...@in.ibm.com 2018-07-20 01:38 EDT--- (In reply to comment #42) > I built a .deb version of the kernel that is currently in -proposed. > The test kernel can be downloaded from: > http://kernel.ubuntu.com/~jsalisbury/lp1777194 > > Can you test this kernel and see if it resolves this bug? I am seeing the following LOCKUP on the above kernel. # uname -a Linux lep8d 4.15.0-29-generic #32~lp1777194 SMP Thu Jul 19 16:21:31 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux [ 349.854120] Watchdog CPU:16 Hard LOCKUP [ 349.854129] Modules linked in: userio(E) serpent_generic(E) hci_vhci(E) bluetooth(E) ecdh_generic(E) twofish_generic(E) twofish_common(E) cuse(E) vhost_vsock(E) vmw_vsock_virtio_transport_common(E) vsock(E) lrw(E) uhid(E) hid(E) algif_skcipher(E) tgr192(E) wp512(E) rmd320(E) snd_seq(E) snd_seq_device(E) unix_diag(E) snd_timer(E) sctp(E) rmd256(E) snd(E) soundcore(E) rmd160(E) rmd128(E) md4(E) vhost_net(E) vhost(E) tap(E) algif_hash(E) dccp_ipv4(E) dccp(E) af_alg(E) xt_CHECKSUM(E) iptable_mangle(E) ipt_MASQUERADE(E) nf_nat_masquerade_ipv4(E) iptable_nat(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack_ipv4(E) nf_defrag_ipv4(E) xt_conntrack(E) nf_conntrack(E) libcrc32c(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_tcpudp(E) bridge(E) stp(E) llc(E) ebtable_filter(E) ebtables(E) devlink(E) ip6table_filter(E) ip6_tables(E) [ 349.854595] iptable_filter(E) kvm_hv(E) kvm(E) binfmt_misc(E) dm_multipath(E) scsi_dh_rdac(E) scsi_dh_emc(E) scsi_dh_alua(E) uio_pdrv_genirq(E) uio(E) ipmi_powernv(E) ipmi_devintf(E) ipmi_msghandler(E) vmx_crypto(E) powernv_rng(E) powernv_op_panel(E) ibmpowernv(E) leds_powernv(E) crct10dif_vpmsum(E) nfsd(E) sch_fq_codel(E) auth_rpcgss(E) nfs_acl(E) lockd(E) grace(E) sunrpc(E) ip_tables(E) x_tables(E) autofs4(E) ses(E) enclosure(E) scsi_transport_sas(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) uas(E) usb_storage(E) crc32c_vpmsum(E) tg3(E) ipr(E) [ 349.854940] CPU: 16 PID: 1634 Comm: ureadahead Tainted: GE 4.15.0-29-generic #32~lp1777194 [ 349.854949] NIP: c029738c LR: c0297430 CTR: c00aa8a0 [ 349.854961] REGS: c7ee3d80 TRAP: 0900 Tainted: GE (4.15.0-29-generic) [ 349.854993] MSR: 90009033 CR: 28002848 XER: [ 349.855081] CFAR: c029742c SOFTE: 0 GPR00: c0297430 c0213cc0 c16eaf00 c00fff924360 GPR04: 0010 c00fff924360 c171dd78 b000 GPR08: c1721ee0 0003 c00ff9004928 GPR12: c00aa8a0 c7a2b000 0001 GPR16: 0024 0e8aa6080160 0014 GPR20: 0040 c00fa02a c00fa02a01d0 0001 GPR24: 0002 0010 c02ff3171800 c0213ef0 GPR28: fff8 0002 0003 c00ffa110450 [ 349.855354] NIP [c029738c] irq_work_run_list+0x8c/0x100 [ 349.855373] LR [c0297430] irq_work_run+0x30/0x60 [ 349.855384] Call Trace: [ 349.855400] [c0213cc0] [c02973b0] irq_work_run_list+0xb0/0x100 (unreliable) [ 349.855415] [c0213d00] [c0297430] irq_work_run+0x30/0x60 [ 349.855432] [c0213d30] [c01d7268] flush_smp_call_function_queue+0xd8/0x1d0 [ 349.855439] [c0213db0] [c004923c] smp_ipi_demux_relaxed+0x9c/0x110 [ 349.855445] [c0213df0] [c0098be4] icp_native_ipi_action+0x64/0x80 [ 349.855468] [c0213e10] [c0192440] __handle_irq_event_percpu+0x90/0x300 [ 349.855487] [c0213ed0] [c01926e8] handle_irq_event_percpu+0x38/0x90 [ 349.855510] [c0213f10] [c0198f74] handle_percpu_irq+0x84/0xd0 [ 349.855527] [c0213f40] [c0190b0c] generic_handle_irq+0x4c/0x70 [ 349.855549] [c0213f60] [c001787c] __do_irq+0x7c/0x1c0 [ 349.855569] [c0213f90] [c002a090] call_do_irq+0x14/0x24 [ 349.855575] [c4e4b7a0] [c0017a5c] do_IRQ+0x9c/0x130 [ 349.855582] [c4e4b7f0] [c0008c24] hardware_interrupt_common+0x114/0x120 [ 349.855588] --- interrupt: 501 at replay_interrupt_return+0x0/0x4 LR = arch_local_irq_restore+0x74/0x90 [ 349.855593] [c4e4bae0] [c00fa028] 0xc00fa028 (unreliable) [ 349.855599] [c4e4bb00] [c024b158] ring_buffer_empty_cpu+0xa8/0x150 [ 349.855612] [c4e4bb40] [c024e560] __find_next_entry+0xe0/0x260 [ 349.855617] [c4e4bbe0] [c0257704] trace_find_next_entry_inc+0x34/0xd0 [ 349.855622] [c4e4bc10] [c0257818] s_next+0x78/0x100 [ 349.855626] [c4e4bc50] [c02582c0] s_start+0x180/0x350 [ 349.855643] [c4e4bcd0] [c040ef1c] seq_read+0xfc/0x610 [ 349.855654] [c4e4bd70] [c03d509c] __vfs_read+0x3c/0x70 [ 349.855657] [c4e4bd90] [c03d518c] vfs_read+0x
[Bug 1777194] Comment bridged from LTC Bugzilla
--- Comment From hasri...@in.ibm.com 2018-07-19 09:54 EDT--- (In reply to comment #40) > I just confirmed the two requested patches are in the Ubuntu-4.15.0-29 > kernel: > eeada334156f powerpc: use NMI IPI for smp_send_stop > 4190e27bf0ee powerpc: Fix smp_send_stop NMI IPI handling > > Can you confirm the kernel posted in comment #5 fixes this bug on the same > machine? As this is observed on Tuleta, i am not able to boot this kernel even after I used "apt-setup/proposed=true" while booting. Can you build a deb with the same? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1777194 Title: Hard LOCKUP observed on stressing Ubuntu 18 04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1777194/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1777194] Comment bridged from LTC Bugzilla
--- Comment From hasri...@in.ibm.com 2018-07-18 14:09 EDT--- Able to reproduce the issue with the latest proposed kernel on Tuleta machine. Seems the issue is not resolved with the provided patches. [ 155.121412] Watchdog CPU:65 Hard LOCKUP [ 155.121416] Modules linked in: rmd256 rmd160 rmd128 vhost_net md4 vhost dccp_ipv4(+) sctp(+) unix_diag algif_hash tap dccp af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter kvm_hv kvm binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua uio_pdrv_genirq uio ipmi_powernv ipmi_devintf ipmi_msghandler powernv_rng ibmpowernv vmx_crypto leds_powernv powernv_op_panel crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas btrfs xor zstd_compress raid6_pq uas usb_storage crc32c_vpmsum ipr tg3 [ 155.121531] CPU: 65 PID: 7391 Comm: stress-ng-hrtim Not tainted 4.15.0-29-generic #31-Ubuntu [ 155.121537] NIP: c0d01444 LR: c01269c8 CTR: [ 155.121544] REGS: c7c97d80 TRAP: 0900 Not tainted (4.15.0-29-generic) [ 155.121545] MSR: 90009033 CR: 28000448 XER: [ 155.121560] CFAR: c01269c4 SOFTE: 0 GPR00: c01269c8 c01fe581fc70 c16eaf00 c01fe57ad408 GPR04: c01fe581fd30 007cd2603a7c 007cd261c940 GPR08: 0021 8041 42000448 GPR12: 0001 c7a4cb00 GPR16: 000186a0 7fffd75e1ef0 007cd25c7040 7fffd75e1ee8 GPR20: 007cd25c7048 7fffd75e21ab 7fffd75e21aa GPR24: 0001 7896e1ca 7fffd75e1da8 7fffd75e1ef8 GPR28: 00040002 c01fe581fcf0 c01fe57ad408 [ 155.121625] NIP [c0d01444] _raw_spin_lock_irq+0x44/0xe0 [ 155.121631] LR [c01269c8] __set_current_blocked+0x48/0xb0 [ 155.121633] Call Trace: [ 155.121640] [c01fe581fca0] [007cd25c7048] 0x7cd25c7048 [ 155.121648] [c01fe581fcd0] [c0126ae4] signal_setup_done+0x84/0xe0 [ 155.121657] [c01fe581fd10] [c001da20] do_signal+0x110/0x2c0 [ 155.121665] [c01fe581fe00] [c001dd70] do_notify_resume+0xd0/0x100 [ 155.121672] [c01fe581fe30] [c000b8c4] ret_from_except_lite+0x70/0x74 [ 155.121673] Instruction dump: [ 155.121677] f821ffd1 7c7f1b78 3940 892d028b 994d028b 3940 994d028d 814d0008 [ 155.121692] 7d201829 2c09 40c20010 7d40192d <40c2fff0> 7c2004ac 2fa9 409e0010 # uname -a Linux lep8d 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:37:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux Thanks, Harish -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1777194 Title: Hard LOCKUP observed on stressing Ubuntu 18 04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1777194/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1777194] Comment bridged from LTC Bugzilla
--- Comment From kalsh...@in.ibm.com 2018-07-01 10:35 EDT--- (In reply to comment #36) > SRU request submitted: > https://lists.ubuntu.com/archives/kernel-team/2018-June/093481.html Do we get this fix added to 18.04.1 builds? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1777194 Title: Hard LOCKUP observed on stressing Ubuntu 18 04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1777194/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1777194] Comment bridged from LTC Bugzilla
--- Comment From hasri...@in.ibm.com 2018-06-19 05:39 EDT--- (In reply to comment #34) > > I built a test kernel with commits 6bed3237624e and ac61c1156623. The test > kernel can be downloaded from: > http://kernel.ubuntu.com/~jsalisbury/lp1777194 > > Can you test this kernel and see if it resolves this bug? > > Note about installing test kernels: > ? If the test kernel is prior to 4.15(Bionic) you need to install the > linux-image and linux-image-extra .deb packages. > ? If the test kernel is 4.15(Bionic) or newer, you need to install the > linux-modules, linux-modules-extra and linux-image-unsigned .deb packages. > > Thanks in advance! Issue is not observed with the unsigned kernel provided above. # uname -a Linux ltc-wspoon12 4.15.0-23-generic #26~lp1777194 SMP Mon Jun 18 23:45:25 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux Thanks, Harish -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1777194 Title: Hard LOCKUP observed on stressing Ubuntu 18 04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1777194/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
