[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
@finnegan thanks for the feedback - closing this ticket according to comment #24. ** Changed in: linux (Ubuntu) Status: Incomplete => Invalid ** Changed in: ubuntu-z-systems Status: Incomplete => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
** Changed in: linux (Ubuntu) Assignee: Skipper Bug Screeners (skipper-screen-team) => (unassigned) ** Changed in: ubuntu-z-systems Assignee: Canonical Kernel Team (canonical-kernel-team) => (unassigned) ** Changed in: linux (Ubuntu) Status: Triaged => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
** Tags added: cscc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
** Changed in: linux (Ubuntu) Importance: High => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
** Changed in: ubuntu-z-systems Importance: High => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
After discussion with IBM this will now de-prioritized to Medium. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
IBM, have you had any luck with the test kernel? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
** Changed in: ubuntu-z-systems Status: Triaged => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
Can you please test the test kernel at: https://kernel.ubuntu.com/~juergh/lp1804149/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
Can you please give this test kernel a try: https://kernel.ubuntu.com/~juergh/lp1804149/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
Having a look at the new logs it shows that there are many different disks and disk systems in use. I see DASDs, that get partly unhappy, like: Dec 8 07:43:29 ilabg3 kernel: [651348.534047] dasd(eckd): I/O status report for device 0.0.248b: Dec 8 07:43:29 ilabg3 kernel: [651348.534047] dasd(eckd): in req: 6004d11c CC:00 FC:01 AC:00 SC:01 DS:00 CS:00 RC:0 Dec 8 07:43:29 ilabg3 kernel: [651348.534047] dasd(eckd): device 0.0.248b: Failing CCW: (null) Dec 8 07:43:29 ilabg3 kernel: [651348.534047] dasd(eckd): SORRY - NO VALID SENSE AVAILABLE Dec 8 07:43:29 ilabg3 kernel: [651348.534072] dasd(eckd): Related CP in req: 6004d11c Dec 8 07:43:29 ilabg3 kernel: [651348.534072] dasd(eckd): CCW 584ecfe4: 274C 7FE86BC0 DAT: 1800 0e00 Dec 8 07:43:29 ilabg3 kernel: [651348.534072] dasd(eckd): CCW a235ae5d: 3E000200 7FEDD400 DAT: (of course not related to the reported issue) I see 'local' scsi disks: scsi 3:0:12:0: Direct-Access IBM FlashSystem-9840 1600 PQ: 0 ANSI: 5 and I see XIV Storage as well. I noticed a lot of 'Power-on or device reset's, as well as many 'LUN scan's and multipath path failures on a single day on that environment, like: sd 9:0:0:1: tag#58 Add. Sense: Power on, reset, or bus device reset occurred ... Dec 8 14:01:07 ilabg3 kernel: [2.127745] sd 3:0:0:0: scsi scan: REPORT LUN scan ... [458582.527232] device-mapper: multipath: Failing path 66:112. [458582.527234] print_req_error: I/O error, dev sdan, sector 15001576 That seems to be very odd and points to an issue with the SAN infrastructure and/or setup... The error messages that belong to the Ops seem to be known and points to a potential race discussed here: [RFC,1/1] libiscsi: Fix race between iscsi_xmit_task and iscsi_complete_task https://patchwork.kernel.org/patch/10501773/ Well, the situation above may even favor a race condition like this ... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
Not sure if I understand the overall situation correctly, but: In the bug description I read that the issue occurs with 4.15.0-36-generic, and in comment 4 its also mentioned that it happens with 4.15.0-36-generic and 4.15.0-23-generic and again in comment 6 also on 4.15.0-15-generic (I assume all on 16.04). But in comment 8 (according to LP 1782085) the issue doesn't happen (at least for the last 47 days) on a kernel 4.15.0-36-generic (on that particular system) - but this is the same kernel level like the one mentioned in the bug description?! Either there is something wrong with the mentioned kernel levels, or it doesn't depend on the kernel, but for example on the infrastructure?! LP 1782085 describes a situation on an LPAR system, but LP 1804149 on a zVM 7.1 guest system. Are there other differences between both systems? Can it have something todo with the underlying hypervisor? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
In bug 1782085 it looks like the bug stopped happening after upgrading to version 4.15.0-23. Do you know if that is still the case with that older system, or has the bug come back after upgrading to a newer kernel version? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
--- Comment From finne...@us.ibm.com 2018-11-28 13:52 EDT--- (In reply to comment #9) > Were you able to test the mainline kernel to see if this bug still exists > there? Our testing typically requires us to be on released kernels to support release schedules. Since this issue is very intermittent (seen on this host on 201809121135 and 201811161956) we can't stay on a mainline kernel for 2+ months and also continue testing on released kernels to support release schedules --- Comment From finne...@us.ibm.com 2018-11-28 13:54 EDT--- Similar Stack trace output also seen on another host running on older kernel i.e. 4.15.0-15-generic #16-Ubuntu" Reference Bug 169770 https://bugzilla.linux.ibm.com/show_bug.cgi?id=169770 Bug 169770 - LP1782085 : Kernel panic,iSCSI,Unable to handle kernel pointer dereference in virtual kernel address space ** Bug watch added: bugzilla.linux.ibm.com/ #169770 https://bugzilla.linux.ibm.com/show_bug.cgi?id=169770 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
Were you able to test the mainline kernel to see if this bug still exists there? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem? Would it be possible for you to test the latest upstream kernel to see if the bug exists there as well? That kernel is available from: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.20-rc3 ** Changed in: linux (Ubuntu) Importance: Undecided => High ** Changed in: linux (Ubuntu) Status: New => Triaged ** Tags added: kernel-da-key -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
** Package changed: kernel-package (Ubuntu) => linux (Ubuntu) ** Also affects: ubuntu-z-systems Importance: Undecided Status: New ** Changed in: ubuntu-z-systems Status: New => Triaged ** Changed in: ubuntu-z-systems Importance: Undecided => High ** Changed in: ubuntu-z-systems Assignee: (unassigned) => Canonical Kernel Team (canonical-kernel-team) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1804149] Re: Kernel panic, Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi]
** Tags removed: targetmilestone-inin--- ** Tags added: targetmilestone-inin1804 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1804149 Title: Kernel panic,Oops: 0004 ilc:3 [#1] SMP, iscsi_q_20 iscsi_xmitworker [libiscsi] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/kernel-package/+bug/1804149/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
