[Kernel-packages] [Bug 1867118] Re: dasd_fba io error and kernel panic
Patch landed in Ubuntu-5.4.0-55.61 (or newer) and since we are currently at: linux-generic | 5.4.0.56.59 | focal-updates| s390x the patch is included and rolled out (via -updates). Hence updating focal entry to Fix Released. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1867118 Title: dasd_fba io error and kernel panic Status in Ubuntu on IBM z Systems: Fix Released Status in linux package in Ubuntu: Fix Released Status in linux source package in Focal: Fix Released Status in linux source package in Groovy: Fix Released Bug description: Generated I/O errors every hour in syslog and caused kernel panic with workloads running ---uname output--- Linux ubuntutest01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:02 UTC 2020 s390x s390x s390x GNU/Linux ---Patches Installed--- distro provided. up-to-date from Ubuntu18.04. Machine Type = z15 ---System Hang--- The hourly generated I/O errors didn't impact the system up and running. When there are workloads on it, it will cause kernel panic and rebooted the server. ---Debugger--- A debugger is not configured ---Steps to Reproduce--- After migrated from 16.04 to 18.04, will see the i/o errors. Stack trace output: no Oops output: no System Dump Info: The system is not configured to capture a system dump. Device driver error code: [52297.898232] dasd.5c0c98: 0.0.0100: default ERP has run out of retries and failed [52297.898244] dasd(fba): I/O status report for device 0.0.0100: dasd(fba): in req: 2e142bcd CS: 0x20 DS: 0x0C dasd(fba): device 0.0.0100: Failing CCW: a84f6c9f dasd(fba): SORRY - NO VALID SENSE AVAILABLE [52297.898255] dasd(fba): Related CP in req: 2e142bcd dasd(fba): CCW d15f44d3: 63400010 7FE2BBA0 DAT: 0200 008150F2 0017 dasd(fba): CCW b09ad76e: 4348 7FE2BBB0 DAT: 0506 dasd(fba): CCW c44dfb20: 41600C00 F360 DAT: dasd(fba): CCW a84f6c9f: 4348 7FE2BBB8 DAT: 0510 0006 dasd(fba): CCW 22dac88c: 4160 DAT: dasd(fba): CCW 69686544: 4348 7FE2BBC0 DAT: 0502 0016 dasd(fba): CCW 7bb6a17c: 41200400 F360 DAT: [52297.898278] print_req_error: I/O error, dev dasda, sector 8474866 *Additional Instructions for y...@cn.ibm.com: -Attach sysctl -a output output to the bug. [reply] [?] Internal Only Comment 1 QI YE 2020-03-12 02:29:20 CDT The z/VM version 6.4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1867118/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1867118] Re: dasd_fba io error and kernel panic
Checking the focal git tree this is included in the Ubuntu kernels: Ubuntu-5.4.0-55.61 Ubuntu-5.4.0-56.62 Ubuntu-5.4.0-57.63 And Ubuntu-5.4.0-56 just migrated to updates: linux-generic | 5.4.0.56.59 | focal-updates| s390x hence I'm updating this (focal) to Fix Released. ** Changed in: linux (Ubuntu Focal) Status: Fix Committed => Fix Released ** Changed in: ubuntu-z-systems Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1867118 Title: dasd_fba io error and kernel panic Status in Ubuntu on IBM z Systems: Fix Released Status in linux package in Ubuntu: Fix Released Status in linux source package in Focal: Fix Released Status in linux source package in Groovy: Fix Released Bug description: Generated I/O errors every hour in syslog and caused kernel panic with workloads running ---uname output--- Linux ubuntutest01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:02 UTC 2020 s390x s390x s390x GNU/Linux ---Patches Installed--- distro provided. up-to-date from Ubuntu18.04. Machine Type = z15 ---System Hang--- The hourly generated I/O errors didn't impact the system up and running. When there are workloads on it, it will cause kernel panic and rebooted the server. ---Debugger--- A debugger is not configured ---Steps to Reproduce--- After migrated from 16.04 to 18.04, will see the i/o errors. Stack trace output: no Oops output: no System Dump Info: The system is not configured to capture a system dump. Device driver error code: [52297.898232] dasd.5c0c98: 0.0.0100: default ERP has run out of retries and failed [52297.898244] dasd(fba): I/O status report for device 0.0.0100: dasd(fba): in req: 2e142bcd CS: 0x20 DS: 0x0C dasd(fba): device 0.0.0100: Failing CCW: a84f6c9f dasd(fba): SORRY - NO VALID SENSE AVAILABLE [52297.898255] dasd(fba): Related CP in req: 2e142bcd dasd(fba): CCW d15f44d3: 63400010 7FE2BBA0 DAT: 0200 008150F2 0017 dasd(fba): CCW b09ad76e: 4348 7FE2BBB0 DAT: 0506 dasd(fba): CCW c44dfb20: 41600C00 F360 DAT: dasd(fba): CCW a84f6c9f: 4348 7FE2BBB8 DAT: 0510 0006 dasd(fba): CCW 22dac88c: 4160 DAT: dasd(fba): CCW 69686544: 4348 7FE2BBC0 DAT: 0502 0016 dasd(fba): CCW 7bb6a17c: 41200400 F360 DAT: [52297.898278] print_req_error: I/O error, dev dasda, sector 8474866 *Additional Instructions for y...@cn.ibm.com: -Attach sysctl -a output output to the bug. [reply] [?] Internal Only Comment 1 QI YE 2020-03-12 02:29:20 CDT The z/VM version 6.4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1867118/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1867118] Re: dasd_fba io error and kernel panic
No, the issue is not yet fixed and rolled out with 20.04 updates. It means that the issue is handled for focal/20.04 by the LP bug 1900624 and that is just Fix Committed (and not yet Fix Released). The status and plan is (as of today) that the commit 709192d531e5 ("s390/dasd: Fix zero write for FBA devices") landed in Ubuntu's focal master-next git tree. And according to 'kernel.ubuntu.com' an new binary kernel will be build and made available for testing in '-proposed' around 13-Nov. And if the "Bug verification & Regression testing" doesn't show any problems, this kernel will then be released to '-updates' around 30-Nov. All kernel updates are usually processed by 3-weekly 'kernel SRU' cycles that are posted here: https://kernel.ubuntu.com/ But due to internal rearrangements and changes one SRU cycle is skipped this time ("There will be no SRU cycle from October 12 through November 8."). But the fix is already included in Ubuntu 20.10, which was released just yesterday! To shorten the time a bit until it get's released for focal/20.04, I just compile a kernel based on the current master-next (that should have the fix in) and made it available for testing here: https://people.canonical.com/~fheimes/lp1867118/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1867118 Title: dasd_fba io error and kernel panic Status in Ubuntu on IBM z Systems: Fix Committed Status in linux package in Ubuntu: Fix Released Status in linux source package in Focal: Fix Committed Status in linux source package in Groovy: Fix Released Bug description: Generated I/O errors every hour in syslog and caused kernel panic with workloads running ---uname output--- Linux ubuntutest01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:02 UTC 2020 s390x s390x s390x GNU/Linux ---Patches Installed--- distro provided. up-to-date from Ubuntu18.04. Machine Type = z15 ---System Hang--- The hourly generated I/O errors didn't impact the system up and running. When there are workloads on it, it will cause kernel panic and rebooted the server. ---Debugger--- A debugger is not configured ---Steps to Reproduce--- After migrated from 16.04 to 18.04, will see the i/o errors. Stack trace output: no Oops output: no System Dump Info: The system is not configured to capture a system dump. Device driver error code: [52297.898232] dasd.5c0c98: 0.0.0100: default ERP has run out of retries and failed [52297.898244] dasd(fba): I/O status report for device 0.0.0100: dasd(fba): in req: 2e142bcd CS: 0x20 DS: 0x0C dasd(fba): device 0.0.0100: Failing CCW: a84f6c9f dasd(fba): SORRY - NO VALID SENSE AVAILABLE [52297.898255] dasd(fba): Related CP in req: 2e142bcd dasd(fba): CCW d15f44d3: 63400010 7FE2BBA0 DAT: 0200 008150F2 0017 dasd(fba): CCW b09ad76e: 4348 7FE2BBB0 DAT: 0506 dasd(fba): CCW c44dfb20: 41600C00 F360 DAT: dasd(fba): CCW a84f6c9f: 4348 7FE2BBB8 DAT: 0510 0006 dasd(fba): CCW 22dac88c: 4160 DAT: dasd(fba): CCW 69686544: 4348 7FE2BBC0 DAT: 0502 0016 dasd(fba): CCW 7bb6a17c: 41200400 F360 DAT: [52297.898278] print_req_error: I/O error, dev dasda, sector 8474866 *Additional Instructions for y...@cn.ibm.com: -Attach sysctl -a output output to the bug. [reply] [?] Internal Only Comment 1 QI YE 2020-03-12 02:29:20 CDT The z/VM version 6.4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1867118/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1867118] Re: dasd_fba io error and kernel panic
The focal part of this ticket is addressed by: "Focal update: v5.4.69 upstream stable release" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900624 And LP 1900624 is Fix Committed and the patches are already applied to focal master-next. Hence aligning the status of this ticket to LP 1900624 and updating it to Fix Committed. ** Changed in: linux (Ubuntu Focal) Status: In Progress => Fix Committed ** Changed in: ubuntu-z-systems Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1867118 Title: dasd_fba io error and kernel panic Status in Ubuntu on IBM z Systems: Fix Committed Status in linux package in Ubuntu: Fix Released Status in linux source package in Focal: Fix Committed Status in linux source package in Groovy: Fix Released Bug description: Generated I/O errors every hour in syslog and caused kernel panic with workloads running ---uname output--- Linux ubuntutest01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:02 UTC 2020 s390x s390x s390x GNU/Linux ---Patches Installed--- distro provided. up-to-date from Ubuntu18.04. Machine Type = z15 ---System Hang--- The hourly generated I/O errors didn't impact the system up and running. When there are workloads on it, it will cause kernel panic and rebooted the server. ---Debugger--- A debugger is not configured ---Steps to Reproduce--- After migrated from 16.04 to 18.04, will see the i/o errors. Stack trace output: no Oops output: no System Dump Info: The system is not configured to capture a system dump. Device driver error code: [52297.898232] dasd.5c0c98: 0.0.0100: default ERP has run out of retries and failed [52297.898244] dasd(fba): I/O status report for device 0.0.0100: dasd(fba): in req: 2e142bcd CS: 0x20 DS: 0x0C dasd(fba): device 0.0.0100: Failing CCW: a84f6c9f dasd(fba): SORRY - NO VALID SENSE AVAILABLE [52297.898255] dasd(fba): Related CP in req: 2e142bcd dasd(fba): CCW d15f44d3: 63400010 7FE2BBA0 DAT: 0200 008150F2 0017 dasd(fba): CCW b09ad76e: 4348 7FE2BBB0 DAT: 0506 dasd(fba): CCW c44dfb20: 41600C00 F360 DAT: dasd(fba): CCW a84f6c9f: 4348 7FE2BBB8 DAT: 0510 0006 dasd(fba): CCW 22dac88c: 4160 DAT: dasd(fba): CCW 69686544: 4348 7FE2BBC0 DAT: 0502 0016 dasd(fba): CCW 7bb6a17c: 41200400 F360 DAT: [52297.898278] print_req_error: I/O error, dev dasda, sector 8474866 *Additional Instructions for y...@cn.ibm.com: -Attach sysctl -a output output to the bug. [reply] [?] Internal Only Comment 1 QI YE 2020-03-12 02:29:20 CDT The z/VM version 6.4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1867118/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1867118] Re: dasd_fba io error and kernel panic
** Changed in: ubuntu-z-systems Assignee: (unassigned) => Skipper Bug Screeners (skipper-screen-team) ** Changed in: ubuntu-z-systems Status: Expired => In Progress ** Also affects: linux (Ubuntu Groovy) Importance: Undecided Assignee: Skipper Bug Screeners (skipper-screen-team) Status: Incomplete ** Also affects: linux (Ubuntu Focal) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Groovy) Status: Incomplete => Fix Released ** Changed in: linux (Ubuntu Focal) Status: New => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1867118 Title: dasd_fba io error and kernel panic Status in Ubuntu on IBM z Systems: In Progress Status in linux package in Ubuntu: Fix Released Status in linux source package in Focal: In Progress Status in linux source package in Groovy: Fix Released Bug description: Generated I/O errors every hour in syslog and caused kernel panic with workloads running ---uname output--- Linux ubuntutest01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:02 UTC 2020 s390x s390x s390x GNU/Linux ---Patches Installed--- distro provided. up-to-date from Ubuntu18.04. Machine Type = z15 ---System Hang--- The hourly generated I/O errors didn't impact the system up and running. When there are workloads on it, it will cause kernel panic and rebooted the server. ---Debugger--- A debugger is not configured ---Steps to Reproduce--- After migrated from 16.04 to 18.04, will see the i/o errors. Stack trace output: no Oops output: no System Dump Info: The system is not configured to capture a system dump. Device driver error code: [52297.898232] dasd.5c0c98: 0.0.0100: default ERP has run out of retries and failed [52297.898244] dasd(fba): I/O status report for device 0.0.0100: dasd(fba): in req: 2e142bcd CS: 0x20 DS: 0x0C dasd(fba): device 0.0.0100: Failing CCW: a84f6c9f dasd(fba): SORRY - NO VALID SENSE AVAILABLE [52297.898255] dasd(fba): Related CP in req: 2e142bcd dasd(fba): CCW d15f44d3: 63400010 7FE2BBA0 DAT: 0200 008150F2 0017 dasd(fba): CCW b09ad76e: 4348 7FE2BBB0 DAT: 0506 dasd(fba): CCW c44dfb20: 41600C00 F360 DAT: dasd(fba): CCW a84f6c9f: 4348 7FE2BBB8 DAT: 0510 0006 dasd(fba): CCW 22dac88c: 4160 DAT: dasd(fba): CCW 69686544: 4348 7FE2BBC0 DAT: 0502 0016 dasd(fba): CCW 7bb6a17c: 41200400 F360 DAT: [52297.898278] print_req_error: I/O error, dev dasda, sector 8474866 *Additional Instructions for y...@cn.ibm.com: -Attach sysctl -a output output to the bug. [reply] [?] Internal Only Comment 1 QI YE 2020-03-12 02:29:20 CDT The z/VM version 6.4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1867118/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1867118] Re: dasd_fba io error and kernel panic
I just 'found' that it's included in upstream stable update 5.4.69: https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.4.69 So it will find it's way into focal via this route. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1867118 Title: dasd_fba io error and kernel panic Status in Ubuntu on IBM z Systems: Expired Status in linux package in Ubuntu: Incomplete Bug description: Generated I/O errors every hour in syslog and caused kernel panic with workloads running ---uname output--- Linux ubuntutest01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:02 UTC 2020 s390x s390x s390x GNU/Linux ---Patches Installed--- distro provided. up-to-date from Ubuntu18.04. Machine Type = z15 ---System Hang--- The hourly generated I/O errors didn't impact the system up and running. When there are workloads on it, it will cause kernel panic and rebooted the server. ---Debugger--- A debugger is not configured ---Steps to Reproduce--- After migrated from 16.04 to 18.04, will see the i/o errors. Stack trace output: no Oops output: no System Dump Info: The system is not configured to capture a system dump. Device driver error code: [52297.898232] dasd.5c0c98: 0.0.0100: default ERP has run out of retries and failed [52297.898244] dasd(fba): I/O status report for device 0.0.0100: dasd(fba): in req: 2e142bcd CS: 0x20 DS: 0x0C dasd(fba): device 0.0.0100: Failing CCW: a84f6c9f dasd(fba): SORRY - NO VALID SENSE AVAILABLE [52297.898255] dasd(fba): Related CP in req: 2e142bcd dasd(fba): CCW d15f44d3: 63400010 7FE2BBA0 DAT: 0200 008150F2 0017 dasd(fba): CCW b09ad76e: 4348 7FE2BBB0 DAT: 0506 dasd(fba): CCW c44dfb20: 41600C00 F360 DAT: dasd(fba): CCW a84f6c9f: 4348 7FE2BBB8 DAT: 0510 0006 dasd(fba): CCW 22dac88c: 4160 DAT: dasd(fba): CCW 69686544: 4348 7FE2BBC0 DAT: 0502 0016 dasd(fba): CCW 7bb6a17c: 41200400 F360 DAT: [52297.898278] print_req_error: I/O error, dev dasda, sector 8474866 *Additional Instructions for y...@cn.ibm.com: -Attach sysctl -a output output to the bug. [reply] [?] Internal Only Comment 1 QI YE 2020-03-12 02:29:20 CDT The z/VM version 6.4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1867118/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1867118] Re: dasd_fba io error and kernel panic
Ican see that this patch was brought upstream on 20200923 and was picked up with v5.9-rc7, hence it's first of all not in stock 5.4 or 5.8. Digging a bit deeper into this I found that it already landed in groovy (Ubuntu-5.8.0-21.22) via: "Groovy update: v5.8.13 upstream stable release" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1898076 Hence it was obviously tagged upstream as stable update for the 5.8 series. But I couldn't find it in current focal master-next. So the questions is: was it also tagged as upstream stable update for the 5.4 series (which would be recommended)? Or does it need to be manually SRUed to focal? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1867118 Title: dasd_fba io error and kernel panic Status in Ubuntu on IBM z Systems: Expired Status in linux package in Ubuntu: Incomplete Bug description: Generated I/O errors every hour in syslog and caused kernel panic with workloads running ---uname output--- Linux ubuntutest01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:02 UTC 2020 s390x s390x s390x GNU/Linux ---Patches Installed--- distro provided. up-to-date from Ubuntu18.04. Machine Type = z15 ---System Hang--- The hourly generated I/O errors didn't impact the system up and running. When there are workloads on it, it will cause kernel panic and rebooted the server. ---Debugger--- A debugger is not configured ---Steps to Reproduce--- After migrated from 16.04 to 18.04, will see the i/o errors. Stack trace output: no Oops output: no System Dump Info: The system is not configured to capture a system dump. Device driver error code: [52297.898232] dasd.5c0c98: 0.0.0100: default ERP has run out of retries and failed [52297.898244] dasd(fba): I/O status report for device 0.0.0100: dasd(fba): in req: 2e142bcd CS: 0x20 DS: 0x0C dasd(fba): device 0.0.0100: Failing CCW: a84f6c9f dasd(fba): SORRY - NO VALID SENSE AVAILABLE [52297.898255] dasd(fba): Related CP in req: 2e142bcd dasd(fba): CCW d15f44d3: 63400010 7FE2BBA0 DAT: 0200 008150F2 0017 dasd(fba): CCW b09ad76e: 4348 7FE2BBB0 DAT: 0506 dasd(fba): CCW c44dfb20: 41600C00 F360 DAT: dasd(fba): CCW a84f6c9f: 4348 7FE2BBB8 DAT: 0510 0006 dasd(fba): CCW 22dac88c: 4160 DAT: dasd(fba): CCW 69686544: 4348 7FE2BBC0 DAT: 0502 0016 dasd(fba): CCW 7bb6a17c: 41200400 F360 DAT: [52297.898278] print_req_error: I/O error, dev dasda, sector 8474866 *Additional Instructions for y...@cn.ibm.com: -Attach sysctl -a output output to the bug. [reply] [?] Internal Only Comment 1 QI YE 2020-03-12 02:29:20 CDT The z/VM version 6.4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1867118/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1867118] Re: dasd_fba io error and kernel panic
[Expired for Ubuntu on IBM z Systems because there has been no activity for 60 days.] ** Changed in: ubuntu-z-systems Status: Incomplete => Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1867118 Title: dasd_fba io error and kernel panic Status in Ubuntu on IBM z Systems: Expired Status in linux package in Ubuntu: Incomplete Bug description: Generated I/O errors every hour in syslog and caused kernel panic with workloads running ---uname output--- Linux ubuntutest01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:02 UTC 2020 s390x s390x s390x GNU/Linux ---Patches Installed--- distro provided. up-to-date from Ubuntu18.04. Machine Type = z15 ---System Hang--- The hourly generated I/O errors didn't impact the system up and running. When there are workloads on it, it will cause kernel panic and rebooted the server. ---Debugger--- A debugger is not configured ---Steps to Reproduce--- After migrated from 16.04 to 18.04, will see the i/o errors. Stack trace output: no Oops output: no System Dump Info: The system is not configured to capture a system dump. Device driver error code: [52297.898232] dasd.5c0c98: 0.0.0100: default ERP has run out of retries and failed [52297.898244] dasd(fba): I/O status report for device 0.0.0100: dasd(fba): in req: 2e142bcd CS: 0x20 DS: 0x0C dasd(fba): device 0.0.0100: Failing CCW: a84f6c9f dasd(fba): SORRY - NO VALID SENSE AVAILABLE [52297.898255] dasd(fba): Related CP in req: 2e142bcd dasd(fba): CCW d15f44d3: 63400010 7FE2BBA0 DAT: 0200 008150F2 0017 dasd(fba): CCW b09ad76e: 4348 7FE2BBB0 DAT: 0506 dasd(fba): CCW c44dfb20: 41600C00 F360 DAT: dasd(fba): CCW a84f6c9f: 4348 7FE2BBB8 DAT: 0510 0006 dasd(fba): CCW 22dac88c: 4160 DAT: dasd(fba): CCW 69686544: 4348 7FE2BBC0 DAT: 0502 0016 dasd(fba): CCW 7bb6a17c: 41200400 F360 DAT: [52297.898278] print_req_error: I/O error, dev dasda, sector 8474866 *Additional Instructions for y...@cn.ibm.com: -Attach sysctl -a output output to the bug. [reply] [?] Internal Only Comment 1 QI YE 2020-03-12 02:29:20 CDT The z/VM version 6.4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1867118/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1867118] Re: dasd_fba io error and kernel panic
** Tags removed: targetmilestone-inin--- ** Tags added: targetmilestone-inin18044 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1867118 Title: dasd_fba io error and kernel panic Status in Ubuntu on IBM z Systems: Incomplete Status in linux package in Ubuntu: Incomplete Bug description: Generated I/O errors every hour in syslog and caused kernel panic with workloads running ---uname output--- Linux ubuntutest01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:02 UTC 2020 s390x s390x s390x GNU/Linux ---Patches Installed--- distro provided. up-to-date from Ubuntu18.04. Machine Type = z15 ---System Hang--- The hourly generated I/O errors didn't impact the system up and running. When there are workloads on it, it will cause kernel panic and rebooted the server. ---Debugger--- A debugger is not configured ---Steps to Reproduce--- After migrated from 16.04 to 18.04, will see the i/o errors. Stack trace output: no Oops output: no System Dump Info: The system is not configured to capture a system dump. Device driver error code: [52297.898232] dasd.5c0c98: 0.0.0100: default ERP has run out of retries and failed [52297.898244] dasd(fba): I/O status report for device 0.0.0100: dasd(fba): in req: 2e142bcd CS: 0x20 DS: 0x0C dasd(fba): device 0.0.0100: Failing CCW: a84f6c9f dasd(fba): SORRY - NO VALID SENSE AVAILABLE [52297.898255] dasd(fba): Related CP in req: 2e142bcd dasd(fba): CCW d15f44d3: 63400010 7FE2BBA0 DAT: 0200 008150F2 0017 dasd(fba): CCW b09ad76e: 4348 7FE2BBB0 DAT: 0506 dasd(fba): CCW c44dfb20: 41600C00 F360 DAT: dasd(fba): CCW a84f6c9f: 4348 7FE2BBB8 DAT: 0510 0006 dasd(fba): CCW 22dac88c: 4160 DAT: dasd(fba): CCW 69686544: 4348 7FE2BBC0 DAT: 0502 0016 dasd(fba): CCW 7bb6a17c: 41200400 F360 DAT: [52297.898278] print_req_error: I/O error, dev dasda, sector 8474866 *Additional Instructions for y...@cn.ibm.com: -Attach sysctl -a output output to the bug. [reply] [?] Internal Only Comment 1 QI YE 2020-03-12 02:29:20 CDT The z/VM version 6.4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1867118/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1867118] Re: dasd_fba io error and kernel panic
I tried to re-create the situation over here. I have a z/VM 6.4: q cplevel z/VM Version 6 Release 4.0, service level 1901 (64-bit) Generated at 06/14/19 14:15:49 UTC and there is a VM guest running from the time I did 18.04.4 regression testing, hence it's up since quite some time: buntu@zbox:~$ uptime 05:16:47 up 36 days, 22:17, 1 user, load average: 0.00, 0.00, 0.00 but has an older kernel: ubuntu@zbox:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 18.04.4 LTS Release:18.04 Codename: bionic ubuntu@zbox:~$ uname -a Linux zbox 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:23:44 UTC 2020 s390x s390x s390x GNU/Linux ubuntu@zbox:~$ It has 4 FBA devices: $ lszdev | grep yes dasd-eckd0.0.0200yes yes dasda dasd-fba 0.0.0101yes yes dasdc dasd-fba 0.0.0102yes yes dasdb dasd-fba 0.0.0103yes yes dasdd dasd-fba 0.0.0104yes yes dasde qeth 0.0.0600:0.0.0601:0.0.0602 yes yes enc600 generic-ccw 0.0.0009yes yes I had a look at the logs (dmesg,syslog) and couldn't find any problems with FBA devices: ubuntu@zbox:~$ dmesg -xT -l emerg,alert,crit,err ubuntu@zbox:~$ grep 'error\|fail\|crash\|warn\|crit' /var/log/syslog Mar 12 06:57:46 zbox systemd-resolved[371]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. Mar 12 09:21:16 zbox systemd-resolved[371]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. ubuntu@zbox:~$ So I upgraded the system to the latest package and kernel level: ubuntu@zbox:~$ sudo apt -y -q update && sudo apt -y -q full-upgrade && sudo apt autoremove --purge ubuntu@zbox:~$ lsb_release -r Release:18.04 ubuntu@zbox:~$ uname -a Linux zbox 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:02 UTC 2020 s390x s390x s390x GNU/Linux ubuntu@zbox:~$ ubuntu@zbox:~$ dmesg -xT -l emerg,alert,crit,err And I still cannot find any FBA issues: ubuntu@zbox:~$ grep 'error\|fail\|crash\|warn\|crit' /var/log/syslog Mar 12 06:57:46 zbox systemd-resolved[371]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. Mar 12 09:21:16 zbox systemd-resolved[371]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. ubuntu@zbox:~$ You mentioned that things might happen under load, so I created and copied files with dd, and ran bonnie++ to create some more disk utilization and even ran partially stress-ng in parallel, but still cannot recreate the FBA issue. In other words I was not able to recreate the problem. I work on a z13 with a DS8870. What about your system? Did you double checked that there are no issues in the environment and the FBA definition? ** Changed in: linux (Ubuntu) Status: New => Incomplete ** Changed in: ubuntu-z-systems Status: New => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1867118 Title: dasd_fba io error and kernel panic Status in Ubuntu on IBM z Systems: Incomplete Status in linux package in Ubuntu: Incomplete Bug description: Generated I/O errors every hour in syslog and caused kernel panic with workloads running ---uname output--- Linux ubuntutest01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:02 UTC 2020 s390x s390x s390x GNU/Linux ---Patches Installed--- distro provided. up-to-date from Ubuntu18.04. Machine Type = z15 ---System Hang--- The hourly generated I/O errors didn't impact the system up and running. When there are workloads on it, it will cause kernel panic and rebooted the server. ---Debugger--- A debugger is not configured ---Steps to Reproduce--- After migrated from 16.04 to 18.04, will see the i/o errors. Stack trace output: no Oops output: no System Dump Info: The system is not configured to capture a system dump. Device driver error code: [52297.898232] dasd.5c0c98: 0.0.0100: default ERP has run out of retries and failed [52297.898244] dasd(fba): I/O status report for device 0.0.0100: dasd(fba): in req: 2e142bcd CS: 0x20 DS: 0x0C dasd(fba): device 0.0.0100: Failing CCW: a84f6c9f dasd(fba): SORRY - NO VALID SENSE AVAILABLE [52297.898255] dasd(fba): Related CP in req: 2e142bcd dasd(fba): CCW d15f44d3: 63400010 7FE2BBA0 DAT: 0200 008150F2 0017 dasd(fba): CCW b09ad76e: 4348 7FE2BBB0 DAT: 0506
[Kernel-packages] [Bug 1867118] Re: dasd_fba io error and kernel panic
** Also affects: ubuntu-z-systems Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1867118 Title: dasd_fba io error and kernel panic Status in Ubuntu on IBM z Systems: New Status in linux package in Ubuntu: New Bug description: Generated I/O errors every hour in syslog and caused kernel panic with workloads running ---uname output--- Linux ubuntutest01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:02 UTC 2020 s390x s390x s390x GNU/Linux ---Patches Installed--- distro provided. up-to-date from Ubuntu18.04. Machine Type = z15 ---System Hang--- The hourly generated I/O errors didn't impact the system up and running. When there are workloads on it, it will cause kernel panic and rebooted the server. ---Debugger--- A debugger is not configured ---Steps to Reproduce--- After migrated from 16.04 to 18.04, will see the i/o errors. Stack trace output: no Oops output: no System Dump Info: The system is not configured to capture a system dump. Device driver error code: [52297.898232] dasd.5c0c98: 0.0.0100: default ERP has run out of retries and failed [52297.898244] dasd(fba): I/O status report for device 0.0.0100: dasd(fba): in req: 2e142bcd CS: 0x20 DS: 0x0C dasd(fba): device 0.0.0100: Failing CCW: a84f6c9f dasd(fba): SORRY - NO VALID SENSE AVAILABLE [52297.898255] dasd(fba): Related CP in req: 2e142bcd dasd(fba): CCW d15f44d3: 63400010 7FE2BBA0 DAT: 0200 008150F2 0017 dasd(fba): CCW b09ad76e: 4348 7FE2BBB0 DAT: 0506 dasd(fba): CCW c44dfb20: 41600C00 F360 DAT: dasd(fba): CCW a84f6c9f: 4348 7FE2BBB8 DAT: 0510 0006 dasd(fba): CCW 22dac88c: 4160 DAT: dasd(fba): CCW 69686544: 4348 7FE2BBC0 DAT: 0502 0016 dasd(fba): CCW 7bb6a17c: 41200400 F360 DAT: [52297.898278] print_req_error: I/O error, dev dasda, sector 8474866 *Additional Instructions for y...@cn.ibm.com: -Attach sysctl -a output output to the bug. [reply] [?] Internal Only Comment 1 QI YE 2020-03-12 02:29:20 CDT The z/VM version 6.4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1867118/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp