I can't run this command right now as it's currently in production. Will try to run it soon.
System is a Dell R7515, AMD Epyc 7543P, 256GB RAM -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2023674 Title: E810 Ubuntu 20.04 LTS - Crash - Device is in unrecoverable state Status in linux package in Ubuntu: Incomplete Bug description: Hi, On intensive load, the interface goes in "unrecoverable state". We already using the latest firmware from Dell https://www.dell.com/support/home/en- ca/drivers/driversdetails?driverid=25ffj which is supposed to prevent that issue "- Resolved an issue with E810 adapters where the queues are not freed up properly and causes kernel issues under heavy traffic. {208689}" Here are the syslog when the bug appears: @timestamp type message program 2023-04-11T23:34:01.406Z syslog [6833005.040087] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:34:01.406Z syslog [6833005.040086] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 911, hung_queue 22 kernel 2023-04-11T23:34:01.406Z syslog [6833005.040085] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:33:56.290Z syslog [6832999.924350] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:33:56.290Z syslog [6832999.924349] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 910, hung_queue 22 kernel 2023-04-11T23:33:56.290Z syslog [6832999.924347] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:33:46.365Z syslog [6832989.936862] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:33:46.365Z syslog [6832989.936864] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 909, hung_queue 22 kernel 2023-04-11T23:33:46.365Z syslog [6832989.936864] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:33:40.414Z syslog [6832984.049162] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:33:40.414Z syslog [6832984.049164] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 908, hung_queue 22 kernel 2023-04-11T23:33:40.414Z syslog [6832984.049165] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:33:35.323Z syslog [6832978.929427] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:33:35.322Z syslog [6832978.929426] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 907, hung_queue 22 kernel 2023-04-11T23:33:35.322Z syslog [6832978.929424] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:33:29.487Z syslog [6832973.041730] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 906, hung_queue 22 kernel 2023-04-11T23:33:29.487Z syslog [6832973.041731] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:33:29.486Z syslog [6832973.041729] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:33:24.327Z syslog [6832967.921995] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:33:24.326Z syslog [6832967.921993] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:33:24.326Z syslog [6832967.921994] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 905, hung_queue 22 kernel 2023-04-11T23:33:14.306Z syslog [6832957.942505] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:33:14.306Z syslog [6832957.942502] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:33:14.306Z syslog [6832957.942504] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 904, hung_queue 22 kernel 2023-04-11T23:33:08.414Z syslog [6832952.050807] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:33:08.414Z syslog [6832952.050809] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 903, hung_queue 22 kernel 2023-04-11T23:33:08.414Z syslog [6832952.050810] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:33:03.294Z syslog [6832946.931073] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:33:03.294Z syslog [6832946.931075] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:33:03.294Z syslog [6832946.931075] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 902, hung_queue 22 kernel 2023-04-11T23:32:57.406Z syslog [6832941.043373] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:32:57.406Z syslog [6832941.043372] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 901, hung_queue 22 kernel 2023-04-11T23:32:57.406Z syslog [6832941.043371] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:32:52.286Z syslog [6832935.923637] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:32:52.286Z syslog [6832935.923639] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 900, hung_queue 22 kernel 2023-04-11T23:32:52.286Z syslog [6832935.923639] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:32:42.355Z syslog [6832925.940152] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:32:42.355Z syslog [6832925.940151] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 899, hung_queue 22 kernel 2023-04-11T23:32:42.355Z syslog [6832925.940149] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:32:36.441Z syslog [6832920.052458] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:32:36.441Z syslog [6832920.052457] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 898, hung_queue 22 kernel 2023-04-11T23:32:36.440Z syslog [6832920.052455] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel 2023-04-11T23:32:31.298Z syslog [6832914.936718] ice 0000:c4:00.3 ens3f3: tx_timeout recovery unsuccessful, device is in unrecoverable state. kernel 2023-04-11T23:32:31.298Z syslog [6832914.936717] ice 0000:c4:00.3 ens3f3: tx_timeout recovery level 897, hung_queue 22 kernel 2023-04-11T23:32:31.298Z syslog [6832914.936716] ice 0000:c4:00.3 ens3f3: tx_timeout: VSI_num: 15, Q 22, NTC: 0x8e, HW_HEAD: 0x8d, NTU: 0x78, INT: 0x1 kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2023674/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
