Yes, I believe your problems are network-related. I would advise taking iscsi NOPs off the table -- if you have a slow connection, the error recovery involved in a ping timeout can screw up I/O big time.
On Friday, November 26, 2021 at 6:52:45 AM UTC-8 Mauricio wrote: > Now I was able to address my issue with the testbox, I can mount the > LUN in that host without issues. So it is time to switch back to the > problem box, which started having issues since the last reboot. I apply the > solution used in the testbox and then restart the service: > > [root@problembox ~]# systemctl restart iscsi > [root@problembox ~]# > > And it acts like it is happy (so far; did not check dmesg or fdisk): > > [root@problembox ~]# systemctl status iscsi > o iscsi.service - Login and scanning of iSCSI devices > Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled; vendor > preset: disabled) > Active: active (exited) since Thu 2021-11-25 23:21:40 EST; 9h ago > Docs: man:iscsiadm(8) > man:iscsid(8) > Process: 3414 ExecStart=/usr/sbin/iscsiadm -m node --loginall=automatic > (code=exited, status=0/SUCCESS) > Main PID: 3414 (code=exited, status=0/SUCCESS) > Tasks: 0 (limit: 203741) > Memory: 0B > CGroup: /system.slice/iscsi.service > > Nov 25 23:17:52 problembox systemd[1]: Starting Login and scanning of > iSCSI devices... > Nov 25 23:21:40 problembox iscsiadm[3414]: Logging in to [iface: default, > target: iqn.2000-01.com.synology-iSCSI:storage.01, portal: > 192.168.10.18,3260] > Nov 25 23:21:40 problembox iscsiadm[3414]: Login to [iface: default, > target: iqn.2000-01.com.synology-iSCSI:storage.01, portal: > 192.168.10.18,3260] successful. > Nov 25 23:21:40 problembox systemd[1]: Started Login and scanning of iSCSI > devices. > [root@problembox ~]# > > [root@problembox ~]# ls -lh /dev/sd* > brw-rw----. 1 root disk 8, 0 Nov 25 21:42 /dev/sda > brw-rw----. 1 root disk 8, 1 Nov 25 21:42 /dev/sda1 > brw-rw----. 1 root disk 8, 2 Nov 25 21:42 /dev/sda2 > brw-rw----. 1 root disk 8, 3 Nov 25 21:42 /dev/sda3 > brw-rw----. 1 root disk 8, 16 Nov 25 23:33 /dev/sdb > [root@problembox ~]# ls -l /dev/disk/by-path/|grep ip > lrwxrwxrwx. 1 root root 9 Nov 25 23:33 > ip-192.168.10.18:3260-iscsi-iqn.2000-01.com.synology-iSCSI:storage.01-lun-0 > -> ../../sdb > [root@problembox ~]# > > Time to go probe the elephant in the room > > [root@problembox ~]# fdisk -l /dev/sdb > fdisk: cannot open /dev/sdb: Input/output error > [root@problembox ~]# > > What does dmesg has to tell me? The expected behaviour as seen in the > testbox (mounting the very same LUN): > > [root@testbox ~]# dmesg -T > [...] > [Thu Nov 25 19:58:00 2021] Loading iSCSI transport class v2.0-870. > [Thu Nov 25 19:58:00 2021] iscsi: registered transport (tcp) > [Thu Nov 25 19:58:00 2021] scsi host2: iSCSI Initiator over TCP/IP > [Thu Nov 25 19:58:00 2021] scsi 2:0:0:0: Direct-Access SYNOLOGY iSCSI > Storage 3.1 PQ: 0 ANSI: 5 > [Thu Nov 25 19:58:00 2021] scsi 2:0:0:0: alua: supports implicit TPGS > [Thu Nov 25 19:58:00 2021] scsi 2:0:0:0: alua: device > naa.6001405e61f8c59d35fdd4481da3e1d3 port group 1 rel port 1 > [Thu Nov 25 19:58:00 2021] scsi 2:0:0:0: Attached scsi generic sg1 type 0 > [Thu Nov 25 19:58:00 2021] scsi 2:0:0:0: alua: transition timeout set to > 60 seconds > [Thu Nov 25 19:58:00 2021] scsi 2:0:0:0: alua: port group 01 state A > non-preferred supports TOlUSNA > [Thu Nov 25 19:58:00 2021] sd 2:0:0:0: [sda] 754974720 512-byte logical > blocks: (387 GB/360 GiB) > [Thu Nov 25 19:58:00 2021] sd 2:0:0:0: [sda] Write Protect is off > [Thu Nov 25 19:58:00 2021] sd 2:0:0:0: [sda] Mode Sense: 3b 00 00 00 > [Thu Nov 25 19:58:00 2021] sd 2:0:0:0: [sda] Write cache: disabled, read > cache: enabled, doesn't support DPO or FUA > [Thu Nov 25 19:58:00 2021] sda: sda1 > [Thu Nov 25 19:58:00 2021] sd 2:0:0:0: [sda] Attached SCSI disk > [root@testbox ~]# > > Behaviour seen in the problembox > > [root@problembox ~]# dmesg -T > [Thu Nov 25 23:17:51 2021] scsi host8: iSCSI Initiator over TCP/IP > [Thu Nov 25 23:17:51 2021] scsi 8:0:0:0: Direct-Access SYNOLOGY iSCSI > Storage 3.1 PQ: 0 ANSI: 5 > [Thu Nov 25 23:17:51 2021] scsi 8:0:0:0: alua: supports implicit TPGS > [Thu Nov 25 23:17:51 2021] scsi 8:0:0:0: alua: device > naa.6001405e61f8c59d35fdd4481da3e1d3 port group 1 rel port 1 > [Thu Nov 25 23:17:51 2021] sd 8:0:0:0: Attached scsi generic sg1 type 0 > [Thu Nov 25 23:18:02 2021] connection4:0: ping timeout of 5 secs expired, > recv timeout 5, last rx 4300399244, last ping 4300404736, now 4300409856 > [Thu Nov 25 23:18:02 2021] connection4:0: detected conn error (1022) > [...] > [Thu Nov 25 23:31:56 2021] connection4:0: detected conn error (1022) > [Thu Nov 25 23:31:56 2021] sd 8:0:0:0: [sdb] tag#76 FAILED Result: > hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=72s > [Thu Nov 25 23:31:56 2021] sd 8:0:0:0: [sdb] tag#76 CDB: Read(10) 28 00 2c > ff ff 80 00 00 08 00 > [Thu Nov 25 23:31:56 2021] blk_update_request: I/O error, dev sdb, sector > 754974592 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [Thu Nov 25 23:32:10 2021] connection4:0: ping timeout of 5 secs expired, > recv timeout 5, last rx 4301247437, last ping 4301252608, now 4301257728 > [Thu Nov 25 23:32:10 2021] connection4:0: detected conn error (1022) > [Thu Nov 25 23:32:22 2021] connection4:0: ping timeout of 5 secs expired, > recv timeout 5, last rx 4301259759, last ping 4301264896, now 4301270016 > [Thu Nov 25 23:32:22 2021] connection4:0: detected conn error (1022) > [Thu Nov 25 23:32:35 2021] connection4:0: ping timeout of 5 secs expired, > recv timeout 5, last rx 4301273035, last ping 4301278208, now 4301283328 > [Thu Nov 25 23:32:35 2021] connection4:0: detected conn error (1022) > [Thu Nov 25 23:32:48 2021] connection4:0: ping timeout of 5 secs expired, > recv timeout 5, last rx 4301285359, last ping 4301290496, now 4301295616 > [Thu Nov 25 23:32:48 2021] connection4:0: detected conn error (1022) > [Thu Nov 25 23:33:00 2021] connection4:0: ping timeout of 5 secs expired, > recv timeout 5, last rx 4301297653, last ping 4301302784, now 4301307904 > [Thu Nov 25 23:33:00 2021] connection4:0: detected conn error (1022) > [Thu Nov 25 23:33:13 2021] connection4:0: ping timeout of 5 secs expired, > recv timeout 5, last rx 4301310922, last ping 4301316096, now 4301321216 > [Thu Nov 25 23:33:13 2021] connection4:0: detected conn error (1022) > [Thu Nov 25 23:33:13 2021] sd 8:0:0:0: [sdb] tag#81 FAILED Result: > hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=73s > [Thu Nov 25 23:33:13 2021] sd 8:0:0:0: [sdb] tag#81 CDB: Read(10) 28 00 2c > ff ff 80 00 00 08 00 > [Thu Nov 25 23:33:13 2021] blk_update_request: I/O error, dev sdb, sector > 754974592 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [Thu Nov 25 23:33:13 2021] Buffer I/O error on dev sdb, logical block > 94371824, async page read > [Fri Nov 26 00:46:06 2021] connection4:0: ping timeout of 5 secs expired, > recv timeout 5, last rx 4305683173, last ping 4305688576, now 4305693696 > [Fri Nov 26 00:46:06 2021] connection4:0: detected conn error (1022) > [root@problembox ~]# > > Given that problembox is a baremetal server and testbox is a vm guest in a > KVM server which is connected to the same switch as problembox, and that > problembox has been working fine for 2 years until the very last reboot, > are these network issues or something else. > -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/open-iscsi/a3dc8390-aac6-47ca-89d5-25d65a7c17e8n%40googlegroups.com.
