Hello, Thanks. On the app side, with iSCSI SANs I extend the disk\ timeout value in the OS. To better handle any transitory network events, and controller failovers. In linux that's important to prevent filesystems like EXT4 from remounting RO on an error.
I would like to know which vendor they are using for iSCSI storage. Regards, Don On Wed, Feb 16, 2022 at 5:12 AM Ulrich Windl < [email protected]> wrote: > >>> Donald Williams <[email protected]> schrieb am 15.02.2022 um > 17:25 in > Nachricht > <cak3e-ezbjmdhkozgiz8lnmnaz+soca+qek0kpkqm4vq4pz8...@mail.gmail.com>: > > Hello, > > Something else to check is your MPIO configuration. I have seen this > > same symptom when the linux MPIO feature "queue_if_no_path" was enabled > > > > From the /etc/multipath.conf file showing it enabled. > > > > failback immediate > > features "1 queue_if_no_path" > > Yes, the actual config is interesting. Especially when usind MD-RAID, you > typically do not want "1 queue_if_no_path", but if the app can't handle I/O > errors, one might want it. > For a FC SAN featuring ALUA we use: > ... > polling_interval 5 > max_polling_interval 20 > path_selector "service-time 0" > ... > path_checker "tur" > ... > fast_io_fail_tmo 5 > dev_loss_tmo 600 > > The logs are helpful, too. For example (there were some paths remaining > all the time): > Cable was unplugged: > Feb 14 12:56:05 h16 kernel: qla2xxx [0000:41:00.0]-500b:3: LOOP DOWN > detected (2 7 0 0). > Feb 14 12:56:10 h16 multipathd[5225]: sdbi: mark as failed > Feb 14 12:56:10 h16 multipathd[5225]: SAP_V11-PM: remaining active paths: 7 > Feb 14 12:56:10 h16 kernel: sd 3:0:6:3: rejecting I/O to offline device > Feb 14 12:56:10 h16 kernel: sd 3:0:6:14: rejecting I/O to offline device > Feb 14 12:56:10 h16 kernel: sd 3:0:6:15: rejecting I/O to offline device > > So 5 seconds later the paths are offlined. > > Cable was re-plugged: > Feb 14 12:56:22 h16 kernel: qla2xxx [0000:41:00.0]-500a:3: LOOP UP > detected (8 Gbps). > Feb 14 12:56:22 h16 kernel: qla2xxx [0000:41:00.0]-11a2:3: FEC=enabled > (data rate). > Feb 14 12:56:26 h16 multipathd[5225]: SAP_CJ1-PM: sdbc - tur checker > reports path is up > Feb 14 12:56:26 h16 multipathd[5225]: 67:96: reinstated > Feb 14 12:56:26 h16 multipathd[5225]: SAP_CJ1-PM: remaining active paths: 5 > Feb 14 12:56:26 h16 kernel: device-mapper: multipath: 254:4: Reinstating > path 67:96. > Feb 14 12:56:26 h16 kernel: device-mapper: multipath: 254:6: Reinstating > path 67:112. > > So 4 seconds later new paths are discovered. > > > Regards, > Ulrich > > > > > > > Also, in the past some versions of linux multipathd would wait for a > > very long time before moving all I/O to the remaining path. > > > > Regards, > > Don > > > > > > On Tue, Feb 15, 2022 at 10:49 AM Zhengyuan Liu < > [email protected]> > > wrote: > > > >> Hi, all > >> > >> We have an online server which uses multipath + iscsi to attach storage > >> from Storage Server. There are two NICs on the server and for each it > >> carries about 20 iscsi sessions and for each session it includes about > 50 > >> iscsi devices (yes, there are totally about 2*20*50=2000 iscsi block > >> devices > >> on the server). The problem is: once a NIC gets faulted, it will take > too > >> long > >> (nearly 80s) for multipath to switch to another good NIC link, because > it > >> needs to block all iscsi devices over that faulted NIC firstly. The > >> callstack is > >> shown below: > >> > >> void iscsi_block_session(struct iscsi_cls_session *session) > >> { > >> queue_work(iscsi_eh_timer_workq, &session->block_work); > >> } > >> > >> __iscsi_block_session() -> scsi_target_block() -> target_block() -> > >> device_block() -> scsi_internal_device_block() -> scsi_stop_queue() > -> > >> blk_mq_quiesce_queue()>synchronize_rcu() > >> > >> For all sessions and all devices, it was processed sequentially, and we > >> have > >> traced that for each synchronize_rcu() call it takes about 80ms, so > >> the total cost > >> is about 80s (80ms * 20 * 50). It's so long that the application can't > >> tolerate and > >> may interrupt service. > >> > >> So my question is that can we optimize the procedure to reduce the time > >> cost on > >> blocking all iscsi devices? I'm not sure if it is a good idea to > increase > >> the > >> workqueue's max_active of iscsi_eh_timer_workq to improve concurrency. > >> > >> Thanks in advance. > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "open-iscsi" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to [email protected]. > >> To view this discussion on the web visit > >> > > > https://groups.google.com/d/msgid/open-iscsi/CAOOPZo4uNCicVmoHa2za0%3DO1_XiBd > > tBvTuUzqBTeBc3FmDqEJw%40mail.gmail.com > >> . > >> > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "open-iscsi" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected]. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/open-iscsi/CAK3e-EZbJMDHkozGiz8LnMNAZ%2BSoC > > A%2BQeK0kpkqM4vQ4pz86SQ%40mail.gmail.com. > > > > -- > You received this message because you are subscribed to the Google Groups > "open-iscsi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/open-iscsi/620CCE20020000A100047D30%40gwsmtp.uni-regensburg.de > . > -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/open-iscsi/CAK3e-EbuwT%3DeMxTOGY_h9SO-RVrW2xA1sFfvQP_ijWJ1vYM3vg%40mail.gmail.com.
