Re: Question about iscsi session block
Hi, Mike,
Sorry for the delayed reply since I have no environment to check your
bellow patcheset untile recently
https://lore.kernel.org/all/20220226230435.38733-1-michael.chris...@oracle.com/
After applied those series, the total time has dropped from 80s to
nearly 10s, it's a great improvement.
Thanks, again
On Sun, Feb 27, 2022 at 7:00 AM Mike Christie
wrote:
>
> On 2/15/22 8:19 PM, michael.chris...@oracle.com wrote:
> > On 2/15/22 7:28 PM, Zhengyuan Liu wrote:
> >> On Wed, Feb 16, 2022 at 12:31 AM Mike Christie
> >> wrote:
> >>>
> >>> On 2/15/22 9:49 AM, Zhengyuan Liu wrote:
> >>>> Hi, all
> >>>>
> >>>> We have an online server which uses multipath + iscsi to attach storage
> >>>> from Storage Server. There are two NICs on the server and for each it
> >>>> carries about 20 iscsi sessions and for each session it includes about 50
> >>>> iscsi devices (yes, there are totally about 2*20*50=2000 iscsi block
> >>>> devices
> >>>> on the server). The problem is: once a NIC gets faulted, it will take
> >>>> too long
> >>>> (nearly 80s) for multipath to switch to another good NIC link, because it
> >>>> needs to block all iscsi devices over that faulted NIC firstly. The
> >>>> callstack is
> >>>> shown below:
> >>>>
> >>>> void iscsi_block_session(struct iscsi_cls_session *session)
> >>>> {
> >>>> queue_work(iscsi_eh_timer_workq, >block_work);
> >>>> }
> >>>>
> >>>> __iscsi_block_session() -> scsi_target_block() -> target_block() ->
> >>>> device_block() -> scsi_internal_device_block() -> scsi_stop_queue() ->
> >>>> blk_mq_quiesce_queue()>synchronize_rcu()
> >>>>
> >>>> For all sessions and all devices, it was processed sequentially, and we
> >>>> have
> >>>> traced that for each synchronize_rcu() call it takes about 80ms, so
> >>>> the total cost
> >>>> is about 80s (80ms * 20 * 50). It's so long that the application can't
> >>>> tolerate and
> >>>> may interrupt service.
> >>>>
> >>>> So my question is that can we optimize the procedure to reduce the time
> >>>> cost on
> >>>> blocking all iscsi devices? I'm not sure if it is a good idea to
> >>>> increase the
> >>>> workqueue's max_active of iscsi_eh_timer_workq to improve concurrency.
> >>>
> >>> We need a patch, so the unblock call waits/cancels/flushes the block call
> >>> or
> >>> they could be running in parallel.
> >>>
> >>> I'll send a patchset later today so you can test it.
> >>
> >> I'm glad to test once you push the patchset.
> >>
> >> Thank you, Mike.
> >
> > I forgot I did this recently :)
> >
> > commit 7ce9fc5ecde0d8bd64c29baee6c5e3ce7074ec9a
> > Author: Mike Christie
> > Date: Tue May 25 13:18:09 2021 -0500
> >
> > scsi: iscsi: Flush block work before unblock
> >
> > We set the max_active iSCSI EH works to 1, so all work is going to
> > execute
> > in order by default. However, userspace can now override this in sysfs.
> > If
> > max_active > 1, we can end up with the block_work on CPU1 and
> > iscsi_unblock_session running the unblock_work on CPU2 and the session
> > and
> > target/device state will end up out of sync with each other.
> >
> > This adds a flush of the block_work in iscsi_unblock_session.
> >
> >
> > It was merged in 5.14.
>
> Hey, I found one more bug when max_active > 1. While fixing it I decided to
> just
> fix this so we can do the sessions recoveries in parallel and the user
> doesn't have
> to worry about setting max_active.
>
> I'll send a patchset and cc you.
--
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to open-iscsi+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/open-iscsi/CAOOPZo58sfhqFEMvpUfnoU1ceHDpPnagMbPvDGrDXHZHq7bZLg%40mail.gmail.com.
Re: Question about iscsi session block
On Wed, Feb 16, 2022 at 12:31 AM Mike Christie
wrote:
>
> On 2/15/22 9:49 AM, Zhengyuan Liu wrote:
> > Hi, all
> >
> > We have an online server which uses multipath + iscsi to attach storage
> > from Storage Server. There are two NICs on the server and for each it
> > carries about 20 iscsi sessions and for each session it includes about 50
> > iscsi devices (yes, there are totally about 2*20*50=2000 iscsi block
> > devices
> > on the server). The problem is: once a NIC gets faulted, it will take too
> > long
> > (nearly 80s) for multipath to switch to another good NIC link, because it
> > needs to block all iscsi devices over that faulted NIC firstly. The
> > callstack is
> > shown below:
> >
> > void iscsi_block_session(struct iscsi_cls_session *session)
> > {
> > queue_work(iscsi_eh_timer_workq, >block_work);
> > }
> >
> > __iscsi_block_session() -> scsi_target_block() -> target_block() ->
> > device_block() -> scsi_internal_device_block() -> scsi_stop_queue() ->
> > blk_mq_quiesce_queue()>synchronize_rcu()
> >
> > For all sessions and all devices, it was processed sequentially, and we have
> > traced that for each synchronize_rcu() call it takes about 80ms, so
> > the total cost
> > is about 80s (80ms * 20 * 50). It's so long that the application can't
> > tolerate and
> > may interrupt service.
> >
> > So my question is that can we optimize the procedure to reduce the time
> > cost on
> > blocking all iscsi devices? I'm not sure if it is a good idea to increase
> > the
> > workqueue's max_active of iscsi_eh_timer_workq to improve concurrency.
>
> We need a patch, so the unblock call waits/cancels/flushes the block call or
> they could be running in parallel.
>
> I'll send a patchset later today so you can test it.
I'm glad to test once you push the patchset.
Thank you, Mike.
--
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to open-iscsi+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/open-iscsi/CAOOPZo4Z2x_W7i%3DVbnm-SsDgj5PndLVtOz6MqRzQxW-NeBwhRg%40mail.gmail.com.
Question about iscsi session block
Hi, all
We have an online server which uses multipath + iscsi to attach storage
from Storage Server. There are two NICs on the server and for each it
carries about 20 iscsi sessions and for each session it includes about 50
iscsi devices (yes, there are totally about 2*20*50=2000 iscsi block devices
on the server). The problem is: once a NIC gets faulted, it will take too long
(nearly 80s) for multipath to switch to another good NIC link, because it
needs to block all iscsi devices over that faulted NIC firstly. The callstack is
shown below:
void iscsi_block_session(struct iscsi_cls_session *session)
{
queue_work(iscsi_eh_timer_workq, >block_work);
}
__iscsi_block_session() -> scsi_target_block() -> target_block() ->
device_block() -> scsi_internal_device_block() -> scsi_stop_queue() ->
blk_mq_quiesce_queue()>synchronize_rcu()
For all sessions and all devices, it was processed sequentially, and we have
traced that for each synchronize_rcu() call it takes about 80ms, so
the total cost
is about 80s (80ms * 20 * 50). It's so long that the application can't
tolerate and
may interrupt service.
So my question is that can we optimize the procedure to reduce the time cost on
blocking all iscsi devices? I'm not sure if it is a good idea to increase the
workqueue's max_active of iscsi_eh_timer_workq to improve concurrency.
Thanks in advance.
--
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to open-iscsi+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/open-iscsi/CAOOPZo4uNCicVmoHa2za0%3DO1_XiBdtBvTuUzqBTeBc3FmDqEJw%40mail.gmail.com.
Re: open-iscsi Ping timeout error.
Thanks for you tips, I would have a try as you said to disable the NOOP. I had make a XFS file system on the LUN at the Initiator side . Finally, I catch the sync_cache command was issued by XFS log infrastructure actually. When I replace the XFS with EXT2 that dmesg error don`t appear anymore during the iozone test. I think pings/Nops got no response because it still stay in TCP stack not received by the target rx-thread. The Ping time out would lead to IO failure from upper layers? or it can recovery from re-connection and continue to transfer data so upper layers applicaction can not feel the underlying IO error? 2016-05-21 0:39 GMT+08:00 The Lee-Man <leeman.dun...@gmail.com>: > Hi: > > It seems like your backend is getting busy and not replying in time when > it gets very busy. You can disable the NOOP, or you can lengthen its > interval, I believe. > > If there is a bug, it would be in the kernel target subsystem. Have you > tried the target-devel @ vger kernel mailing list? > > > On Friday, May 13, 2016 at 9:52:46 AM UTC-7, Zhengyuan Liu wrote: >> >> Hi everyone: >> I create a target using fileio as the backend storage on ARM64 server. >> The initiator reported some errors showed bellow while perform iozone test. >> >> [178444.145679] connection14:0: ping timeout of 5 secs expired, recv >> timeout 5, last rx 4339462894, last ping 4339464146, now 4339465400 >> [178444.145706] connection14:0: detected conn error (1011) >> [178469.674313] connection14:0: detected conn error (1020) >> [178504.420979] connection14:0: ping timeout of 5 secs expired, recv >> timeout 5, last rx 4339477953, last ping 4339479204, now 4339480456 >> [178504.421001] connection14:0: detected conn error (1011) >> [178532.064262] connection14:0: detected conn error (1020) >> [178564.584087] connection14:0: ping timeout of 5 secs expired, recv >> timeout 5, last rx 4339492980, last ping 4339494232, now 4339495484 >> .. >> >> I try to trace the function call of target iscsi. Then, I found the >> receiving thread of target iscsi blocked at fd_execute_sync_cache -> >> vfs_fsync_range. Further, vfs_fsync_range may takes more than 10 seconds to >> return,while initiator Ping timeout would happened after 5 seconds. >> vfs_fsync_range was call with the form vfs_fsync_range(fd_dev->fd_file, 0, >> LLONG_MAX, 1) every times which means sync all device cache. >> So, is this a bug? >> How does Initiator send sync_cache scsi command? >> Does it need to sync all device cache at once? >> Any reply would be thankful. >> > -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
open-iscsi Ping timeout error.
Hi everyone: I create a target using fileio as the backend storage on ARM64 server. The initiator reported some errors showed bellow while perform iozone test. [178444.145679] connection14:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4339462894, last ping 4339464146, now 4339465400 [178444.145706] connection14:0: detected conn error (1011) [178469.674313] connection14:0: detected conn error (1020) [178504.420979] connection14:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4339477953, last ping 4339479204, now 4339480456 [178504.421001] connection14:0: detected conn error (1011) [178532.064262] connection14:0: detected conn error (1020) [178564.584087] connection14:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4339492980, last ping 4339494232, now 4339495484 .. I try to trace the function call of target iscsi. Then, I found the receiving thread of target iscsi blocked at fd_execute_sync_cache -> vfs_fsync_range. Further, vfs_fsync_range may takes more than 10 seconds to return,while initiator Ping timeout would happened after 5 seconds. vfs_fsync_range was call with the form vfs_fsync_range(fd_dev->fd_file, 0, LLONG_MAX, 1) every times which means sync all device cache. So, is this a bug? How does Initiator send sync_cache scsi command? Does it need to sync all device cache at once? Any reply would be thankful. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
