Re: Removing the transport module before iscsid
Mike Christie wrote: > Erez Zilber wrote: > >> Mike Christie wrote: >> >>> Erez Zilber wrote: >>> >>> Mike, We're trying to solve a problem, but first the motivation: a user may want to stop the InfiniBand modules (/etc/init.d/openibd stop) while open-iscsi is running. In order to do that, we must remove ib_iser. Now, the question: is it ok to modprobe -r ib_iser (or iscsi_tcp) before >>> It is and it isn't :) >>> >>> >>> stopping iscsid? From what I see in the init.d scripts in open-iscsi, we kill iscsid first. Is there a reason for that? >>> We kill iscsid first because it has a ref to scsi_transport_iscsi (load >>> that module and run iscsid and then do a lsmod). So if you ran iscsid, >>> then just modprobe -r ib_iser, it would leave scsi_transport_iscsi loaded. >>> >>> >>> >> If I run /etc/init.d/openibd stop, it will (with a patch that I'm going >> to submit) call iscsi_conn_failure for all iSER connections, and then >> remove ib_iser. scsi_transport_iscsi will be removed later when >> > > > This does not sound right or even possible with current kernels. If a > session is runing and scsi_devices exists for it and if a user has a > device open then there are references to the ib_iser module. > > If the user does a close then it needs the scsi_host_tempalte that was > allocated in the ib_iser module. For libiscsi/scsi_transport_iscsi there > is similar problems with the those modules accessing the iscsi_transport > struct allocated in the module. > > The thing is that scsi-ml should be incrementing the ib_iser refcount if > a device is open. And open-iscsi should be incrementing the ib_iser > refctoun if there is a session. So you should not be able to remove the > ib_iser module if there are connections. > > I'm adding Vlad & openfabrics-ewg to this discussion, as it seems relevant for OFED. Mike - you're right. I guess that I missed something here. My patch (http://groups.google.com/group/open-iscsi/browse_frm/thread/2c99cce2956ac4d7) only allows the removal of the HCA driver (e.g. ib_mthca). This is better than the BUG() call that we had there until now. The motivation was to allow users to stop openibd even while open-iscsi is running. The problem is that even after removing the HCA driver, ib_iser is still referenced and cannot be unloaded (unless /etc/init.d/open-iscsi stop is called). Therefore, most of the IB stack cannot be unloaded (rdma_cm, ib_cm, ib_core etc). Here's my suggestion: in addition to this patch (which is good if a user decides to remove only ib_mthca), we should add some code to /etc/init.d/openibd that will check if open-iscsi is loaded and if there are iSER sessions. If yes, openibd will not stop until the user logs out from the iSER sessions. When there are no more iSER sessions, openibd will stop and will also remove ib_iser which is required for the removal of other IB modules. Mike - the removal of ib_iser will require the other patch that we talked about (deleting the iSER transport entry from the transport list). If everyone's ok with this, I will make the required changes and send a patch. Erez --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Removing the transport module before iscsid
Erez Zilber wrote: > Mike Christie wrote: > >> There is also a bug in iscsid (iscsi_sysfs.c) where if iscsid had >> detected ib_iser it would grab (iscsi_sysfs.c:read_transports) the >> transport handle for it. And if you were to unload and reload it, then >> iscsid is not picking up the new transport handle (transport handle is >> just the pointer value of that iscsi_transport remember). So if you made >> the module loading and unloading more dynamic but did not restart >> iscsid, you have to just tweak iscsi_sysfs.c:read_transports to pick up >> the new handles. >> > > So all you need to do is delete the iSER transport entry from the > transports list (defined in iscsi_sysfs.h), right? When iSER is loaded > again, it won't find its entry in the list and will recreate it. > Yeah. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Removing the transport module before iscsid
Erez Zilber wrote: > Mike Christie wrote: >> Erez Zilber wrote: >> >>> Mike, >>> >>> We're trying to solve a problem, but first the motivation: a user may >>> want to stop the InfiniBand modules (/etc/init.d/openibd stop) while >>> open-iscsi is running. In order to do that, we must remove ib_iser. >>> >>> Now, the question: is it ok to modprobe -r ib_iser (or iscsi_tcp) before >>> >> It is and it isn't :) >> >> >>> stopping iscsid? From what I see in the init.d scripts in open-iscsi, we >>> kill iscsid first. Is there a reason for that? >>> >>> >> We kill iscsid first because it has a ref to scsi_transport_iscsi (load >> that module and run iscsid and then do a lsmod). So if you ran iscsid, >> then just modprobe -r ib_iser, it would leave scsi_transport_iscsi loaded. >> >> > > If I run /etc/init.d/openibd stop, it will (with a patch that I'm going > to submit) call iscsi_conn_failure for all iSER connections, and then > remove ib_iser. scsi_transport_iscsi will be removed later when This does not sound right or even possible with current kernels. If a session is runing and scsi_devices exists for it and if a user has a device open then there are references to the ib_iser module. If the user does a close then it needs the scsi_host_tempalte that was allocated in the ib_iser module. For libiscsi/scsi_transport_iscsi there is similar problems with the those modules accessing the iscsi_transport struct allocated in the module. The thing is that scsi-ml should be incrementing the ib_iser refcount if a device is open. And open-iscsi should be incrementing the ib_iser refctoun if there is a session. So you should not be able to remove the ib_iser module if there are connections. > /etc/init.d/iscsi stop is called. BTW - libiscsi will also remain loaded > and will be removed when iscsi is stopped. > >> The iscsi script will also remove the running sessions, before killing >> iscsid, because if there is a session running then ib_iser would also >> have a module refcount greater than 0, and the modprobe -r would fail. >> > > This won't happen with the new patch (will send it today). I do not see how your patch works. As far as libiscsi knows there are still sessions right? Is the openib script stopping sessions and unmounting file systems? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Removing the transport module before iscsid
Mike Christie wrote: > There is also a bug in iscsid (iscsi_sysfs.c) where if iscsid had > detected ib_iser it would grab (iscsi_sysfs.c:read_transports) the > transport handle for it. And if you were to unload and reload it, then > iscsid is not picking up the new transport handle (transport handle is > just the pointer value of that iscsi_transport remember). So if you made > the module loading and unloading more dynamic but did not restart > iscsid, you have to just tweak iscsi_sysfs.c:read_transports to pick up > the new handles. > So all you need to do is delete the iSER transport entry from the transports list (defined in iscsi_sysfs.h), right? When iSER is loaded again, it won't find its entry in the list and will recreate it. Erez --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Removing the transport module before iscsid
Mike Christie wrote: > Erez Zilber wrote: > >> Mike, >> >> We're trying to solve a problem, but first the motivation: a user may >> want to stop the InfiniBand modules (/etc/init.d/openibd stop) while >> open-iscsi is running. In order to do that, we must remove ib_iser. >> >> Now, the question: is it ok to modprobe -r ib_iser (or iscsi_tcp) before >> > > It is and it isn't :) > > >> stopping iscsid? From what I see in the init.d scripts in open-iscsi, we >> kill iscsid first. Is there a reason for that? >> >> > > We kill iscsid first because it has a ref to scsi_transport_iscsi (load > that module and run iscsid and then do a lsmod). So if you ran iscsid, > then just modprobe -r ib_iser, it would leave scsi_transport_iscsi loaded. > > If I run /etc/init.d/openibd stop, it will (with a patch that I'm going to submit) call iscsi_conn_failure for all iSER connections, and then remove ib_iser. scsi_transport_iscsi will be removed later when /etc/init.d/iscsi stop is called. BTW - libiscsi will also remain loaded and will be removed when iscsi is stopped. > The iscsi script will also remove the running sessions, before killing > iscsid, because if there is a session running then ib_iser would also > have a module refcount greater than 0, and the modprobe -r would fail. > This won't happen with the new patch (will send it today). > There is also a bug in iscsid (iscsi_sysfs.c) where if iscsid had > detected ib_iser it would grab (iscsi_sysfs.c:read_transports) the > transport handle for it. And if you were to unload and reload it, then > iscsid is not picking up the new transport handle (transport handle is > just the pointer value of that iscsi_transport remember). So if you made > the module loading and unloading more dynamic but did not restart > iscsid, you have to just tweak iscsi_sysfs.c:read_transports to pick up > the new handles. > I will take a look at it and try to send a patch. Erez --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Removing the transport module before iscsid
Erez Zilber wrote: > Mike, > > We're trying to solve a problem, but first the motivation: a user may > want to stop the InfiniBand modules (/etc/init.d/openibd stop) while > open-iscsi is running. In order to do that, we must remove ib_iser. > > Now, the question: is it ok to modprobe -r ib_iser (or iscsi_tcp) before It is and it isn't :) > stopping iscsid? From what I see in the init.d scripts in open-iscsi, we > kill iscsid first. Is there a reason for that? > We kill iscsid first because it has a ref to scsi_transport_iscsi (load that module and run iscsid and then do a lsmod). So if you ran iscsid, then just modprobe -r ib_iser, it would leave scsi_transport_iscsi loaded. The iscsi script will also remove the running sessions, before killing iscsid, because if there is a session running then ib_iser would also have a module refcount greater than 0, and the modprobe -r would fail. There is also a bug in iscsid (iscsi_sysfs.c) where if iscsid had detected ib_iser it would grab (iscsi_sysfs.c:read_transports) the transport handle for it. And if you were to unload and reload it, then iscsid is not picking up the new transport handle (transport handle is just the pointer value of that iscsi_transport remember). So if you made the module loading and unloading more dynamic but did not restart iscsid, you have to just tweak iscsi_sysfs.c:read_transports to pick up the new handles. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Removing the transport module before iscsid
Mike, We're trying to solve a problem, but first the motivation: a user may want to stop the InfiniBand modules (/etc/init.d/openibd stop) while open-iscsi is running. In order to do that, we must remove ib_iser. Now, the question: is it ok to modprobe -r ib_iser (or iscsi_tcp) before stopping iscsid? From what I see in the init.d scripts in open-iscsi, we kill iscsid first. Is there a reason for that? Thanks, Erez --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
