I appear to be having a similar issue with a Citrix XenServer host. Using the XenAPI I attempt to launch a number of VMs at the same time. Each VM is backed by it's own iscsi target so there is a bunch of logging in, discovery and session listing during this time. Roughly 50% of the launches fail due to iscsiadm -m session (called by XenServer, presumably to get a list of sessions) returning a response code of '5' when enumerating targets. My theory is that it gets to a session that's in the process of being changed by one of the other operations and without any locking it just reads the half-populated data and falls over.
If I do the same operations (via the XenAPI) one at a time it does not encounter any issues. This problem I'm having seems very similar to the problem described in this thread and a couple other recent threads I've seen on the list. Unfortunately Citrix isn't being very helpful with this issue and I'd like to avoid locking in my code as it's dealing with much larger operations from a time to run POV and I'd prefer to only have it block when it's attempting the specific operation (iscsiadm wise) that's causing the problem. (The main goal for me is to be able to launch as many VMs as possible within the shortest period of time, on a single node). Anyway, all this Citrix talk has no bearing to open-iscsi itself, of course. I'm just curious if beyond this thread anyone else has encountered this type of issue and if anything is planned to resolve it. Thanks a bunch, -jwb On Jun 23, 7:04 pm, Mike Christie <[email protected]> wrote: > On 06/22/2011 11:58 AM, Jason Smith wrote: > > > > > > > > > > > Hello, > > > I'm seeing an error reported by iscsiadm when attempting to run > > concurrent login/logout operations on the same server. I'm using > > version 2.0-871 on Ubuntu 10.04 (64-bit). > > > In my setup, I'm running two instances of the same test script which > > makes various iscsiadm calls. Each test script has it's own unique > > set of initiator IQN, target IQN, and interface names. (all of the > > targets are hosted on the same IET instance, so the portal IPs are all > > the same). > > > The script does the following: > > > iscsiadm --mode iface --interface <iface> --op=new > > iscsiadm --mode iface --interface <iface> --op=update -- > > name=iface.initiatorname --value=<init_iqn> > > iscsiadm --mode discovery --type sendtargets --portal <ip> -- > > interface <iface> > > iscsiadm --mode node --portal <ip> --targetname <iqn> --interface > > <iface> --login > > // format partitions > > // mount > > // install files > > // umount > > iscsiadm --mode node --interface <iface> --logout > > iscsiadm --mode iface --interface <iface> --op=delete > > > In the error scenario, I have two instances running; lets call them T1 > > and T2. > > - T1 has just issued it's 'iscsiadm --login' > > - T2 issues 'iscsiadm ... --logout', which is reported to succeed > > - T2 then attempts to delete it's interface, but fails (error text > > below). > > - T1 login call finishes successfully, and the test goes off to > > twiddle bits on the LUNs > > - T2 attempts cleanup, which is to simply re-issue the identical > > iscsiadm logout/delete calls; both succeed. > > > Here's the output when the interface delete failed: > > Executing: iscsiadm --mode node --interface iface_6037_iqn. > > 1996-04.de.suse:01:b25a95fc46d --logout > > Logging out of session [sid: 18, target: iqn. > > 2009-02.com.rackwareinc:18233-31862-13174-18081-30072, portal: > > 192.168.111.5,3260] > > Logout of [sid: 18, target: iqn.2009-02.com.rackwareinc: > > 18233-31862-13174-18081-30072, portal: 192.168.111.5,3260]: successful > > Executing: iscsiadm --mode iface --interface iface_6037_iqn. > > 1996-04.de.suse:01:b25a95fc46d --op=delete > > So is the problem here that this command's delete is picking up sessions > that belong to the other script? Is session21 supposed to be managed by > T2, but T1 here is failing because it is seeing T2's sessions in a > incomplete state (When iscsiadm tells the kernel to logout the kernel > will not delete all of the sysfs tree for the session atomoically so we > might see parts of the session like here)? > > What version of open-iscsi is this? > > I think I can fix this, but it might take some time. There is a similar > issue where when we login the kernel adds the sysfs objects one at a > time, so I need to add some code to figure out if the session belongs to > the operation and if it is a login or logout (for logout we would be ok > but for login we would not want to delete the iface). > > > iscsiadm: could not read session targetname: 5 > > iscsiadm: could not find session info for session21 > > iscsiadm: This command will remove the record [iface: > > iface_6037_iqn.1996-04.de.suse:01:b25a95fc46d, target: iqn. > > 2009-02.com.rackwareinc:18233-31862-13174-18081-30072, portal: > > 192.168.111.5,3260], but a session is using it. Logout session then > > rerun command to remove record. > > iscsiadm: Could not delete iface iface_6037_iqn.1996-04.de.suse: > > 01:b25a95fc46d. A session is is using it or it could not be found. > > > After T1's login finishes, T2's cleanup code re-issues the same > > commands which succeed: > > > Executing: iscsiadm --mode node --interface iface_6037_iqn. > > 1996-04.de.suse:01:b25a95fc46d --logout > > I got confused here. I thought in the beginning of the mail it said that > the 2 scripts had different ifaces, but the above ifacename is the same > as the output for the the first script? > > > Executing: iscsiadm --mode iface --interface iface_6037_iqn. > > 1996-04.de.suse:01:b25a95fc46d --op=delete > > iface_6037_iqn.1996-04.de.suse:01:b25a95fc46d unbound and deleted. > > > Right now my plan is to work around this by adding a lock to my higher > > level code, to ensure that iscsiadm commands are not run > > concurrently. Any other advice would be appreciated. > > That would be the safest. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
