Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
On Mon, Apr 14, 2008 at 3:38 PM, Andrew Moise [EMAIL PROTECTED] wrote: Debian kernel guys, how do you feel about the situation? Would you be willing to apply a patch upgrading 2.6.24's iscsi subsystem to the version from 2.6.25 if that fixed serious known bugs? Would you prefer for a backported fix to be applied to the stable series and then get the fix from there? How do you feel in general? As it turns out, I'm wholly mistaken. Debian is planning to use 2.6.25 or higher for the next stable release, so this will be a non-issue. Hooray! http://lists.debian.org/debian-kernel/2008/03/msg00531.html Thanks. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
Andrew Moise wrote: On Thu, Feb 28, 2008 at 10:00 AM, Mike Christie [EMAIL PROTECTED] wrote: Andrew Moise wrote: Are there any iscsi fixes in 2.6.24, or is it 2.6.25 or nothing? There are major fixes in 2.6.23 for error handler races and an oops. There is one major fix in 2.6.24 for a write out race. If you guys tell me what kernel you are using, I can try to port the upstream patches to your kernel for you guys. Okay, it looks to me like it's settled that Debian is going to use 2.6.24 for the next stable release (currently it's based on 2.6.24.4). If you can point me to a patch which fixes the write out race, I can apply it to Debian's kernel and give it some testing (and also talk to the Debian kernel maintainers to try to get it included). Cheers! You mean you want the sync cache fix right? Can I just send a patch for what went into 2.6.25 since there are several fixes in there you guys probably want and because the patch will be easier for me to make? :) Just doing a patch for the cache sync will be difficult because it was made on top of other patches. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
On Mon, Apr 14, 2008 at 12:56 PM, Mike Christie [EMAIL PROTECTED] wrote: You mean you want the sync cache fix right? Can I just send a patch for what went into 2.6.25 since there are several fixes in there you guys probably want and because the patch will be easier for me to make? :) Just doing a patch for the cache sync will be difficult because it was made on top of other patches. Hm... I can't speak for the kernel maintainers, but the smaller the patch is the more likely it will be to get accepted I would imagine. Perhaps I should wait until hearing word from them before asking you to prepare particular patches. Debian kernel guys, how do you feel about the situation? Would you be willing to apply a patch upgrading 2.6.24's iscsi subsystem to the version from 2.6.25 if that fixed serious known bugs? Would you prefer for a backported fix to be applied to the stable series and then get the fix from there? How do you feel in general? Thanks. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
On Thu, Feb 28, 2008 at 10:00 AM, Mike Christie [EMAIL PROTECTED] wrote: Andrew Moise wrote: Are there any iscsi fixes in 2.6.24, or is it 2.6.25 or nothing? There are major fixes in 2.6.23 for error handler races and an oops. There is one major fix in 2.6.24 for a write out race. If you guys tell me what kernel you are using, I can try to port the upstream patches to your kernel for you guys. Okay, it looks to me like it's settled that Debian is going to use 2.6.24 for the next stable release (currently it's based on 2.6.24.4). If you can point me to a patch which fixes the write out race, I can apply it to Debian's kernel and give it some testing (and also talk to the Debian kernel maintainers to try to get it included). Cheers! -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
On 2/28/08, Mike Christie [EMAIL PROTECTED] wrote: Andrew Moise wrote: On 2/23/08, Andrew Moise [EMAIL PROTECTED] wrote: Okay, the userspace tools in open-iscsi 2.0.868-rc1, when combined with either the 2.6.25-rc2 kernel or the Debian 4.0 kernel (2.6.18) with the backported driver, seem not to have this problem. They also each survived several hours of bonnie++ Are there any plans to push these iscsi updates back into the stable kernel series? For me this is definitely a critical fix, and for I think the patches that went upstream are too big for the stable tree. I am trying to do a basic hack that will work though. I need it for Fedora too, but I have not been able to get it smaller and ported. Okay, makes sense. Debian folks: I'd definitely like to see the new open-iscsi make it into the kernel for lenny one way or another, although for myself I'd be perfectly okay with installing backported modules again if lenny ships with 2.6.22 and no updated iscsi. Of course, I have a feeling that the Debian maintainers may be inclined to trust upstream's judgement over mine as to what stable is :-), which would leave me installing the backported modules again if lenny ships with 2.6.22. Are there any iscsi fixes in 2.6.24, or is it 2.6.25 or nothing? Cheers! -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
On 2/28/08, Mike Christie [EMAIL PROTECTED] wrote: I am not familiar with Debain. Is lenny the name of the stable tree for Debain? It's the name for the next stable release of Debian. If you guys tell me what kernel you are using, I can try to port the upstream patches to your kernel for you guys. It's not settled yet. They're currently shipping 2.6.22 kernels in the testing version of the release, so it's guaranteed that it will be at least 2.6.22, but they haven't entered freeze yet (although a soft freeze is coming possibly next month). Maybe we should revisit this once it's decided which kernel will be used for lenny (if the kernel maintainers decide that they're interested in a Debian-specific iscsi stability patch at all; I don't want to ask you to prepare a patch if they're not going to apply it). Thanks! -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
Andrew Moise wrote: On 2/28/08, Mike Christie [EMAIL PROTECTED] wrote: Andrew Moise wrote: On 2/23/08, Andrew Moise [EMAIL PROTECTED] wrote: Okay, the userspace tools in open-iscsi 2.0.868-rc1, when combined with either the 2.6.25-rc2 kernel or the Debian 4.0 kernel (2.6.18) with the backported driver, seem not to have this problem. They also each survived several hours of bonnie++ Are there any plans to push these iscsi updates back into the stable kernel series? For me this is definitely a critical fix, and for I think the patches that went upstream are too big for the stable tree. I am trying to do a basic hack that will work though. I need it for Fedora too, but I have not been able to get it smaller and ported. Okay, makes sense. Debian folks: I'd definitely like to see the new open-iscsi make it into the kernel for lenny one way or another, although for myself I'd be perfectly okay with installing backported modules again if lenny I am not familiar with Debain. Is lenny the name of the stable tree for Debain? ships with 2.6.22 and no updated iscsi. Of course, I have a feeling that the Debian maintainers may be inclined to trust upstream's judgement over mine as to what stable is :-), which would leave me installing the backported modules again if lenny ships with 2.6.22. Are there any iscsi fixes in 2.6.24, or is it 2.6.25 or nothing? There are major fixes in 2.6.23 for error handler races and an oops. There is one major fix in 2.6.24 for a write out race. If you guys tell me what kernel you are using, I can try to port the upstream patches to your kernel for you guys. That is what I do for Red Hat and Fedora normally (I think it will be easer for this bug's patch too now that I think of it and considering you will probably want the other bug fixes I mentioned above) and it would make it easier on me since I could hopefully use similar code for Red Hat and Debian. Plus, instead of having to do stable kernel fixes, iscsi tarball fixes, and distro kernel backport fixes, I would only have to do two :) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
clone 466954 -1 reassign -1 open-iscsi found -1 2.0.865-1 thanks Okay, Mike Christie (the upstream maintainer) said that this was a known bug and recommended that I use _both_ the new kernel driver (available in kernel 2.6.25-rc2) and the new upstream userspace code (version 2.0-868-rc1). After upgrading both, I can't reproduce this bug. I'm therefore cloning it, and plan to mark each bug fixed when the appropriate Debian package is released. Cheers! -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
On 2/22/08, Andrew Moise [EMAIL PROTECTED] wrote: In any case, I'll try the new userspace tools with 2.6.25-rc2 over the weekend and let you know. If it seems to work out okay (and I have a ton of time to kill :-), I'll also try the new kernel modules on the 2.6.18 kernel that I was using previous to this whole endeavor. Okay, the userspace tools in open-iscsi 2.0.868-rc1, when combined with either the 2.6.25-rc2 kernel or the Debian 4.0 kernel (2.6.18) with the backported driver, seem not to have this problem. They also each survived several hours of bonnie++, so I'll plan to gingerly put 2.6.18 with the backported driver into production this week and see how it does. Thanks very much for providing the backported drivers; they make things a lot easier for me :-). Cheers. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
reassign 466954 linux-2.6 found 466954 2.6.22-6~bpo40+2 thanks So some more investigation reveals that the 'modprobe' command in '/etc/init.d/open-iscsi stop' will hang if you terminate the iscsiadm command. I'm inclined to guess, then, that this is a kernel bug. I also discovered what's going on with the /dev/sde device; it's the in-band management interface for the controller. I was able to disable it, and the behavior is exactly the same (except that there's no /dev/sde node remaining during the hang, obviously). Thanks! -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
Andrew Moise wrote: forwarded 466954 [EMAIL PROTECTED] thanks Hi all. I'm having a bit of trouble with open-iscsi on kernel 2.6.22.18 from Debian backports; I seem to be able to log in to my storage array, but when I log out, iscsiadm (and later modprobe) seem to hang forever. I've filed a Debian bug report about this at http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=466954, and attached a bug report in Linux kernel bug report format (since it's quite large, I didn't paste it inline). This is a production system, but I've got some ability to muck with it on weekends and such, so I can try more recent vanilla/git kernels if that will help with the debugging. It seems that the iSCSI system in the kernels I've been trying is not very mature; I note that http://www.open-iscsi.org/ says, Stable What other issues are you hitting. They might be fixed in the new release or we might still need a bug report. Let me know. releases: not available yet. I had been planning to put this into production -- is iSCSI in a state such that that's a reasonable goal, or should I just be planning to return this hardware and/or find an alternative way of interfacing with it? Thanks. This one was the fault of the upstream kernel changing behavior on us, and not iscsi. It is fixed in 2.6.25-rc2, and the kernel modules in the current rc release: http://www.open-iscsi.org/bits/open-iscsi-2.0-868-rc1.tar.gz To work around the problem you have to disable write caching on the target (not all targets can do this though). -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
On 2/22/08, Mike Christie [EMAIL PROTECTED] wrote: What other issues are you hitting. They might be fixed in the new release or we might still need a bug report. Let me know. The etch kernel I was originally using (2.6.18) crashed the machine when I tried to log in to the MD3000i enclosure, 2.6.22 hung on logout as I mentioned, the inband management interface was detected as a separate disk by open-iscsi (which caused I/O errors when the kernel tried to read a partition table from it and couldn't), and one person on a Dell mailing list mentioned kernel panics in 2.6.22 in addition to the hang on logout. If I still see the problem with the management interface in a newer kernel, I'll file a separate bug report about it. Of course, it might just be ridiculous behavior on the part of the hardware that the driver is expected to know how to handle properly in order to represent as a management interface. It's also easily worked around on my end by configuring the enclosure not to present an inband management interface. This one was the fault of the upstream kernel changing behavior on us, and not iscsi. Makes sense. I didn't mean to sound critical of open-iscsi; I was just wondering what the intended status of the code was. It is fixed in 2.6.25-rc2, and the kernel modules in the current rc release: http://www.open-iscsi.org/bits/open-iscsi-2.0-868-rc1.tar.gz Okay, I'll try to test that kernel this weekend and let you know how it works for me. Thanks! Just to check: Are you saying that I need a 2.6.25-rc2 kernel _and_ the 2.0-868-rc1 userspace tools, or are you saying that I either need a 2.6.25-rc2 kernel _or_ the new kernel module, which is included with the 2.0-868-rc1 userspace tools? To work around the problem you have to disable write caching on the target (not all targets can do this though). Unfortunately I don't see a way to do this with my target :-(. Thanks! -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
Andrew Moise wrote: On 2/22/08, Mike Christie [EMAIL PROTECTED] wrote: What other issues are you hitting. They might be fixed in the new release or we might still need a bug report. Let me know. The etch kernel I was originally using (2.6.18) crashed the machine when I tried to log in to the MD3000i enclosure, 2.6.22 hung on logout as I mentioned, the inband management interface was detected as a separate disk by open-iscsi (which caused I/O errors when the kernel tried to read a partition table from it and couldn't), and one person on a Dell mailing list mentioned kernel panics in 2.6.22 in addition to the hang on logout. If I still see the problem with the management interface in a newer kernel, I'll file a separate bug report about it. Of course, it might just be ridiculous behavior on the part of the hardware that the driver is expected to know how to handle properly in order to represent as a management interface. It's also easily worked around on my end by configuring the enclosure not to present an inband management interface. Ok thanks for the info. I do not think open-iscsi will be able to do anything about that. The scsi layer does the scanning and it creates the devices, so we have no control. cc linux-scsi on the bug report if you make one. This one was the fault of the upstream kernel changing behavior on us, and not iscsi. Makes sense. I didn't mean to sound critical of open-iscsi; I was just wondering what the intended status of the code was. It is no problem. It says not stable on the web page, for a good reason :) There are just too many combos and I cannot back port to every distro kernel, so we try to distribute kernel modules in the open-iscsi.org which can work with older kernels. It is a pain for distros to integrate, so I am not sure what else to do. For Red Hat I have to actually back port the code, and one distro is all I can take :) It is fixed in 2.6.25-rc2, and the kernel modules in the current rc release: http://www.open-iscsi.org/bits/open-iscsi-2.0-868-rc1.tar.gz Okay, I'll try to test that kernel this weekend and let you know how it works for me. Thanks! Just to check: Are you saying that I need a 2.6.25-rc2 kernel _and_ the 2.0-868-rc1 userspace tools, or are you saying that I either need a 2.6.25-rc2 kernel _or_ the new kernel module, which is included with the 2.0-868-rc1 userspace tools? Sorry for the confusion. You need the new userspace tools for all the combos. - You can use the new userspace tools and kernel modules on open-iscsi.org with 2.6.16 - 2.6.24. - You can use the new userspace tools with 2.6.25-rc2 and above. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
On 2/22/08, Mike Christie [EMAIL PROTECTED] wrote: Andrew Moise wrote: If I still see the problem with the management interface in a newer kernel, I'll file a separate bug report about it. Ok thanks for the info. I do not think open-iscsi will be able to do anything about that. The scsi layer does the scanning and it creates the devices, so we have no control. cc linux-scsi on the bug report if you make one. Okay, will do. It is no problem. It says not stable on the web page, for a good reason :) Heh. When you say it's not stable, do you mean that there's still a lot of active development happening, or do you mean that the code may still be somewhat unreliable? Or both? :-) There are just too many combos and I cannot back port to every distro kernel, so we try to distribute kernel modules in the open-iscsi.org which can work with older kernels. It is a pain for distros to integrate, so I am not sure what else to do. For Red Hat I have to actually back port the code, and one distro is all I can take :) You need the new userspace tools for all the combos. - You can use the new userspace tools and kernel modules on open-iscsi.org with 2.6.16 - 2.6.24. - You can use the new userspace tools with 2.6.25-rc2 and above. Hm... so is each version of the userspace tools intended to work only with the kernel modules included in the same tarball on open-iscsi.org? Or is it just that in this particular case, there are necessary bug fixes in both the new userspace tools and the new kernel modules? Sorry if I'm asking basic questions about the relationship between the userspace tools, the modules in the open-iscsi.org tarballs, and the modules in the stock kernels. I'm still a little unclear on how it all fits together, though. In any case, I'll try the new userspace tools with 2.6.25-rc2 over the weekend and let you know. If it seems to work out okay (and I have a ton of time to kill :-), I'll also try the new kernel modules on the 2.6.18 kernel that I was using previous to this whole endeavor. Thanks! -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#466954: open-iscsi: Hangs trying to log out of Dell MD3000i
Package: open-iscsi Version: 2.0.865-1 Severity: normal When I attempt to run /etc/init.d/open-iscsi stop or shut down my machine, I get an infinite hang in iscsiadm. This may be because my Dell MD3000i array provides three disks when I've only configured two, like so: Feb 21 22:18:38 localhost kernel: Loading iSCSI transport class v2.0-724. Feb 21 22:18:38 localhost kernel: iscsi: registered transport (tcp) Feb 21 22:18:38 localhost kernel: iscsi: registered transport (iser) Feb 21 22:18:38 localhost iscsid: iSCSI logger with pid=3508 started! Feb 21 22:18:39 localhost iscsid: transport class version 2.0-724. iscsid version 2.0-865 Feb 21 22:18:39 localhost iscsid: iSCSI daemon with pid=3509 started! Feb 21 22:19:15 localhost kernel: scsi1 : iSCSI Initiator over TCP/IP Feb 21 22:19:15 localhost kernel: scsi 1:0:0:0: Direct-Access DELL MD3000i 0670 PQ: 0 ANSI: 5 Feb 21 22:19:15 localhost kernel: sd 1:0:0:0: [sdc] 4269801472 512-byte hardware sectors (2186138 MB) Feb 21 22:19:15 localhost kernel: sd 1:0:0:0: [sdc] Write Protect is off Feb 21 22:19:15 localhost kernel: sd 1:0:0:0: [sdc] Mode Sense: 77 00 10 08 Feb 21 22:19:15 localhost kernel: sd 1:0:0:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA Feb 21 22:19:15 localhost kernel: sd 1:0:0:0: [sdc] 4269801472 512-byte hardware sectors (2186138 MB) Feb 21 22:19:15 localhost kernel: sd 1:0:0:0: [sdc] Write Protect is off Feb 21 22:19:15 localhost kernel: sd 1:0:0:0: [sdc] Mode Sense: 77 00 10 08 Feb 21 22:19:15 localhost kernel: sd 1:0:0:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA Feb 21 22:19:15 localhost kernel: sdc: unknown partition table Feb 21 22:19:15 localhost kernel: sd 1:0:0:0: [sdc] Attached SCSI disk Feb 21 22:19:15 localhost kernel: scsi 1:0:0:1: Direct-Access DELL MD3000i 0670 PQ: 0 ANSI: 5 Feb 21 22:19:15 localhost kernel: sd 1:0:0:1: [sdd] 1586597888 512-byte hardware sectors (812338 MB) Feb 21 22:19:15 localhost kernel: sd 1:0:0:1: [sdd] Write Protect is off Feb 21 22:19:15 localhost kernel: sd 1:0:0:1: [sdd] Mode Sense: 77 00 10 08 Feb 21 22:19:15 localhost kernel: sd 1:0:0:1: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA Feb 21 22:19:15 localhost kernel: sd 1:0:0:1: [sdd] 1586597888 512-byte hardware sectors (812338 MB) Feb 21 22:19:15 localhost kernel: sd 1:0:0:1: [sdd] Write Protect is off Feb 21 22:19:15 localhost kernel: sd 1:0:0:1: [sdd] Mode Sense: 77 00 10 08 Feb 21 22:19:15 localhost kernel: sd 1:0:0:1: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA Feb 21 22:19:15 localhost kernel: sdd: unknown partition table Feb 21 22:19:15 localhost kernel: sd 1:0:0:1: [sdd] Attached SCSI disk Feb 21 22:19:15 localhost kernel: scsi 1:0:0:31: Direct-Access DELL Universal Xport 0670 PQ: 0 ANSI: 5 Feb 21 22:19:15 localhost kernel: sd 1:0:0:31: [sde] 40960 512-byte hardware sectors (21 MB) Feb 21 22:19:15 localhost kernel: sd 1:0:0:31: [sde] Write Protect is off Feb 21 22:19:15 localhost kernel: sd 1:0:0:31: [sde] Mode Sense: 77 00 10 08 Feb 21 22:19:15 localhost kernel: sd 1:0:0:31: [sde] Write cache: disabled, read cache: enabled, supports DPO and FUA Feb 21 22:19:15 localhost kernel: sd 1:0:0:31: [sde] 40960 512-byte hardware sectors (21 MB) Feb 21 22:19:15 localhost kernel: sd 1:0:0:31: [sde] Write Protect is off Feb 21 22:19:15 localhost kernel: sd 1:0:0:31: [sde] Mode Sense: 77 00 10 08 Feb 21 22:19:15 localhost kernel: sd 1:0:0:31: [sde] Write cache: disabled, read cache: enabled, supports DPO and FUA Feb 21 22:19:15 localhost kernel: sde: unknown partition table Feb 21 22:19:15 localhost kernel: sd 1:0:0:31: [sde] Attached SCSI disk Feb 21 22:19:15 localhost iscsid: received iferror -22 Feb 21 22:19:15 localhost last message repeated 2 times Feb 21 22:19:15 localhost iscsid: connection1:0 is operational now Feb 21 22:19:16 localhost kernel: end_request: I/O error, dev sde, sector 48 Feb 21 22:19:16 localhost kernel: Buffer I/O error on device sde, logical block 6 Feb 21 22:19:16 localhost kernel: Buffer I/O error on device sde, logical block 7 Feb 21 22:19:16 localhost kernel: Buffer I/O error on device sde, logical block 8 Feb 21 22:19:16 localhost kernel: Buffer I/O error on device sde, logical block 9 Feb 21 22:19:16 localhost kernel: Buffer I/O error on device sde, logical block 10 Feb 21 22:19:16 localhost kernel: Buffer I/O error on device sde, logical block 11 Feb 21 22:19:16 localhost kernel: Buffer I/O error on device sde, logical block 12 Feb 21 22:19:16 localhost kernel: Buffer I/O error on device sde, logical block 13 Feb 21 22:19:16 localhost kernel: Buffer I/O error on device sde, logical block 14 Feb 21 22:19:16 localhost kernel: Buffer I/O error on device sde, logical block 15 /dev/sdc and /dev/sdd are what I have configured -- I don't know what /dev/sde is about, but it's still available during the hang. During the hang this is what goes
