Re: iscsi errors with debian-kernel 2.6.24-3-686
Hello, thanks for quick reply. During weekend I did backups on the iscsi-device with not one single error on kernel 2.6.22. Am Freitag, den 13.06.2008, 19:11 +0200 schrieb Mike Christie: Michael Kindermann wrote: Hello, We receive errors below on a standard debian lenny/testing sytem since kernelupdate from 2.6.22-3-686 to latest debian-kernel 2.6.24-3-686. The iscsi-device is Eonstore E16A-2130. The open-iscsi deb package is 2.0.869.2-2. When we use the older kernel the errors disappear. Errors only happen during copying on the iscsi-devices. Is this behaviour a debian specific problem and I have to compile open-scsi? Jun 13 14:32:30 hg2 kernel: connection1:0: iscsi: detected conn error (1011) Jun 13 14:32:31 hg2 iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3) Jun 13 14:32:41 hg2 kernel: iscsi: host reset succeeded The READs or WRITEs from the copy operations are timing out. The SCSI layer sets a timer on each command which is probably the default of 60 seconds (scsi layer sets to 30 and udev normal raises this to 60). If the command does not complete in that time it starts the scsi error handler and you end up getting these errors in the worst case where we cannot just abort and restart the command or reset the device. Are you copying to the iscsi device or from it (and are you then copying to to/from a non-iscsi device), or is it mixed? These errors posted earlier resulting from simply copying a dvdimage from a local logical volume on SATA-Drives to a logical volume on the iscsi-device. Normal usage is to do backups by rsync (dirvish) to this device, which are very slow due to timeouts (normal rsync linux backups via ssh and the windows-filesystems by local rsyncing cifs-mounts). Results stay the same. The errors only happen when copying to the iscsi-device. Restoring of data from iscsi to local volumes works great without errors on the 2.6.24 -Kernel. Another effect is backups are slow and no keyboard interaction is possible during these timeouts. When you were using 2.6.22-3-686, were you also using the open-iscsi deb package 2.0.869.2-2 or was it a older version. We just boot from the former 2.6.22 -Kernel. dpkg -l |grep iscsi ii open-iscsi2.0.869.2-2 On the broken setup could you run iscsiadm -m session -P 3 and send all the output? iscsiadm -m session -P 3 iSCSI Transport Class version 2.0-724 iscsiadm version 2.0-869 Target: iqn.2002-10.com.infortrend:raid.sn7457154.20 Current Portal: 192.168.7.227:3260,1 Persistent Portal: 192.168.7.227:3260,1 ** Interface: ** Iface Name: default Iface Transport: tcp Iface Initiatorname: iqn.1993-08.org.debian:01.b75ebc4b5f99 Iface IPaddress: 192.168.7.2 Iface HWaddress: default Iface Netdev: default SID: 1 iSCSI Connection State: LOGGED IN iSCSI Session State: Unknown Internal iscsid Session State: NO CHANGE Negotiated iSCSI params: HeaderDigest: None DataDigest: None MaxRecvDataSegmentLength: 131072 MaxXmitDataSegmentLength: 65536 FirstBurstLength: 65536 MaxBurstLength: 262144 ImmediateData: Yes InitialR2T: No MaxOutstandingR2T: 1 Attached SCSI devices: Host Number: 5 State: running scsi5 Channel 00 Id 0 Lun: 0 Attached scsi disk sdc State: running scsi5 Channel 00 Id 0 Lun: 1 Attached scsi disk sdd State: running scsi5 Channel 00 Id 0 Lun: 2 Attached scsi disk sde State: running scsi5 Channel 00 Id 0 Lun: 3 Attached scsi disk sdf State: running scsi5 Channel 00 Id 0 Lun: 4 Attached scsi disk sdg State: running scsi5 Channel 00 Id 0 Lun: 5 Attached scsi disk sdh State: running scsi5 Channel 00 Id 0 Lun: 6 Attached scsi disk sdi State: running greets Michael -- Michael Kindermann Systemadministrator HandyGames www.handy-games.com GmbH i_Park Klingholz 13 97232 Giebelstadt Germany Tel: +49 (0) 9334 9757 - 35 mail:[EMAIL PROTECTED] --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this
Re: Connection Errors
swejis wrote: Up to the parts where you see the last scsi device get added. I previously posted this file: http://www.wehay.com/messages.1.gz wont that do ? No. I do not need any of the kernel info. It all seems to be telling us that the kernel really does think we set the timers as 5 seconds. We want to see why iscsid thinks it wants to set these timers as 5 seconds even though you set it so it shouldn't. Also what arch are you running? Are you running x86 or x86_64? If the latter are you running both 64bit kernels and userspace? Everything should have been built for the x86_64 architecture. manjula:/sbin # file iscsid iscsid: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), for GNU/ Linux 2.6.4, dynamically linked (uses shared libs), not stripped ok. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Out of the Office
Hey list, I am traveling to Boston for Red Hat Summit this week, and I am not sure what type of access to email I will have. If I do not answer your mail with my usual quickness :) you know why, so please be patient and I will get to all the emails by next Monday. Thanks Mike --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
iSCSI tape drive problems
Hi all, I've been using open-iscsi to set up an IBM Ultrium LTO-4 tape drive. I can connect and transfer files and everything, but the maximum read or write speed I can get is like 16MB/s by tweaking the block size. I am on a gigabit network, which the tape drive supports. The drive specifications rate it at 30MB/s so I'm off by 50%. Does anyone know what I can do to get the speed up to scratch? I'm pretty much a Linux/ network newbie so I no idea if its an open-iscsi tweak or a network tweak that I could possibly do. Also, I checked and my network card only supports MTU=1500, if that matters. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: iSCSI tape drive problems
Also, may not have anything to do with it but when I peek at the network with wireshark while writing to the drive, I see TCP Checksum Incorrect from my computer to the tape drive. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Dropping the connection
Eddy Quicksall wrote: Sorry, the prints were in the syslog. The kernel is version 2.6.24-rc7bidi_02 #5 Fri Jun 6 13:12:08 EDT 2008 i686 i686 i386 GNU/Linux. The open-iscsi is version iscsiadm version 2.0-869. The code came from a 3rd party who supposedly got it from open-iscsi.org. It looks like you are using 2.6.24-rc7 but with the kernel modules from 2.0-869. How many sessions are you running? Jun 16 08:46:36 localhost kernel: ping timeout of 5 secs expired, last rx 3649030, last ping 3650280, now 3651530 It looks like here that that we sent a nop but did not get a response within 5 seconds. In 2.0-869 there was a bug where we the timer fired incorrectly so we would get these errors when we shouldn't. If you could either turn off nops like how I described in the other mails or try 2.0-869.2 we could see if that narrows down the problem. Also what are you trying to do? Are you running any IO? Is it just normal reads and writes? Is it bidi commands? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
RE: Dropping the connection
I'll try to get updated to the latest version. It will take time. In the meantime, to answer your question, I am getting these closes in simple I/O cases. Here is the first test unit ready and notice that the close was almost immediate: I-T ISID 00023D01, TSIH 0001, CID , CPE# 0, clock=2.281 00 01 02 03 SCSI Command --- +---+---+---+-: 01 81 00 00 |.|I| 0x01 |F|R|W|0 0|ATTR | Reserved 0004: 00 00 00 00 |TotalAHSLength | DataSegmentLength 0008: 00 00 00 00 | Logical Unit Number (LUN) 0012: 00 00 00 00 | 0016: 04 00 00 00 | Initiator Task Tag 0020: 00 00 00 00 | Expected Data Transfer Length 0024: 00 00 00 04 | CmdSN 0028: 00 00 00 12 | ExpStatSN 0032: 00 00 00 00 | SCSI Command Descriptor Block (CDB) 0036: 00 00 00 00 | 0040: 00 00 00 00 | 0044: 00 00 00 00 | Initiator FIN (TCP clean close), TCP Port 2176 --- CPE#0, clock=2.281 -Original Message- From: open-iscsi@googlegroups.com [mailto:[EMAIL PROTECTED] On Behalf Of Mike Christie Sent: Monday, June 16, 2008 3:01 PM To: open-iscsi@googlegroups.com Subject: Re: Dropping the connection Eddy Quicksall wrote: Sorry, the prints were in the syslog. The kernel is version 2.6.24-rc7bidi_02 #5 Fri Jun 6 13:12:08 EDT 2008 i686 i686 i386 GNU/Linux. The open-iscsi is version iscsiadm version 2.0-869. The code came from a 3rd party who supposedly got it from open-iscsi.org. It looks like you are using 2.6.24-rc7 but with the kernel modules from 2.0-869. How many sessions are you running? Jun 16 08:46:36 localhost kernel: ping timeout of 5 secs expired, last rx 3649030, last ping 3650280, now 3651530 It looks like here that that we sent a nop but did not get a response within 5 seconds. In 2.0-869 there was a bug where we the timer fired incorrectly so we would get these errors when we shouldn't. If you could either turn off nops like how I described in the other mails or try 2.0-869.2 we could see if that narrows down the problem. Also what are you trying to do? Are you running any IO? Is it just normal reads and writes? Is it bidi commands? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Removing individual LUNs
I have a Linux machine running as a target with LVM-backed devices and the setup is working fine. However, when I change the size of one of the underlying devices, the initiators (running open-iscsi 2.0-865) don't pick up the change. I know that if I logout the entire session and then log in again it works fine, but I have a number of active LUNs that I can't afford to take offline just to resize one. So I'm assuming I need a way to remove the lun on the initiator and then re- add it. Unfortunately, I can't figure out how to do it. (I've already deleted and readded the lun on the target side using ietadm which worked fine.) What I need is something along the lines of 'iscsiadm -m session --op delete --lun 6'. Sadly, googling for iscsiadm constantly returns results for the Solaris implementation which really isn't helping me any. :) Any help is greatly appreciated. P.S. Is there any way to have iscsiadm list all currently connected luns? I can do iscsiadm -m node|session|etc but none of these are granular enough to show all the luns being provided by that particular target. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Removing individual LUNs
Alex M. wrote: I have a Linux machine running as a target with LVM-backed devices and the setup is working fine. However, when I change the size of one of the underlying devices, the initiators (running open-iscsi 2.0-865) don't pick up the change. I know that if I logout the entire session and then log in again it works fine, but I have a number of active LUNs that I can't afford to take offline just to resize one. So I'm assuming I need a way to remove the lun on the initiator and then re- add it. Unfortunately, I can't figure out how to do it. (I've already deleted and readded the lun on the target side using ietadm which worked fine.) What I need is something along the lines of 'iscsiadm -m session --op delete --lun 6'. Sadly, googling for iscsiadm constantly returns results for the Solaris implementation which really isn't helping me any. :) Any help is greatly appreciated. Do: iscsiadm -m session -r $SID --rescan you get the SID from iscsiadm -m session (it is the value in the []) or if you do iscsiadm -m session -P 3 you can see which session lines with with which lun. or iscsiadm -m node -T target --rescan or you can just take the lazy way and do iscsiadm -m session --rescan which rescans everything. The above commands will rescan existing devices and add new ones. It will not remove old ones. But if you have a disk mounted or have something like LVM or dm-multipath using a disk then you probably need to run one of those tools to pick up the change there. Oh yeah I do not remember if this is in 865. It might only be in the newer 865.X and new 869 releases on open-iscsi.org. P.S. Is there any way to have iscsiadm list all currently connected iscsiadm -m session -P 3 will spit out lots of info including this. luns? I can do iscsiadm -m node|session|etc but none of these are granular enough to show all the luns being provided by that particular target. http://www.open-iscsi.org/docs/README is for the newer 869 release, but I think most of it applies to 865. If not then check out the README with that release (older 865's may not have up to date READMEs though). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Removing individual LUNs
On Jun 16, 4:29 pm, Mike Christie [EMAIL PROTECTED] wrote: Do: iscsiadm -m session -r $SID --rescan Doing a rescan doesn't appear to pick up the new size of the devices. Having manually resized the filesystem on the target end, I now get errors on the initiator since the filesystem extents are outside of the apparent device extents. The above commands will rescan existing devices and add new ones. It will not remove old ones. As you said, rescans don't pick up deleted LUN's, so my only option would be to delete the LUN on the target and then re-create it under a _different_ LUN and then do a rescan. This is obviously not an ideal solution as it's going to result in a large number of zombie LUNs on the initiator as time goes on. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
