Re: Kernel / iscsi problem under high load
Gonçalo Borges wrote: Dear open-iscsi gurus... I'm working with an IBM DS3300 storage system accessible via open- iscsi technology to many different clients. To give you the proper context before describing my problem, I'll introduce my setup: *** The target setup *** - IBM DS3300 storage system as ISCSI target with 12 SATA disks. - 2 ISCSI controllers taking care of 2 network interfaces each (with Jumbo frames enabled). - Each controller owns a Raid 10 with one logical unit / partition each *** The initiators setup (with network interfaces with Jumbo frames enabled) *** - OS: [r...@core26]# cat /etc/redhat-release Scientific Linux SL release 5.2 (Boron) - Kernel: [r...@core26]# uname -a Linux core26.ncg.ingrid.pt 2.6.18-92.1.22.el5xen #1 SMP Tue Dec 16 07:06:23 EST 2008 x86_64 x86_64 x86_64 GNU/Linux Where did you get this kernel? Is it from xen or from Red Hat? If it is from Red Hat? I have not seen some of the error messages in your log in the upstream or RHEL code. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: iscsiadm: InitiatorName is required on the first Login PDU
Boaz Harrosh wrote: On 03/31/2009 03:09 AM, Mike Christie wrote: Boaz Harrosh wrote: Hi Mike, list. In 2.0-870 I use to discover and login in the following command: []$ iscsiadm -m discovery -t sendtargets -p 192.168.0.142:3260 --login But if I compile master: I get: iscsiadm: InitiatorName is required on the first Login PDU iscsiadm: login failed, couldn't make a login PDU iscsiadm: discovery login to 192.168.0.142 failed, giving up I've renamed my /etc/iscsi to /etc/iscsi.old and reran make install_user, but same problem. Also same command as above without --login gives same error. I do have my /etc/iscsi/initiatorname.iscsi exactly as before. [Q] What do I need to add to configuration so I can do discovery? iscsiadm does not read /etc/iscsi/initiatorname.iscsi directly. It talks to iscsid to ask what it is using. Is iscsid running? Did something break where iscsid is now not running when you run the discovery command? Sorry got a bit busy. I was not sure I did not check before. So I try again: [] git checkout master [] make user [r...@testlin2] make install_user then I do: [r...@testlin2]$ service open-iscsi start Starting iSCSI initiator service: [ OK ] Setting up iSCSI targets: iscsiadm: No records found! [ OK ] [r...@testlin2]$ iscsiadm -m discovery -t sendtargets -p 192.168.0.142:3260 --login iscsiadm: InitiatorName is required on the first Login PDU iscsiadm: login failed, couldn't make a login PDU iscsiadm: discovery login to 192.168.0.241 failed, giving up [r...@testlin2]$ ps ax |grep iscsi 32268 ?S 0:00 [iscsi_eh] 32284 ?Ss 0:00 iscsid 32285 ?SLs 0:00 iscsid 32313 pts/0S+ 0:00 grep iscsi If I checkout 01123e0 Update Changelog for new release which is 2.0-870 tag and do exactly like above it works, do you want that I try a bisection? Not yet. Just do iscsiadm -m discovery -t sendtargets -p 192.168.0.142:3260 --login -d 8 and send the debug output. I'm using an OSD target but it is TOMO's tgt target almost latest with only couple of patches behind HEAD, and OSD does not interfere with login. It only comes into play much later. BTW I found that make install_user will install the open-osd service: (install -m 755 etc/initd/initd.redhat /etc/init.d/open-iscsi) But Fedora distro will install iscsi service. I have scripts that start and stop iscsi as part of osd initialization would you say it is better to use the open-iscsi service? Maybe. For Red Hat though, iscsi-initiator-utils is always the iscsi package (it used to be based on linux-iscsi and is now open-iscsi). Same for the service. iscsi used to be linux-iscsi and now it is open-iscsi. It does not matter the upsteam package. What do other distros use as name of the service? I do not know. I think SUSE uses open-iscsi. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: iscsiadm: InitiatorName is required on the first Login PDU
On 04/02/2009 06:39 PM, Mike Christie wrote: So I try again: [] git checkout master [] make user [r...@testlin2] make install_user then I do: [r...@testlin2]$ service open-iscsi start Starting iSCSI initiator service: [ OK ] Setting up iSCSI targets: iscsiadm: No records found! [ OK ] [r...@testlin2]$ iscsiadm -m discovery -t sendtargets -p 192.168.0.142:3260 --login iscsiadm: InitiatorName is required on the first Login PDU iscsiadm: login failed, couldn't make a login PDU iscsiadm: discovery login to 192.168.0.241 failed, giving up [r...@testlin2]$ ps ax |grep iscsi 32268 ?S 0:00 [iscsi_eh] 32284 ?Ss 0:00 iscsid 32285 ?SLs 0:00 iscsid 32313 pts/0S+ 0:00 grep iscsi If I checkout 01123e0 Update Changelog for new release which is 2.0-870 tag and do exactly like above it works, do you want that I try a bisection? Not yet. Just do iscsiadm -m discovery -t sendtargets -p 192.168.0.142:3260 --login -d 8 and send the debug output. Hey hi that was fast here: iscsiadm: Max file limits 1024 1024 iscsiadm: updating defaults from '/etc/iscsi/iscsid.conf' iscsiadm: updated 'discovery.sendtargets.iscsi.MaxRecvDataSegmentLength', '32768' = '32768' iscsiadm: updated 'node.startup', 'manual' = 'manual' iscsiadm: updated 'node.session.timeo.replacement_timeout', '120' = '120' iscsiadm: updated 'node.conn[0].timeo.login_timeout', '30' = '15' iscsiadm: updated 'node.conn[0].timeo.logout_timeout', '15' = '15' iscsiadm: updated 'node.conn[0].timeo.noop_out_interval', '5' = '5' iscsiadm: updated 'node.conn[0].timeo.noop_out_timeout', '5' = '5' iscsiadm: updated 'node.session.err_timeo.abort_timeout', '15' = '15' iscsiadm: updated 'node.session.err_timeo.lu_reset_timeout', '30' = '20' iscsiadm: updated 'node.session.initial_login_retry_max', '4' = '8' iscsiadm: updated 'node.session.cmds_max', '128' = '128' iscsiadm: updated 'node.session.queue_depth', '32' = '32' iscsiadm: updated 'node.session.iscsi.InitialR2T', 'No' = 'No' iscsiadm: updated 'node.session.iscsi.ImmediateData', 'Yes' = 'Yes' iscsiadm: updated 'node.session.iscsi.FirstBurstLength', '262144' = '262144' iscsiadm: updated 'node.session.iscsi.MaxBurstLength', '16776192' = '16776192' iscsiadm: updated 'node.conn[0].iscsi.MaxRecvDataSegmentLength', '262144' = '131072' iscsiadm: updated 'node.session.iscsi.FastAbort', 'Yes' = 'Yes' iscsiadm: starting sendtargets discovery, address 192.168.0.241:3260, iscsiadm: sendtargets discovery to 192.168.0.241:3260 using isid 0x00023d00 iscsiadm: discovery timeouts: login 15, reopen_cnt 5, auth 45. iscsiadm: connecting to 192.168.0.241:3260 iscsiadm: connected local port 53619 to 192.168.0.241:3260 iscsiadm: connected to discovery address 192.168.0.241 iscsiadm: discovery session to 192.168.0.241:3260 starting iSCSI login on fd 3 iscsiadm: InitiatorName is required on the first Login PDU iscsiadm: login failed, couldn't make a login PDU iscsiadm: discovery login to 192.168.0.241 failed, giving up iscsiadm: disconnecting conn 0xd65040, fd 3 Not much there either should I start from clean, naybe the old records break something. What to do to start with totally clean setup? Boaz --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: iscsiadm: InitiatorName is required on the first Login PDU
Boaz Harrosh wrote:
On 04/02/2009 06:39 PM, Mike Christie wrote:
So I try again:
[] git checkout master
[] make user
[r...@testlin2] make install_user
then I do:
[r...@testlin2]$ service open-iscsi start
Starting iSCSI initiator service: [ OK ]
Setting up iSCSI targets: iscsiadm: No records found! [ OK ]
[r...@testlin2]$ iscsiadm -m discovery -t sendtargets -p 192.168.0.142:3260
--login
iscsiadm: InitiatorName is required on the first Login PDU
iscsiadm: login failed, couldn't make a login PDU
iscsiadm: discovery login to 192.168.0.241 failed, giving up
[r...@testlin2]$ ps ax |grep iscsi
32268 ?S 0:00 [iscsi_eh]
32284 ?Ss 0:00 iscsid
32285 ?SLs 0:00 iscsid
32313 pts/0S+ 0:00 grep iscsi
If I checkout 01123e0 Update Changelog for new release which is 2.0-870
tag and do exactly like above it works, do you want that I try a bisection?
Not yet. Just do
iscsiadm -m discovery -t sendtargets -p 192.168.0.142:3260 --login -d 8
and send the debug output.
Hey hi that was fast here:
iscsiadm: Max file limits 1024 1024
iscsiadm: updating defaults from '/etc/iscsi/iscsid.conf'
iscsiadm: updated 'discovery.sendtargets.iscsi.MaxRecvDataSegmentLength',
'32768' = '32768'
iscsiadm: updated 'node.startup', 'manual' = 'manual'
iscsiadm: updated 'node.session.timeo.replacement_timeout', '120' = '120'
iscsiadm: updated 'node.conn[0].timeo.login_timeout', '30' = '15'
iscsiadm: updated 'node.conn[0].timeo.logout_timeout', '15' = '15'
iscsiadm: updated 'node.conn[0].timeo.noop_out_interval', '5' = '5'
iscsiadm: updated 'node.conn[0].timeo.noop_out_timeout', '5' = '5'
iscsiadm: updated 'node.session.err_timeo.abort_timeout', '15' = '15'
iscsiadm: updated 'node.session.err_timeo.lu_reset_timeout', '30' = '20'
iscsiadm: updated 'node.session.initial_login_retry_max', '4' = '8'
iscsiadm: updated 'node.session.cmds_max', '128' = '128'
iscsiadm: updated 'node.session.queue_depth', '32' = '32'
iscsiadm: updated 'node.session.iscsi.InitialR2T', 'No' = 'No'
iscsiadm: updated 'node.session.iscsi.ImmediateData', 'Yes' = 'Yes'
iscsiadm: updated 'node.session.iscsi.FirstBurstLength', '262144' = '262144'
iscsiadm: updated 'node.session.iscsi.MaxBurstLength', '16776192' =
'16776192'
iscsiadm: updated 'node.conn[0].iscsi.MaxRecvDataSegmentLength', '262144' =
'131072'
iscsiadm: updated 'node.session.iscsi.FastAbort', 'Yes' = 'Yes'
iscsiadm: starting sendtargets discovery, address 192.168.0.241:3260,
iscsiadm: sendtargets discovery to 192.168.0.241:3260 using isid
0x00023d00
iscsiadm: discovery timeouts: login 15, reopen_cnt 5, auth 45.
iscsiadm: connecting to 192.168.0.241:3260
iscsiadm: connected local port 53619 to 192.168.0.241:3260
iscsiadm: connected to discovery address 192.168.0.241
iscsiadm: discovery session to 192.168.0.241:3260 starting iSCSI login on fd 3
iscsiadm: InitiatorName is required on the first Login PDU
iscsiadm: login failed, couldn't make a login PD
iscsiadm: discovery login to 192.168.0.241 failed, giving up
iscsiadm: disconnecting conn 0xd65040, fd 3
Not much there either
could you just do
iscsid -d 8 -f
What gets spit out for the initiator name?
Do you have different versions of iscsid and iscsiadm running?
Could you run
the discovery command with no debugging with the attached patch?
should I start from clean, naybe the old records break something.
It shoudn't we do not get to that point yet.
What to do to start with totally clean setup?
There is no nice uninstall. Let me try to write one up real quick.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
open-iscsi group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~--~~~~--~~--~--~---
diff --git a/usr/discovery.c b/usr/discovery.c
index 8f2d29d..696fdb4 100644
--- a/usr/discovery.c
+++ b/usr/discovery.c
@@ -523,9 +523,12 @@ static int request_initiator_name(void)
req.command = MGMT_IPC_CONFIG_INAME;
rc = do_iscsid(req, rsp);
- if (rc)
+ if (rc) {
+ log_error(request_initiator_name failed\n);
return EIO;
+ }
+ log_error(ok got: %s., rsp.u.config.var[0]);
if (rsp.u.config.var[0] != '\0')
strcpy(initiator_name, rsp.u.config.var);
Re: iscsiadm: InitiatorName is required on the first Login PDU
On 04/02/2009 07:14 PM, Mike Christie wrote: Boaz Harrosh wrote: could you just do iscsid -d 8 -f What gets spit out for the initiator name? Do you have different versions of iscsid and iscsiadm running? Could you run the discovery command with no debugging with the attached patch? should I start from clean, naybe the old records break something. It shoudn't we do not get to that point yet. What to do to start with totally clean setup? There is no nice uninstall. Let me try to write one up real quick. Sorry, it's gating late here, I'll only get to it first thing Sunday. Have a good weekend Boaz --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Kernel / iscsi problem under high load
Apr 1 11:44:13 core26 kernel: 122 [RAIDarray.mpp]iscsi06:1:0:1 Controller IO time expired. Delta 43701 secs Apr 1 11:44:13 core26 kernel: 497 [RAIDarray.mpp]iscsi06:1:0:1 Failed controller to 0. retry. vcmnd SN 458970 pdev H6:C0:T0:L1 0x00/0x00/0x00 0x0002 mpp_status:2 What is the RAIDArray.mpp program? Is that something the IBM docs mentioned needs to be installed? Is that a version of Open-iSCSI module .. or maybe the rdac handler?? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Kernel / iscsi problem under high load
Hi... First of all, thanks for the reply. After recovering my system, I tried to perform the tests you ask for It might be good to know what scsiinfo (or similar) says about the size of the LUN at the start of yout tests. Likewise, show what fdisk -l tells about the partitions, and finally what df -k tells about the capacity of the file system. I have the following multipath devices: [r...@core26 ~]# dmsetup ls iscsi06-apoio1(253, 0) - dm-0 iscsi06-apoio1p1(253, 3)- dm-3 iscsi06-apoio2p1(253, 2)- dm-2 (the one which gave problems previously,it was called dm-10), iscsi06-apoio2(253, 1)- dm-1 [r...@core26 ~]# multipath -ll sda: checker msg is rdac checker reports path is down iscsi06-apoio1 (3600a0b80003ad1e50f2e49ae6d3e) dm-0 IBM,VirtualDisk [size=2.7T][features=1 queue_if_no_path][hwhandler=0] \_ round-robin 0 [prio=100][active] \_ 25:0:0:0 sdb 8:16 [active][ready] iscsi06-apoio2 (3600a0b80003ad2130f8649ae6d5b) dm-1 IBM,VirtualDisk [size=2.7T][features=1 queue_if_no_path][hwhandler=0] \_ round-robin 0 [prio=100][active] \_ 25:0:0:1 sdc 8:32 [active][ready] So, we are interested in iscsi06-apoio2 (dm-2, sdc) and in iscsi06-apoio1 (dm-3, sdb) [r...@core26 ~]# fdisk -l /dev/sdb1 Disk /dev/sdb1: 499.9 GB, 49983104 bytes 255 heads, 63 sectors/track, 60788 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdb1 doesn't contain a valid partition table [r...@core26 ~]# fdisk -l /dev/sdc1 Disk /dev/sdc1: 499.9 GB, 49983104 bytes 255 heads, 63 sectors/track, 60788 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdc1 doesn't contain a valid partition table [r...@core26 ~]# df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 90491396 2008072 83812428 3% / tmpfs 524288 0524288 0% /dev/shm /dev/mapper/iscsi06-apoio1p1 480618344202804 456001480 1% /apoio06-1 /dev/mapper/iscsi06-apoio2p1 480618344202800 456001484 1% /apoio06-2 The sizes, although not exactly the same (but that doesn't happen also for the system disk), are very close. Then one could compare those sizes to those reported by the kernel. Maybe the setup just wrong, and it takes a while until the end of the device is reached. I do not think the difference I see in previous commands is big enough to justify a wrong setup. But I'm just guessing and I'm not really an expert. Then I would start slowly, i.e. with one izone running on one client. I've already performed the same testes with 6 Raid 0 and 6 Raid 1 instead of 2 Raid 10 in similar DS 3300 systems without having this kind of errors. But probably, I could be hitting some kind of limit.. BTW, what do you want to measure: the kernel throughput, the network throughput, the iSCSI throughput, the controller throughput, or the disk throughput? You should have some concrete idea before starting the benchmark. Also with just 12 disks I see little sense in having that many threads accessign the disk. To shorten a lengthy test, it may be advisable to reduce the system memory (iozone recommands to create a file size at least three times the amount of RAM, end even 8GB on a local disk takes hours to perform) I want to measure the I/O performance for the RAID in sequential and random write/reads. What matters for the final user is that he was able to write/read at XXX MB/s. I want to stress the system to know the limit of the ISCSI controllers (this is why I'm starting so many threads). In theory, at the controllers limit, they should take a lot of time to deal with the I/O traffic from the diferent clients but they are not suppose to die. Cheers Goncalo --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Kernel / iscsi problem under high load
Apr 1 11:44:13 core26 kernel: 122 [RAIDarray.mpp]iscsi06:1:0:1 Controller IO time expired. Delta 43701 secs Apr 1 11:44:13 core26 kernel: 497 [RAIDarray.mpp]iscsi06:1:0:1 Failed controller to 0. retry. vcmnd SN 458970 pdev H6:C0:T0:L1 0x00/0x00/0x00 0x0002 mpp_status:2 What is the RAIDArray.mpp program? Is that something the IBM docs mentioned needs to be installed? Is that a version of Open-iSCSI module .. or maybe the rdac handler?? This is just the rdac handler! Teh RDAC handler is activated booting your system with the mpp module which may be configured in a /etc/grub.conf such as: root (hd0,0) kernel /boot/xen.gz-2.6.18-92.1.22.el5 dom0_mem=1024M module /boot/vmlinuz-2.6.18-92.1.22.el5xen ro root=LABEL=/ module /boot/mpp-2.6.18-92.1.22.el5xen.img Cheers Goncalo --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Kernel / iscsi problem under high load
Where did you get this kernel? Is it from xen or from Red Hat? If it is from Red Hat? I have not seen some of the error messages in your log in the upstream or RHEL code. This is a xen kernel but distributed in the Scientific Linux official releases. Check, for example: http://ftp.scientificlinux.org/linux/scientific/5rolling/x86_64/SL/kernel-xen-2.6.18-128.1.1.el5.x86_64.rpm Cheers Gonçalo --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
