Re: troubleshooting connections to wasabi target
Dominik L. Borkowski wrote: > On Monday 30 June 2008 17:18:25 Ken A wrote: >> The problem went away when I turned off nops, reduced queue depth and >> cmds max and used the open-iscsi-2.0-869.2.tar.gz source. > > Would you be willing to share your initiator's final working configuration? > > thanks, > dom > > > I am not sure this is final, but it is working well. The only things changed from the default are nops, queue_depth and cmds_max. HTH, Ken # # Open-iSCSI default configuration. # Could be located at /etc/iscsi/iscsid.conf or ~/.iscsid.conf # # Note: To set any of these values for a specific node/session run # the iscsiadm --mode node --op command for the value. See the README # and man page for iscsiadm for details on the --op command. # # iSNS settings # Address of iSNS server #isns.address = 192.168.0.1 #isns.port = 3205 # # NIC/HBA and driver settings # # open-iscsi can create a session and bind it to a NIC/HBA. # To set this up see the example iface config file. #* # Startup settings #* # To request that the iscsi initd scripts startup a session set to "automatic". # node.startup = automatic # # To manually startup the session set to "manual". The default is manual. node.startup = manual # * # CHAP Settings # * # To enable CHAP authentication set node.session.auth.authmethod # to CHAP. The default is None. #node.session.auth.authmethod = CHAP # To set a CHAP username and password for initiator # authentication by the target(s), uncomment the following lines: #node.session.auth.username = username #node.session.auth.password = password # To set a CHAP username and password for target(s) # authentication by the initiator, uncomment the following lines: #node.session.auth.username_in = username_in #node.session.auth.password_in = password_in # To enable CHAP authentication for a discovery session to the target # set discovery.sendtargets.auth.authmethod to CHAP. The default is None. #discovery.sendtargets.auth.authmethod = CHAP # To set a discovery session CHAP username and password for the initiator # authentication by the target(s), uncomment the following lines: #discovery.sendtargets.auth.username = username #discovery.sendtargets.auth.password = password # To set a discovery session CHAP username and password for target(s) # authentication by the initiator, uncomment the following lines: #discovery.sendtargets.auth.username_in = username_in #discovery.sendtargets.auth.password_in = password_in # # Timeouts # # # See the iSCSI REAME's Advanced Configuration section for tips # on setting timeouts when using multipath or doing root over iSCSI. # # To specify the length of time to wait for session re-establishment # before failing SCSI commands back to the application when running # the Linux SCSI Layer error handler, edit the line. # The value is in seconds and the default is 120 seconds. node.session.timeo.replacement_timeout = 120 # To specify the time to wait for login to complete, edit the line. # The value is in seconds and the default is 15 seconds. node.conn[0].timeo.login_timeout = 15 # To specify the time to wait for logout to complete, edit the line. # The value is in seconds and the default is 15 seconds. node.conn[0].timeo.logout_timeout = 15 # Time interval to wait for on connection before sending a ping. node.conn[0].timeo.noop_out_interval = 0 # To specify the time to wait for a Nop-out response before failing # the connection, edit this line. Failing the connection will # cause IO to be failed back to the SCSI layer. If using dm-multipath # this will cause the IO to be failed to the multipath layer. node.conn[0].timeo.noop_out_timeout = 0 # To specify the time to wait for abort response before # failing the operation and trying a logical unit reset edit the line. # The value is in seconds and the default is 15 seconds. node.session.err_timeo.abort_timeout = 15 # To specify the time to wait for a logical unit response # before failing the operation and trying session re-establishment # edit the line. # The value is in seconds and the default is 30 seconds. node.session.err_timeo.lu_reset_timeout = 20 #** # Retry #** # To speficy the number of times iscsiadm should retry a login # to the target when we first login, modify the following line. # The default is 4. Valid values are any integer value. This only # affects the initial login. Setting it to a high value can slow # down the iscsi service startup. Setting it to a low value can # cause a session to not get logged into, if there are distuptions # during startup or if the network is not ready at that time. node.session.initial_login_retry_max = 4 # session and device queue depth # To control how many commands the session will queue set # node.session.cmds_max to an
Re: troubleshooting connections to wasabi target
On Monday 30 June 2008 17:18:25 Ken A wrote: > The problem went away when I turned off nops, reduced queue depth and > cmds max and used the open-iscsi-2.0-869.2.tar.gz source. Would you be willing to share your initiator's final working configuration? thanks, dom --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: troubleshooting connections to wasabi target
Mike Christie wrote: > Ken A wrote: >> Mike Christie wrote: >>> Ken A wrote: Yes, thanks, turning off nops helped with some of the timeouts. The mkfs succeeded (very slowly), with this block of errors repeating in the log (see below). I should mention that this is on a asynchronous link, (100mbps initiator <-> 1000mbps target). The initiator is running Fedora 8, 2.6.25.6-27.fc8, open-iscsi 2.0-865. >>> Are you using the tools and kernel modules from 2.0-865? Or the iscsi >>> kernel modules from 2.6.25.6-27.fc8 and tools from the rpm? >>> >>> If you did a make and make install with open-iscsi 2.0-865 from >>> open-iscsi.org then you are using the tools and kernel modules from >>> the open-iscsi.org. >>> >>> Target is Wasabi Storage Builder. Any ideas? >>> Is there anything on the target logs? >>> >>> With a 100 mbs link we might have been sending too much IO, but I >>> would have expected to see some messages about scsi commands timing >>> out and host resets firing. You can try setting these in iscsid.conf: >>> >>> >>> >>> node.session.cmds_max = 16 node.session.queue_depth = 8 >>> >> Running open-iscsi from open-iscsi.org rather than iscsi-initiator-utils >> seems to have fixed the problem on one Initiator. >> Thanks very much for your help! >> Ken >> >> > > Are you running iscsi-initiator-utils with nops on or off? And what > version of iscsi-initiator-utils? iscsi-initiator-utils is actually the > open-iscsi userspace tools. We are running the software from the open-iscsi website (open-iscsi-2.0-869.2.tar.gz) - Not the fedora package. nops are off, and I'm playing with node.session.cmds_max and node.session.queue_depth to find a good balance. The issue I was having seems to have been related to the async connection causing too much I/O or triggering some kernel bug. The unfortunate result was a complete lockup of the Initiator machine requiring a reboot with the defaults in iscsid.conf. The problem went away when I turned off nops, reduced queue depth and cmds max and used the open-iscsi-2.0-869.2.tar.gz source. Thanks again for your assistance, Ken > > > > -- Ken Anderson Pacific.Net --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: troubleshooting connections to wasabi target
Ken A wrote: > Mike Christie wrote: >> Ken A wrote: >>> Yes, thanks, turning off nops helped with some of the timeouts. The >>> mkfs succeeded (very slowly), with this block of errors repeating >>> in the log (see below). >>> >>> I should mention that this is on a asynchronous link, (100mbps >>> initiator <-> 1000mbps target). The initiator is running Fedora 8, >>> 2.6.25.6-27.fc8, open-iscsi 2.0-865. >> Are you using the tools and kernel modules from 2.0-865? Or the iscsi >> kernel modules from 2.6.25.6-27.fc8 and tools from the rpm? >> >> If you did a make and make install with open-iscsi 2.0-865 from >> open-iscsi.org then you are using the tools and kernel modules from >> the open-iscsi.org. >> >> >>> Target is Wasabi Storage Builder. >>> >>> Any ideas? >> >> Is there anything on the target logs? >> >> With a 100 mbs link we might have been sending too much IO, but I >> would have expected to see some messages about scsi commands timing >> out and host resets firing. You can try setting these in iscsid.conf: >> >> >> >> node.session.cmds_max = 16 node.session.queue_depth = 8 >> > > Running open-iscsi from open-iscsi.org rather than iscsi-initiator-utils > seems to have fixed the problem on one Initiator. > Thanks very much for your help! > Ken > > Are you running iscsi-initiator-utils with nops on or off? And what version of iscsi-initiator-utils? iscsi-initiator-utils is actually the open-iscsi userspace tools. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: troubleshooting connections to wasabi target
Mike Christie wrote: > Ken A wrote: >> Yes, thanks, turning off nops helped with some of the timeouts. The >> mkfs succeeded (very slowly), with this block of errors repeating >> in the log (see below). >> >> I should mention that this is on a asynchronous link, (100mbps >> initiator <-> 1000mbps target). The initiator is running Fedora 8, >> 2.6.25.6-27.fc8, open-iscsi 2.0-865. > > Are you using the tools and kernel modules from 2.0-865? Or the iscsi > kernel modules from 2.6.25.6-27.fc8 and tools from the rpm? > > If you did a make and make install with open-iscsi 2.0-865 from > open-iscsi.org then you are using the tools and kernel modules from > the open-iscsi.org. > > >> Target is Wasabi Storage Builder. >> >> Any ideas? > > > Is there anything on the target logs? > > With a 100 mbs link we might have been sending too much IO, but I > would have expected to see some messages about scsi commands timing > out and host resets firing. You can try setting these in iscsid.conf: > > > > node.session.cmds_max = 16 node.session.queue_depth = 8 > Running open-iscsi from open-iscsi.org rather than iscsi-initiator-utils seems to have fixed the problem on one Initiator. Thanks very much for your help! Ken -- Ken Anderson Pacific.Net --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: troubleshooting connections to wasabi target
Ken A wrote: > > Yes, thanks, turning off nops helped with some of the timeouts. The mkfs > succeeded (very slowly), with this block of errors repeating in the log > (see below). > > I should mention that this is on a asynchronous link, (100mbps initiator > <-> 1000mbps target). > The initiator is running Fedora 8, 2.6.25.6-27.fc8, open-iscsi 2.0-865. Are you using the tools and kernel modules from 2.0-865? Or the iscsi kernel modules from 2.6.25.6-27.fc8 and tools from the rpm? If you did a make and make install with open-iscsi 2.0-865 from open-iscsi.org then you are using the tools and kernel modules from the open-iscsi.org. > Target is Wasabi Storage Builder. > > Any ideas? Is there anything on the target logs? With a 100 mbs link we might have been sending too much IO, but I would have expected to see some messages about scsi commands timing out and host resets firing. You can try setting these in iscsid.conf: node.session.cmds_max = 16 node.session.queue_depth = 8 Again, set them then rerun discovery and relogin. But if that does not work, and I think if it does it might just be a bandaid, then you should remove the iscsi-initiator-utils rpm by doing rpm -q iscsi-initiator-utils Then run these tools and kernel modules frmom http://www.open-iscsi.org/bits/open-iscsi-2.0-869.2.tar.gz and build it with make DEBUG_SCSI=1 DEBUG_TCP=1 make DEBUG_SCSI=1 DEBUG_TCP=1 install then rerun your test. You might have to reboot or manually unload the old iscsi modules too. > > Thanks, > Ken > > Jun 25 14:16:24 testserver kernel: connection3:0: detected conn error > (1011) > Jun 25 14:16:24 testserver kernel: sd 4:0:0:0: [sde] Result: > hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK > Jun 25 14:16:24 testserver kernel: end_request: I/O error, dev sde, > sector 1838511 > Jun 25 14:16:24 testserver kernel: printk: 474 messages suppressed. > Jun 25 14:16:24 testserver kernel: Buffer I/O error on device sde1, > logical block 919224 > Jun 25 14:16:24 testserver kernel: lost page write due to I/O error on sde1 > Jun 25 14:16:24 testserver kernel: Buffer I/O error on device sde1, > logical block 919225 > Jun 25 14:16:24 testserver kernel: lost page write due to I/O error on sde1 > Jun 25 14:16:24 testserver kernel: Buffer I/O error on device sde1, > logical block 919226 > Jun 25 14:16:24 testserver kernel: lost page write due to I/O error on sde1 > Jun 25 14:16:24 testserver kernel: sd 4:0:0:0: [sde] Result: > hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK > Jun 25 14:16:24 testserver kernel: end_request: I/O error, dev sde, > sector 1314223 > Jun 25 14:16:25 testserver iscsid: Kernel reported iSCSI connection 3:0 > error (1011) state (3) > Jun 25 14:16:28 testserver iscsid: connection3:0 is operational after > recovery (2 attempts) > > > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: troubleshooting connections to wasabi target
Mike Christie wrote: > Ken A wrote: >> Hi, >> >> I'm new to iscsi, and this group, and I'm hoping someone can't point me >> in the right direction. >> >> We have a Wasabi Storage Builder target and I'm having trouble with >> connections to it using open-iscsi initiators on several fedora core >> boxes running iscsi-initiator-utils.i386 6.2.0.868-0.7.fc9 >> >> I can connect to targets on the Wasabi box, and can create partitions. >> But, when I try to create a filesystem "mkfs -j -L test /dev/sdXY" >> things break badly. I/O errors "conn error (1011)" are reported in the >> log and the initiator locks up - the machine locks up, not just the >> initiator software. A hard reboot is required to get it back. :-( >> >> Here's what the log says (over 2000 lines of the same): >> >> kernel: Buffer I/O error on device sdc1, logical block 2568 >> iscsid: Nop-out timedout after 15 seconds on connection 1:0 state (3). >> Dropping session. > > Could you try turning nops off? > > Logout the existing sessions, set > node.conn[0].timeo.noop_out_interval = 0 > node.conn[0].timeo.noop_out_timeout = 0 > in iscsid.conf, rerun the iscsiadm -m discovery command, then > relogin and try the mkfs. > Yes, thanks, turning off nops helped with some of the timeouts. The mkfs succeeded (very slowly), with this block of errors repeating in the log (see below). I should mention that this is on a asynchronous link, (100mbps initiator <-> 1000mbps target). The initiator is running Fedora 8, 2.6.25.6-27.fc8, open-iscsi 2.0-865. Target is Wasabi Storage Builder. Any ideas? Thanks, Ken Jun 25 14:16:24 testserver kernel: connection3:0: detected conn error (1011) Jun 25 14:16:24 testserver kernel: sd 4:0:0:0: [sde] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Jun 25 14:16:24 testserver kernel: end_request: I/O error, dev sde, sector 1838511 Jun 25 14:16:24 testserver kernel: printk: 474 messages suppressed. Jun 25 14:16:24 testserver kernel: Buffer I/O error on device sde1, logical block 919224 Jun 25 14:16:24 testserver kernel: lost page write due to I/O error on sde1 Jun 25 14:16:24 testserver kernel: Buffer I/O error on device sde1, logical block 919225 Jun 25 14:16:24 testserver kernel: lost page write due to I/O error on sde1 Jun 25 14:16:24 testserver kernel: Buffer I/O error on device sde1, logical block 919226 Jun 25 14:16:24 testserver kernel: lost page write due to I/O error on sde1 Jun 25 14:16:24 testserver kernel: sd 4:0:0:0: [sde] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Jun 25 14:16:24 testserver kernel: end_request: I/O error, dev sde, sector 1314223 Jun 25 14:16:25 testserver iscsid: Kernel reported iSCSI connection 3:0 error (1011) state (3) Jun 25 14:16:28 testserver iscsid: connection3:0 is operational after recovery (2 attempts) > > > -- Ken Anderson Pacific.Net --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: troubleshooting connections to wasabi target
Ken A wrote: > Hi, > > I'm new to iscsi, and this group, and I'm hoping someone can't point me > in the right direction. > > We have a Wasabi Storage Builder target and I'm having trouble with > connections to it using open-iscsi initiators on several fedora core > boxes running iscsi-initiator-utils.i386 6.2.0.868-0.7.fc9 > > I can connect to targets on the Wasabi box, and can create partitions. > But, when I try to create a filesystem "mkfs -j -L test /dev/sdXY" > things break badly. I/O errors "conn error (1011)" are reported in the > log and the initiator locks up - the machine locks up, not just the > initiator software. A hard reboot is required to get it back. :-( > > Here's what the log says (over 2000 lines of the same): > > kernel: Buffer I/O error on device sdc1, logical block 2568 > iscsid: Nop-out timedout after 15 seconds on connection 1:0 state (3). > Dropping session. Could you try turning nops off? Logout the existing sessions, set node.conn[0].timeo.noop_out_interval = 0 node.conn[0].timeo.noop_out_timeout = 0 in iscsid.conf, rerun the iscsiadm -m discovery command, then relogin and try the mkfs. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: troubleshooting connections to wasabi target
Mike Christie wrote: > Ken A wrote: >> Hi, >> >> I'm new to iscsi, and this group, and I'm hoping someone can't point me >> in the right direction. >> >> We have a Wasabi Storage Builder target and I'm having trouble with >> connections to it using open-iscsi initiators on several fedora core >> boxes running iscsi-initiator-utils.i386 6.2.0.868-0.7.fc9 >> > > What kernel are they running? > > > 2.6.25.6-55.fc9 2.6.22.14-72.fc6 -- Ken Anderson Pacific.Net --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: troubleshooting connections to wasabi target
Ken A wrote: > Hi, > > I'm new to iscsi, and this group, and I'm hoping someone can't point me > in the right direction. > > We have a Wasabi Storage Builder target and I'm having trouble with > connections to it using open-iscsi initiators on several fedora core > boxes running iscsi-initiator-utils.i386 6.2.0.868-0.7.fc9 > What kernel are they running? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
troubleshooting connections to wasabi target
Hi, I'm new to iscsi, and this group, and I'm hoping someone can't point me in the right direction. We have a Wasabi Storage Builder target and I'm having trouble with connections to it using open-iscsi initiators on several fedora core boxes running iscsi-initiator-utils.i386 6.2.0.868-0.7.fc9 I can connect to targets on the Wasabi box, and can create partitions. But, when I try to create a filesystem "mkfs -j -L test /dev/sdXY" things break badly. I/O errors "conn error (1011)" are reported in the log and the initiator locks up - the machine locks up, not just the initiator software. A hard reboot is required to get it back. :-( Here's what the log says (over 2000 lines of the same): kernel: Buffer I/O error on device sdc1, logical block 2568 iscsid: Nop-out timedout after 15 seconds on connection 1:0 state (3). Dropping session. kernel: lost page write due to I/O error on sdc1 iscsid: received iferror -22 kernel: Buffer I/O error on device sdc1, logical block 2569 iscsid: received iferror -22 kernel: lost page write due to I/O error on sdc1 kernel: sd 1:0:0:0: [sdc] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK kernel: end_request: I/O error, dev sdc, sector 2884175 kernel: sd 1:0:0:0: [sdc] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK kernel: end_request: I/O error, dev sdc, sector 2884687 kernel: sd 1:0:0:0: [sdc] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK kernel: end_request: I/O error, dev sdc, sector 2885199 kernel: sd 1:0:0:0: [sdc] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK kernel: end_request: I/O error, dev sdc, sector 2885711 Thanks, Ken Anderson --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
