Re: Open/iSCSI + Logout Response Timeout + replacement_timeout firing
Hi Mike, On Mon, 2008-06-30 at 11:27 -0500, Mike Christie wrote: Konrad Rzeszutek wrote: Ah if your disk are using write back cache then you are going to hit some problems. So if you see this in /var/log/messages when you loging: kernel: sd 9:0:0:1: [sdb] Write cache: enabled, then later when you run iscsiadm to log out you see: kernel: sd 9:0:0:1: [sdb] Synchronizing SCSI cache Then you are going to hit problems due to the scsi sysfs interface changing on us. iscsiadm is going to hang. IO is going to hang. You basically have to reboot the box by hand. Mike, Are you sure about this? When the sysfs entries are deleted (during the iscsiadm logout phase), the SCSI ml finishes all of the I/Os and the last operation is sending the SCSI Cache command. Wouldn't that quiesce I/O ? Granted See below. You are right if everything goes ok. this means doing these steps which are outside the normal iscsiadm: 1). flush dirty pages (call 'sync') 2). delete the sysfs entries (echo 1 /sys/block/sdX/device/delete) 3). wait until /sys/class/scsi_host/hostZ/host_busy reaches zero 4). iscsiadm -m logout The problem that I described occurs when we run the iscsiadm logout command and we used the sysfs delete file. When iscsiadm wrote to that attr in 2.6.18 it would return when the cache sync was sent and the device was fully deleted in the kernel. In 2.6.21 and above it returned right away. So iscsiadm's logout code would write to that attr and think the devices were cleaned up, then it would call the iscsi shutdown code which would send a logout and cleanup the kernel session, connection and host structs, thinking that the devices were properly flushed but IO could still be waiting to get written so all types of fun things can happen like Yikes.. We could get to the scsi host remove call and all the scsi device sysfs delete calls would still be starting up, so the host remove call and those delete calls would race (so this is we would have bypassed the host_busy check in the connection deletion function in the kernel). When this happens if the sysfs delete device got the scan mutex first, but the iscsi shutdown code had blocked the devices, while we were trying to remove the host then the iscsiadm logout command will hang, because the delete device would wait forever to try and send the command (it is not yet in the host so the command timer is not running and the device is blocked), and the remove host call is waiting on the scan mutex which the device has. If you have multiple devices then the remove host command can also end up failing IO, because we will have sent the logout and later set the session internal state to terminiate and incoming IO on the other devices that was queued will be failed when the remove devices functions flush the IO. If you do not have a write back cache we have other problems, where IO can be failed when it should not have for the reason above where the logout is sent, the terminate bit is set, and the remove host runs before the devices were properly removed and that causes IO to be failed. And actually in some kernels you can hang (the app would hang not iscsiadm in this case) when a cache sync was not needed, because if a cache sync was not needed when we would remove the host and it would delete the device but IO would be stuck in the queues and no one did a unplug of the queue when the scsi device was removed. We added a iscsi_unblock_session in the iscsi_session_teardown to flush everything so at least apps would not hang there (but that resulted in IO getting failed like above). Ok, so with Open/iSCSI 2.0-869.2 running on 2.6.25.9 w/ LIO-Core IBLOCK + DRBD with write cache disabled, we are no longer seeing hanging umounts during the rsc_stop() section of ISCSI_MOUNT VHACS cluster resource. Thanks for the helpful info mnc!! :-) Things have started to look up with the latest kernel + open/iscsi userspace code. The only item left that I have been running into on Etch are strange sshd segfaults (requring a service restart) during ISCSI_MOUNT:rsc_stop. They seem to be somehow be Open/iSCSI related (not exactly sure how yet) and happens on multiple machines, from both myself and Jerome's clusters. We have been trying to track this issue down with little luck, but with the new code it definately seems to be happening alot less frequently on my setup (only once so far with the latest Open/iSCSI). Pretty vague I know.. :) Oh yeah, jerome also mentioned something to me this morning about left-over iscsid processes after ISCSI_MOUNT:rsc_stop() as well. What was the problem case on this one again..? --nab --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe
How can I synchronize among different computers?
Dear all, I've got a Dell md3000i storage server. And I've got two ubuntu 8.04 servers. Each mounts disk of the storage server successfully. But each I make a directory on one machine. It fails to appear on the other one. If I umount the disk and mount it again, the directory will appear. So here's my question: Can I make the directory appear on the other machine within 1s or 10s? Thanks in advance for any advice. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Fedora iscsi
Well, the QLA4xxx works OK, but you need to use the QLogic tools which are woefully out of date. Once you have it setup it works nicely. Thank´s Konrad... I have been studing the PCI Express for higher volume information throughput. And the iscsi Qlogic´s HBA model is the QLE4xxx, instead of QLA4xxx. Do you have any expirience about it It is the same chipset. Just a different form-factor. ? Would this work as well as the QLA4xxx for Fedora environment ? Yes. The qla4xxx driver supports both form-factors and both port-options (dual or single port). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: open-iscsi configuration for wasabi storage builder
Dominik L. Borkowski wrote: On Friday 27 June 2008 15:48:13 Mike Christie wrote: Would it be worth getting sniffer dump from the existing 2.0-866 initiator? I placed the dump at: http://staff.vbi.vt.edu/dom/debug/debug.tar.bz2 In that archive I included tcpdump, sample script session of what commands were issued, initiator configs and the kernel logs. Not sure if joining this thread with Ken's would be a good idea. Somehow I didn't notice his, when I was searching the archive. However, a lot of his symptoms fit my problem, including the os completely freezing. Jon France at Wasabi looked into the issue, and he thinks this is fixed in newer firmware. He asked you guys to contact Wasabi to get a firmware update. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Fedora iscsi
Konrad Rzeszutek escreveu: Well, the QLA4xxx works OK, but you need to use the QLogic tools which are woefully out of date. Once you have it setup it works nicely. Thank´s Konrad... I have been studing the PCI Express for higher volume information throughput. And the iscsi Qlogic´s HBA model is the QLE4xxx, instead of QLA4xxx. Do you have any expirience about it It is the same chipset. Just a different form-factor. ? Would this work as well as the QLA4xxx for Fedora environment ? Yes. The qla4xxx driver supports both form-factors and both port-options (dual or single port). OK! So, do you think that I´m in the right direction, when I say that I´m going to create my own repository with some proprietary hardware and free software? Or it could be a bad trip... Thank´s! -- Marcos G. M. Santos SysAdmin - DIGILAB S.A. Tel: 55 48 3234 4041 www.digilab.com.br --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: open-iscsi configuration for wasabi storage builder
On Tuesday 01 July 2008 12:20:32 Mike Christie wrote: Jon France at Wasabi looked into the issue, and he thinks this is fixed in newer firmware. He asked you guys to contact Wasabi to get a firmware update. Yep, we've been in touch, just waiting to work out some basic details. We'll see how the latest firmware 4.0.2 will behave with openiscsi. Thank you, dom --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Fedora iscsi
OK! So, do you think that I´m in the right direction, when I say that I´m going to create my own repository with some proprietary hardware and free software? Or it could be a bad trip... What is it that you are intending to do? If you are just looking to use iSCSI I would recommend you first play with the Open-ISCSI initator (I presume you already have an iSCSI target?). If you want no CPU load when doing iSCSI, then the hardware HBA's are the choice. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: How can I synchronize among different computers?
On Wed, Jul 02, 2008 at 12:13:16AM +0800, Kun Niu wrote: Thank you for your fast reply. Then will nfs will be a good choice? Well, your Linux server would export the NFS directory - which would be based on a filesystem. So you would be back to the same problem (still mounting ext3 from two machines). Unless the MD3000i can export an NFS filesystem which I don't think it can. Any other cluster filesystem suggestion? http://en.wikipedia.org/wiki/Global_File_System --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: How can I synchronize among different computers?
I think gfs2 hits the point for me. I've found the package on my ubuntu. I'll dig into it. Really appreciate all your detailed help. 2008/7/2 Konrad Rzeszutek [EMAIL PROTECTED]: On Tue, Jul 01, 2008 at 12:41:13PM -0400, Konrad Rzeszutek wrote: On Wed, Jul 02, 2008 at 12:13:16AM +0800, Kun Niu wrote: Thank you for your fast reply. Then will nfs will be a good choice? Well, your Linux server would export the NFS directory - which would be based on a filesystem. So you would be back to the same problem (still mounting ext3 from two machines). Unless the MD3000i can export an NFS filesystem which I don't think it can. Any other cluster filesystem suggestion? http://en.wikipedia.org/wiki/Global_File_System Also GFS2 is an option. Use Google for more details. -- 失业 牛坤 MSN:[EMAIL PROTECTED] --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Fedora iscsi
Marcos Gileno wrote: Konrad Rzeszutek escreveu: OK! So, do you think that I´m in the right direction, when I say that I´m going to create my own repository with some proprietary hardware and free software? Or it could be a bad trip... What is it that you are intending to do? If you are just looking to use iSCSI I would recommend you first play with the Open-ISCSI initator (I presume you already have an iSCSI target?). If you want no CPU load when doing iSCSI, then the hardware HBA's are the choice. What I intend is to have the best performance and reliability storage as cheaper as possible... So, I already tested the open-iscsi target and iniciator on a fedora box. The results were not so good, the best result were with a fedora target and windows XP iniciator. I do not think I have ever heard that result before. For fedora target do you mean, scsi-target-utils rpm that comes with fedora or is it a IET/iscsi-target based rpm? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Fedora iscsi
Mike Christie escreveu: Marcos Gileno wrote: Konrad Rzeszutek escreveu: OK! So, do you think that I´m in the right direction, when I say that I´m going to create my own repository with some proprietary hardware and free software? Or it could be a bad trip... What is it that you are intending to do? If you are just looking to use iSCSI I would recommend you first play with the Open-ISCSI initator (I presume you already have an iSCSI target?). If you want no CPU load when doing iSCSI, then the hardware HBA's are the choice. What I intend is to have the best performance and reliability storage as cheaper as possible... So, I already tested the open-iscsi target and iniciator on a fedora box. The results were not so good, the best result were with a fedora target and windows XP iniciator. I do not think I have ever heard that result before. For fedora target do you mean, scsi-target-utils rpm that comes with fedora or is it a IET/iscsi-target based rpm? Unfortunately I don´t have it... I tested it at the end of last year, and a used the package that came with fedora at that time. But I clearly remember that with XP it was transparently and worked fine. And from linux to linux I had some little problems to even make it work out. So I wrote to the list, and somebody told me that there was some problem with the iniciator package for linux... -- Marcos G. M. Santos SysAdmin - DIGILAB S.A. Tel: 55 48 3234 4041 www.digilab.com.br --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Fedora iscsi
Marcos Gileno wrote: Mike Christie escreveu: Marcos Gileno wrote: Konrad Rzeszutek escreveu: OK! So, do you think that I´m in the right direction, when I say that I´m going to create my own repository with some proprietary hardware and free software? Or it could be a bad trip... What is it that you are intending to do? If you are just looking to use iSCSI I would recommend you first play with the Open-ISCSI initator (I presume you already have an iSCSI target?). If you want no CPU load when doing iSCSI, then the hardware HBA's are the choice. What I intend is to have the best performance and reliability storage as cheaper as possible... So, I already tested the open-iscsi target and iniciator on a fedora box. The results were not so good, the best result were with a fedora target and windows XP iniciator. I do not think I have ever heard that result before. For fedora target do you mean, scsi-target-utils rpm that comes with fedora or is it a IET/iscsi-target based rpm? Unfortunately I don´t have it... I tested it at the end of last year, and a used the package that came with fedora at that time. But I clearly remember that with XP it was transparently and worked fine. And from linux to linux I had some little problems to even make it work out. So I wrote to the list, and somebody told me that there was some problem with the iniciator package for linux... What list? This one or one of the fedora lists? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Connection Errors
I increased the I/O on the initiator yesterday by placing the LUN holding our mySQL database on this server. Everything looks 'quite' well part from the error I got this morning. All nop time out set to zero. Jul 1 07:15:23 manjula syslog-ng[17906]: STATS: dropped 0 Jul 1 07:20:38 manjula kernel: Pid: 0, comm: swapper Tainted: GN 2.6.25.5-1.1-default #1 Jul 1 07:20:38 manjula kernel: Jul 1 07:20:38 manjula kernel: Call Trace: Jul 1 07:20:38 manjula kernel: [8020d686] dump_trace +0xc4/0x576 Jul 1 07:20:38 manjula kernel: [8020db78] show_trace +0x40/0x57 Jul 1 07:20:38 manjula kernel: [8044f865] _etext+0x72/0x7b Jul 1 07:20:38 manjula kernel: [8841c47a] :libiscsi:iscsi_conn_failure+0x1a/0x90 Jul 1 07:20:38 manjula kernel: [8842960b] :iscsi_tcp:iscsi_tcp_state_change+0x41/0x5e Jul 1 07:20:38 manjula kernel: [803ff632] tcp_done +0x61/0x70 Jul 1 07:20:38 manjula kernel: [8040339f] tcp_reset +0x55/0x59 Jul 1 07:20:38 manjula kernel: [80409add] tcp_rcv_established+0x981/0xd30 Jul 1 07:20:38 manjula kernel: [8040fc7d] tcp_v4_do_rcv +0x31/0x1d3 Jul 1 07:20:38 manjula kernel: [8041035f] tcp_v4_rcv +0x540/0x7c0 Jul 1 07:20:38 manjula kernel: [803f5bd6] ip_local_deliver_finish+0x124/0x1f9 Jul 1 07:20:38 manjula kernel: [803f5d1d] ip_local_deliver +0x72/0x7a Jul 1 07:20:38 manjula kernel: [803f57e6] ip_rcv_finish +0x306/0x32d Jul 1 07:20:38 manjula kernel: [803f5a7c] ip_rcv+0x26f/ 0x2a5 Jul 1 07:20:38 manjula kernel: [803d478a] netif_receive_skb +0x408/0x42d Jul 1 07:20:38 manjula kernel: [882350f8] :tg3:tg3_poll +0x7d8/0xae1 Jul 1 07:20:38 manjula kernel: [803d2e55] net_rx_action +0xba/0x1fc Jul 1 07:20:38 manjula kernel: [8023ca52] __do_softirq +0x6d/0xe1 Jul 1 07:20:38 manjula kernel: [8020d3fc] call_softirq +0x1c/0x28 Jul 1 07:20:38 manjula kernel: DWARF2 unwinder stuck at call_softirq +0x1c/0x28 Jul 1 07:20:38 manjula kernel: Jul 1 07:20:38 manjula kernel: Leftover inexact backtrace: Jul 1 07:20:38 manjula kernel: Jul 1 07:20:38 manjula kernel: IRQ [8020e94c] ? do_softirq+0x44/0x8b Jul 1 07:20:38 manjula kernel: [8023c823] ? irq_exit+0x3f/ 0x80 Jul 1 07:20:38 manjula kernel: [8020ec03] ? do_IRQ+0xba/ 0xd8 Jul 1 07:20:38 manjula kernel: [8020b0d2] ? default_idle +0x0/0x78 Jul 1 07:20:38 manjula kernel: [8020c59d] ? ret_from_intr +0x0/0x19 Jul 1 07:20:38 manjula kernel: EOI [8024eee7] ? getnstimeofday+0x31/0x88 Jul 1 07:20:38 manjula kernel: [8020b133] ? default_idle +0x61/0x78 Jul 1 07:20:38 manjula kernel: [8020b12e] ? default_idle +0x5c/0x78 Jul 1 07:20:38 manjula kernel: [8020b0d2] ? default_idle +0x0/0x78 Jul 1 07:20:38 manjula kernel: [8020b08a] ? cpu_idle +0x92/0xda Jul 1 07:20:38 manjula kernel: [804447fc] ? start_secondary +0x408/0x417 Jul 1 07:20:38 manjula kernel: Jul 1 07:20:38 manjula kernel: connection3:0: detected conn error (1011) Jul 1 07:20:39 manjula iscsid: Target dropping connection 0, reconnect min 0 max 0 Jul 1 07:20:39 manjula iscsid: Kernel reported iSCSI connection 3:0 error (1011) state (3) Jul 1 07:20:40 manjula iscsid: connection3:0 is operational after recovery (1 attempts) Looks like a tcp-reset was received ? Would it help if I start the initiator by hand and increased debug-level ? TIA Brgds Jonas --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
