Re: iSCSI timeouts on login with 32/64bit user/kernel
On Nov 1, 2012, at 12:46 AM, Lisa Marie nullo...@sdf.org wrote: The problem with these type of patches has not been that we cannot find a solution that works on new setups with the patches. The problem is that we are not allowed to merge patches in the upstream kernel that break existing setups. For example if the user has only upgraded the kernel or only upgraded userspace tools, then we cannot break their setups. This patch does not work on 32 bit kernels/userpsace if the users only updates the tools. Per your request I've written a patch to allow for 32bit machines to work with un-patched 32bit and 64bit kernels. It should be trivial to sunset this patch with a kernel version number check when the kernel has the previous patch I submitted applied. I've tested this patch on 32/32 and 32/64 with an unpatched kernel. Does it work if the kernel is updated but you are using old tools? -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iSCSI timeouts on login with 32/64bit user/kernel
(uname_data.machine, x86_64) != 0) {
+fix_32bit_kernel_structs = 1;
+ }
+}
+#endif
+
int main(int argc, char *argv[])
{
struct utsname host_info; /* will use to compound initiator alias */
@@ -409,6 +436,11 @@ int main(int argc, char *argv[])
exit(ISCSI_ERR);
}
+#ifdef __i386__
+ /* Check and set kernel compatibility options (x86/x86_64 user/kernel).
*/
+ check_kernel_compatibility();
+#endif
+
umask(0177);
mgmt_ipc_fd = -1;
diff --git a/usr/netlink.c b/usr/netlink.c
index d65cd7c..b363439 100644
--- a/usr/netlink.c
+++ b/usr/netlink.c
@@ -4,6 +4,7 @@
* Copyright (C) 2004 Dmitry Yusupov, Alex Aizman
* Copyright (C) 2006 Mike Christie
* Copyright (C) 2006 Red Hat, Inc. All rights reserved.
+ * Copyright (C) 2012 Lisa Maginnis
* maintained by open-iscsi@googlegroups.com
*
* This program is free software; you can redistribute it and/or modify
@@ -90,6 +91,38 @@ struct nlattr *iscsi_nla_alloc(uint16_t type, uint16_t len)
return attr;
}
+#ifdef __i386__
+#define OLD_32BIT_UEVENT_U_END_BYTE 35 /* Location of the iscsi_uevent struct
*/
+/* fix_32bit_uevent_struct(char*)
+ * Aligns the iscsi_uevent struct sent from an i386 (32bit) kernel to match our
+ * alignment in iscsi_if.h by adding a 4 byte (sizeof(uint32)) block of padding
+ * between two unions (u/r) in the struct.
+ * data Pointer to the uevent_iscsi struct.
+ * count Total number of bytes to copy.
+ * peek Set to 1 if uevent_iscsi was pulled from nl_read()/MSG_PEEK, else
+ * set to 0.
+*/
+void fix_32bit_uevent_struct(char *data, size_t count, int peek) {
+
+ log_debug(7, in %s, __FUNCTION__);
+
+ /* Skip creating the offset if we are a PDU, CHAP, or STATS unless it is a
peek
+ * (which is called from nl_read()) */
+ if(((uint8_t)data[0] != ISCSI_KEVENT_RECV_PDU
+ (uint8_t)data[0] != ISCSI_UEVENT_GET_STATS
+ (uint8_t)data[0] != ISCSI_UEVENT_GET_CHAP) || peek) {
+
+/* Move iscsi_uevent.r four bytes (sizeof(uint32_t)) higher */
+memmove(data[OLD_32BIT_UEVENT_U_END_BYTE+sizeof(uint32_t)],
+ data[OLD_32BIT_UEVENT_U_END_BYTE],
+ count-OLD_32BIT_UEVENT_U_END_BYTE-sizeof(uint32_t));
+
+/* Clear the `uint32_t' padding between iscsi_uevent.u and iscsi_uevent.r
*/
+memset(data[OLD_32BIT_UEVENT_U_END_BYTE], 0, sizeof(uint32_t));
+ }
+}
+#endif
+
static int
kread(char *data, int count)
{
@@ -176,8 +209,17 @@ nlpayload_read(int ctrl_fd, char *data, int count, int
flags)
*/
rc = recvmsg(ctrl_fd, msg, flags);
- if (data)
- memcpy(data, NLMSG_DATA(iov.iov_base), count);
+ if (data) {
+ memcpy(data, NLMSG_DATA(iov.iov_base), count);
+
+#ifdef __i386__
+ /* Check if we need to clean this data for compatibility. */
+ if(fix_32bit_kernel_structs == 1) {
+fix_32bit_uevent_struct(data, count, 0);
+ }
+#endif
+
+ }
return rc;
}
@@ -1281,6 +1323,13 @@ static int ctldev_handle(void)
nlh = (struct nlmsghdr *)nlm_ev;
ev = (struct iscsi_uevent *)NLMSG_DATA(nlm_ev);
+#ifdef __i386__
+ /* Check if we need to clean this data for compatibility. */
+ if(fix_32bit_kernel_structs == 1) {
+ fix_32bit_uevent_struct((char*)ev, sizeof(struct iscsi_uevent), 1);
+ }
+#endif
+
log_debug(7, %s got event type %u\n, __FUNCTION__, ev-type);
/* drivers like qla4xxx can be inserted after iscsid is started */
switch (ev-type) {
On Mon, 29 Oct 2012, Mike Christie wrote:
Date: Mon, 29 Oct 2012 14:48:52 -0500
From: Mike Christie micha...@cs.wisc.edu
To: open-iscsi@googlegroups.com
Cc: Lisa Marie nullo...@sdf.org
Subject: Re: iSCSI timeouts on login with 32/64bit user/kernel
On 10/28/2012 11:42 PM, Lisa Marie wrote:
I've attached the below patches which can be applied to both the kernel
and the open-iscsi client. They have been tested on
x86/x86 user/kernel, and a x86/x86_64 user/kernel. This still needs to
be tested on an x86_64/x86_64 user/kernel as well as any
other system that has a 32bit userspace and a 64bit kernel.
Thanks a lot for the patch.
The problem with these type of patches has not been that we cannot find
a solution that works on new setups with the patches. The problem is
that we are not allowed to merge patches in the upstream kernel that
break existing setups. For example if the user has only upgraded the
kernel or only upgraded userspace tools, then we cannot break their
setups. This patch does not work on 32 bit kernels/userpsace if the
users only updates the tools.
nullo...@sdf.org
SDF Public Access UNIX System - http://sdf.org
--
You received this message because you are subscribed to the Google Groups
open-iscsi group.
To post to this group, send email to open-iscsi@googlegroups.com.
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com.
For more options, visit this group at
http
Re: iSCSI timeouts on login with 32/64bit user/kernel
On 10/28/2012 11:42 PM, Lisa Marie wrote: I've attached the below patches which can be applied to both the kernel and the open-iscsi client. They have been tested on x86/x86 user/kernel, and a x86/x86_64 user/kernel. This still needs to be tested on an x86_64/x86_64 user/kernel as well as any other system that has a 32bit userspace and a 64bit kernel. Thanks a lot for the patch. The problem with these type of patches has not been that we cannot find a solution that works on new setups with the patches. The problem is that we are not allowed to merge patches in the upstream kernel that break existing setups. For example if the user has only upgraded the kernel or only upgraded userspace tools, then we cannot break their setups. This patch does not work on 32 bit kernels/userpsace if the users only updates the tools. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts
Thanks for the response. I'm currently toying with this on my Fedora 15 system but eventually this will be implemented on a centos 6.2 system with: root@dus1san1:~# iscsiadm -V iscsiadm version 2.0-872.33.el6 root@dus1san1:~# uname -a Linux dus1san1.cvsn.local 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux My problem is that I'm not sure how the various timeouts relate to each other. What I basically want to be able to do is to guarantee that if e.g. a network outage lasts X seconds I want the virtual machines to recover and not get an I/O error resulting in a corrupt filesystem. From the readme it sound like the first thing that happens are the 5 ping retries and this would last 5*noop_out_timeout seconds. What happens after that? It sounds like a re-establishment of the connection is then attempted. Will this then generate new noop retry cycle and last until the replacement_timeout has passed? At which point does the os device timeout come into play (/sys/block/sdX/...)? I guess what I'm looking for is a sort of timeline. The network gets unplugged and an I/O request is issued (e.g. a simple ls on the filesystem on an iscsi device) to the device. What happens with this I/O request until it hits the wall and the failure manifest itself and show up as an I/O error on the console? (Currently I'm not using multipath in the setup I'm experimenting with) Regards, Dennis -- You received this message because you are subscribed to the Google Groups open-iscsi group. To view this discussion on the web visit https://groups.google.com/d/msg/open-iscsi/-/aLFqRZafDBUJ. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts
On 01/18/2012 12:58 PM, Dennis Jacobfeuerborn wrote: Thanks for the response. I'm currently toying with this on my Fedora 15 system but eventually this will be implemented on a centos 6.2 system with: root@dus1san1:~# iscsiadm -V iscsiadm version 2.0-872.33.el6 root@dus1san1:~# uname -a Linux dus1san1.cvsn.local 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux My problem is that I'm not sure how the various timeouts relate to each other. What I basically want to be able to do is to guarantee that if e.g. a network outage lasts X seconds I want the virtual machines to recover and not get an I/O error resulting in a corrupt filesystem. From the readme it sound like the first thing that happens are the 5 ping retries and this would last 5*noop_out_timeout seconds. What happens after There are not ping retries. Just one chance. There are 5 retries for disk IO. that? It sounds like a re-establishment of the connection is then attempted. Will this then generate new noop retry cycle and last until the replacement_timeout has passed? At which point does the os device timeout come into play (/sys/block/sdX/...)? No. I guess what I'm looking for is a sort of timeline. The network gets unplugged and an I/O request is issued (e.g. a simple ls on the filesystem on an iscsi device) to the device. What happens with this I/O request until it hits the wall and the failure manifest itself and show up as an I/O error on the console? 1 Initiator sends ping if there is not activity (READ/WRITE request being sent) on the connection for timeo.noop_out_interval seconds. 2 If we do not get a responce for the ping in noop_out_timeout seconds we fail the connection. 3. iscsi layer will try to relogin to the target. 4. A. If the command was running (it has not timed out and the scsi eh is not running) then the IO will be failed to the scsi layer and if it has retries left (so if it has been retried less than 5 times for disk IO) it will be queue in the block/scsi layer. B. If the command had already timedout then it is sort of stuck in the scsi eh until we relogin or replacement_timeout fires. It will sit in there waiting for the outcome of #5. 5. A. If we relogin within replacement_timeout seconds then IO will be restarted if the command had enough retries left. B. If cannot relogin withing replacement_timeout seconds then the IO will be failed upwards (if you are using dm-multipath then it will handle the problem). (Currently I'm not using multipath in the setup I'm experimenting with) Regards, Dennis -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts
On 01/17/2012 03:27 PM, Dennis Jacobfeuerborn wrote: Hi, I'm currently looking into setting up my first iscsi storage system(s) for virtualization. The issue I'm currently trying to find more details on is the timeouts involved in the iscsi traffic. From what I found on the web the following timeouts are relevant: os: /sys/block/sda/device/timeout initiator: node.session.timeo.replacement_timeout What version of open-iscsi and kernel are you using? Check out the readme (attached the current one) section 8. Does that help? If not let me know what other info you need. target/initiator: DefaultTime2Retain With ERL0 and single connection sessions this does not really do anything. When there is a problem detected that we have to drop the session and relogin the IO is going to be failed and retried. So if your network disruption caused the kernel network layer to return a error or tcp/ip state change notifcation then we are going to relogin and IO is going to be failed and retried if possible (for block/FS IO you get 5 retries for most errors (if replacement_timeout fires though then IO is failed right away (readme should have more info))). Or, if your disruption last longer than the ping/nop (see the readme) or device timeout then we are going to have to fail the IO and retry if possible. Apparently the Time2Retain is negotiated between initiator and target. Can somebody explain how these values are used in running iscsi session? Which values are relevant if I want to prevent I/O errors on the clients side due to a short disruption of the network connection? Regards, Dennis -- You received this message because you are subscribed to the Google Groups open-iscsi group. To view this discussion on the web visit https://groups.google.com/d/msg/open-iscsi/-/sBEKHG7tz0QJ. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en. = Linux* Open-iSCSI = July 1, 2010 Contents - 1. In This Release - 2. Introduction - 3. Installation - 4. Open-iSCSI daemon - 5. Open-iSCSI Configuration Utility - 6. Configuration - 7. Getting Started - 8. Advanced Configuration - 9. iSCSI System Info 1. In This Release == This file describes the Linux* Open-iSCSI Initiator. The software was tested on AMD Opteron (TM) and Intel Xeon (TM). The latest development release is available at: http://www.open-iscsi.org For questions, comments, contributions send e-mail to: open-iscsi@googlegroups.com 1.1. Features - highly optimized and very small-footprint data path; - persistent configuration database; - SendTargets discovery; - CHAP; - PDU header Digest; - multiple sessions; 2. Introduction === Open-iSCSI project is a high-performance, transport independent, multi-platform implementation of RFC3720 iSCSI. Open-iSCSI is partitioned into user and kernel parts. The kernel portion of Open-iSCSI is a from-scratch code licensed under GPL. The kernel part implements iSCSI data path (that is, iSCSI Read and iSCSI Write), and consists of three loadable modules: scsi_transport_iscsi.ko, libiscsi.ko and iscsi_tcp.ko. User space contains the entire control plane: configuration manager, iSCSI Discovery, Login and Logout processing, connection-level error processing, Nop-In and Nop-Out handling, and (in the future:) Text processing, iSNS, SLP, Radius, etc. The user space Open-iSCSI consists of a daemon process called iscsid, and a management utility iscsiadm. 3. Installation === As of today, the Open-iSCSI Initiator requires a host running the Linux operating system with kernel version 2.6.16, or later. 2.6.14 and 2.6.15 are partially supported. Known issues with 2.6.14 - .15 support: - If the device is using a write back cache, during session logout the cache sync command will fail. - iscsiadm's -P 3 option will not print out scsi devices. - iscsid will not automatically online devices. You need to enable Cryptographic API under Cryptographic options in the kernel config. And you must enable CRC32c CRC algorithm even if you do not use header or data digests. They are the kernel options, CONFIG_CRYPTO and CONFIG_CRYPTO_CRC32C, respectively. By default the kernel source found at /lib/modules/`uname
Re: iscsi timeouts and connection errors
FYI, to disbale connection load balancing on equallogics, you run... Login into the array as grpadmin; at the CLI prompt, type this command: CLI grpparams conn-balancing disable This command can be done w/o a reboot or restart of the arrays and becomes effective immediately on all members in the group. Should you decide to re-enable connection load balancing in the future, type this command at the Group CLI prompt: CLI grpparams conn-balancing enable -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
Oops, I should have added, in our testing, disabling the conn- balancing made no difference with respect to throughput/IO numbers... On May 26, 10:00 am, Taylor taylor.lew...@gmail.com wrote: FYI, to disbale connection load balancing on equallogics, you run... Login into the array as grpadmin; at the CLI prompt, type this command: CLI grpparams conn-balancing disable This command can be done w/o a reboot or restart of the arrays and becomes effective immediately on all members in the group. Should you decide to re-enable connection load balancing in the future, type this command at the Group CLI prompt: CLI grpparams conn-balancing enable -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
On 24 May 2010 at 9:06, Taylor wrote: Adjusting node.conn[0].timeo.noop_out_interval and node.conn[0].timeo.noop_out_timeout to 0 didn't seem to make a difference. But increasing the value of /sys/ block/sd#/device/timeout from 30 to a higher number, like 360, seems to have improved things... Is that something you would expect? Do you know of a sensible value for setting the timeout under /sys/block/sd#/device, and what is impact of increasing this? I think most people are not willing to wait 30s for disk I/O to complete, not to talk about 6 minutes. If a local disk ever needs 30 seconds for a command to complete, it's a clear indication that it had to be repaired or replaced. I think the same should be true for iSCSI. Increasing the timeout you can prevent an I/O error from appearing, but you won't become much more happy I'm afraid. Regards, Ulrich -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
Yeah, and netstat -in does show a few RX-ERR on the two 10 Gig NICs we are connecting to the iSCSI san with. So I don't know if thats because of the extremely high IO load. i.e., iostat is reporting 600,000 Kbytes/sec in writes across 4 LUNs during our heavy testing. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
On 05/24/2010 11:22 PM, Taylor wrote: Yes, when I get to office tomorrow I can attach screen output of make fail, as well as support info with equallogic. I guess to a different point, I understand if we are just killing the disk that can cause problems, but how can I set it up so that IO is slow or queues, and doesn't cause disconnects? Coming from a fiber channel background, you bang away at the disk and if you exceed disk or array IO, then your application seems to hang or be sluggish for a bit, but you don't get disconnected. Is there a way to get similar behavior under iSCSI? I don't know if this is expected behavior, or a bug, or just misconfiguration, or what... On the initiator side the ping timeout feature will sometimes mis-diagnose the connection as bad when it is really just slow. SUSE did some fixes, but I am not sure what version of SLES. If your not using multipath then you can just turn them off by setting those noop values to 0. On the target side, Equalogic has some target side load balancing where if it determines that it is best to use another path it will send that target requests logout request to use, then we will logout, kill the tcp/ip conneciton, then relogin to the new port. You can disable this on the target. I am not sure the command. You will have to ask Equallogic. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
Ok, will find out about the equallogic setting. So if we are using dm-multipath, we should not set those noop values to 0, but possibly increase them for high IO loads? On Tue, May 25, 2010 at 2:47 PM, Mike Christie micha...@cs.wisc.edu wrote: On 05/24/2010 11:22 PM, Taylor wrote: Yes, when I get to office tomorrow I can attach screen output of make fail, as well as support info with equallogic. I guess to a different point, I understand if we are just killing the disk that can cause problems, but how can I set it up so that IO is slow or queues, and doesn't cause disconnects? Coming from a fiber channel background, you bang away at the disk and if you exceed disk or array IO, then your application seems to hang or be sluggish for a bit, but you don't get disconnected. Is there a way to get similar behavior under iSCSI? I don't know if this is expected behavior, or a bug, or just misconfiguration, or what... On the initiator side the ping timeout feature will sometimes mis-diagnose the connection as bad when it is really just slow. SUSE did some fixes, but I am not sure what version of SLES. If your not using multipath then you can just turn them off by setting those noop values to 0. On the target side, Equalogic has some target side load balancing where if it determines that it is best to use another path it will send that target requests logout request to use, then we will logout, kill the tcp/ip conneciton, then relogin to the new port. You can disable this on the target. I am not sure the command. You will have to ask Equallogic. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
On 05/25/2010 04:15 PM, Taylor wrote: So far blowing away SUSE 11 and installing OpenSuse has resolved the Just so you know , the SUSE developers are good at keeping SLES up to date and even have fixes that are not yet upstream, so it is sometimes best to just get their newest SLES kernels. problem, however, throughput numbers have dropped by about half. We were getting nearly 5 Gbit /sec when testing across 5 LUNs, now we are getting more like 2.6 Gbit/sec, but we've had no disconnect errors. Now the question is, is that because we aren't getting disconnected because we are no longer getting the really high IO rates we were seeing? Normally we would expect if the disconnect errors go away then you get higher throughput numbers. However, when the Equalogic load balancing comes into play then if the disconnect errors also come with that target request logout message then it could be that the EQL target knows a better path and you wanted it to load balance. Is there a difference between SLES and opensuse's cpu speed handling (not sure what the feature is called in suse) (for iscsi I normally turn this off to get the best results) or iptables (for this turning it off can improve performance) or io scheduler (switching between noop and cfq sometimes gives differrent results. I'm going to let disk tests run for a few hours and see if we get any errors... On May 25, 3:24 pm, Mike Christiemicha...@cs.wisc.edu wrote: On 05/25/2010 03:01 PM, Taylor Lewick wrote: Ok, will find out about the equallogic setting. So if we are using dm-multipath, we should not set those noop values to 0, but possibly increase them for high IO loads? Yes, but set them lower than your scsi command timeout (/sys//block/sdX/device/timeout). -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
Thanks for the suggestions, I did change the io scheduler to make sure it was same as other server we were testing on. Not using iptables, and cpu frequency scaling or whatever its called is not enabled. To us this looks like an issue with worse network performance with later kernels. The kernel has become very bloated over time and we notice we get much worse per transaction network latencies in more recent kernel versions, as opposed to say the kernel version in use for SLES 10.2, which is 2.6.16. SLES 11 got slower, and OpenSUSE seems even slower then SLES11. I will do some iperf testing to compare a SLES 11 and an OpenSUSE 11 server to see how the network layer performs. So far stability is great, and performance is good, just going to see if we can get performance to be great as well. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
Adjusting node.conn[0].timeo.noop_out_interval and node.conn[0].timeo.noop_out_timeout to 0 didn't seem to make a difference. But increasing the value of /sys/ block/sd#/device/timeout from 30 to a higher number, like 360, seems to have improved things... Is that something you would expect? Do you know of a sensible value for setting the timeout under /sys/block/sd#/device, and what is impact of increasing this? -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
On 05/24/2010 11:06 AM, Taylor wrote:
Adjusting node.conn[0].timeo.noop_out_interval and
node.conn[0].timeo.noop_out_timeout to 0
didn't seem to make a difference. But increasing the value of /sys/
If it did not make a difference then it might not have got set right. If
you set them to 0, then you should not see the ping timeout messages
anymore (you might still see the target request logout ones, and that is
what we are trying to see with this test). Did you just set it in
iscsid.conf or did you set it in the node db by running
iscsiadm -m node -T ... -o update -n
node.conn[0].timeo.noop_out_interval -v 0
then repeat for the other value (you can also set it in the node db by
setting it in iscsid.conf then rerunning the iscsiadm discovery command).
block/sd#/device/timeout from 30 to a higher number, like 360, seems
to have improved things...
Is that something you would expect? Do you know of a sensible value
No. It might improve thoughput of some IO test, but it would not remove
the those log messages you sent before.
for setting the timeout under /sys/block/sd#/device, and what is
impact of increasing this?
It depends on the storage you are using.
8.1.2.1 Running Commands, the SCSI Error Handler, and replacement_timeout
-
Remember, from the Nop-out discussion that if a network problem is detected,
the running commands are failed immediately. There is one exception to this
and that is when the SCSI layer's error handler is running. To check if
the SCSI error handler is running iscsiadm can be run as:
iscsiadm -m session -P 3
You will then see:
Host Number: X State: Recovery
When the SCSI EH is running, commands will not be failed until
node.session.timeo.replacement_timeout seconds.
To modify the timer that starts the SCSI EH, you can either write
directly to the device's sysfs file:
echo X /sys/block/sdX/device/timeout
where X is in seconds or on most distros you can modify the udev rule.
To modify the udev rule open /etc/udev/rules.d/50-udev.rules, and find the
following lines:
ACTION==add, SUBSYSTEM==scsi , SYSFS{type}==0|7|14, \
RUN+=/bin/sh -c 'echo 60 /sys$$DEVPATH/timeout'
And change the echo 60 part of the line to the value that you want.
The default timeout for normal File System commands is 30 seconds when udev
is not being used. If udev is used the default is the above value which
is normally 60 seconds.
--
You received this message because you are subscribed to the Google Groups
open-iscsi group.
To post to this group, send email to open-is...@googlegroups.com.
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
Mike and all, okay, noted about the ping timeout messages. I set both of those values by running: iscsiadm --mode node --o update -n node.conn[0].timeo.noop_out_interval -v 10 iscsiadm --mode node --o update -n node.conn[0].timeo.noop_out_timeout -v 10 I verifed the settings took via iscsiadm --mode node --o show. I'm not seeing ping timeout's in the logs anymore. I was wrong about the block/sd# timeout changes, it didn't make a difference, I still see the: iscisd: Kernel reported iSCSI iscsi connection error #:# (1011) state (3) messages. We are running tests on 3 to 4o filesystems where we loop and do dd if=/dev/zero of=$FILENAME bs=$((1024 *1024)) count=8000 dd if=/dev/zero of=$FILENAME bs=$((1024 *1024)) count=12000 to drive disk utilization to high 90% and occaisionally 100%. We are getting great throughput, but the errors keep occuring. Called equallogic, and they said the diagnostics on the array show they aren't receiving a FIN from the initiator side, said could be due to problems with initiator. We are running with jumbo frames, don't know if that means we should increase MaxRecvDataSegmentLength and/or MaxXmitDataSegmentLength So I tried upgrading open-iscsi by downloading 2.0-871 and building, and succeeds for the three main files, iscsid, iscsiadm, iscsistart, but failes on make -C kernel, gives several errors for libiscsi.c Stuck here at the moment. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
Yes, when I get to office tomorrow I can attach screen output of make fail, as well as support info with equallogic. I guess to a different point, I understand if we are just killing the disk that can cause problems, but how can I set it up so that IO is slow or queues, and doesn't cause disconnects? Coming from a fiber channel background, you bang away at the disk and if you exceed disk or array IO, then your application seems to hang or be sluggish for a bit, but you don't get disconnected. Is there a way to get similar behavior under iSCSI? I don't know if this is expected behavior, or a bug, or just misconfiguration, or what... -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
On 05/21/2010 03:43 PM, Taylor wrote: We have a SLES 11 server connected to an Equallogic 10 Gig disk array. Initially everything seemed to work just fine. When doing some IO testing against the mounted volumes, in which we cause very high IO loads, i.e. 95 to 100% as reported by iostat, we started seeing the following messages in /var/log/messages: May 21 14:26:20 hostA kernel: connection27:0: ping timeout of 5 secs expired, last rx 4426024362, last ping 4426025612, now 4426026862 May 21 14:26:20 hostA kernel: connection27:0: detected conn error (1011) May 21 14:26:21 hostA iscsid: Kernel reported iSCSI connection 27:0 error (1011) state (3) May 21 14:27:03 hostA kernel: connection30:0: detected conn error (1011) May 21 14:27:26 hostA iscsid: Target requests logout within 3 seconds for connection May 21 14:27:26 hostA iscsid: Target dropping connection 0, reconnect min 2 max 0 May 21 14:27:26 hostA iscsid: Kernel reported iSCSI connection 30:0 error (1011) state (4) May 21 14:27:38 hostA kernel: connection25:0: detected conn error (1011) May 21 14:27:39 hostA iscsid: Kernel reported iSCSI connection 21:0 error (1011) state (3) May 21 14:27:39 hostA iscsid: Kernel reported iSCSI connection 25:0 error (1011) state (3) May 21 14:28:16 hostA iscsid: connection27:0 is operational after recovery (3 attempts) May 21 14:28:20 hostA iscsid: connection21:0 is operational after recovery (2 attempts) Is there more to the log? I want to see if the target requests a logout first or if we get a ping timeout first. So far we've turned on flow control on the network switches, tried adjusting multipath.conf, turned off offload parameters on the iscsi NICs, and adjusting iscsid.conf timeouts. For the ping timeouts you can set node.conn[0].timeo.noop_out_interval and node.conn[0].timeo.noop_out_timeout to 0. If you are using dm-multipath though you might want them on, but maybe a little longer. Also on the target side you can turn off their load balancing which would remove the target logout request related disruptions, but that of course messes with load balancing. If you are using dm-multipath you probably do not need the target load balancing on though (not 100% sure what equalogic reccomends, but it seems like each sides algorithms could end up working against each other). We are running a 2.6.27 kernel with open-iscsi 2.0.870-26.5. Is that a SLES 2.6.27 kernel or kernel.org? If a SLES kernel then make sure you have the newest once, because SUSE has added fixes for when we thought a nop/ping timedout but really it was stuck behind a large transfer and that transfer was executing ok. If you are using a 2.6.27 kernel.org kernel I would upgrade to the newest upstream one. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsi timeouts and connection errors
Mike, thanks for the reply. The first line in the log i posted shows the ping timeout. Or were you wanting to see something else? Let me know what specifically you are looking for. Yes we are using multipath, I can post multipath.conf if that helps. I tried tweaking various settings in iscisd.conf, including increasing node.session.cmds_max and node.session.queue_depth and in multipath.conf, including setting features 1 queue_if_no_path and disabling no_path_retry. When you say turn off target load balancing, Do you mean on the equallogic side, i.e. a per volume setting? Understanding you aren't familliar with the equallogic, would that be how you would do it on a generic device, or say a clariion? It is a SLES stock kernel, we are looking at seeing if OpenSuse has a fix in it, they run virtually the same open-iscsi versions, with openSuse being slightly newer. I'll play with the timeout settings, I think they were both set to 5 by default, I tried increasing them, but it didn't seem to have any impact. I think I'm close, but would appreciate further help. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
