/var/lib/iscsi/static
What's the relevance of /var/lib/iscsi/static? I'm going to make an assumption an say that's where we can define static(permanent) target volumes that do not require discovery and will automatically logged in at boot? -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsiadm -m iface + routing
Hi Joseph, we also have Dell equallogics in our shop. I can tell you A LOT about them and how to setup with multipath/iscsi..etc. Mike Christie is an incredible resource for sure, and he helped me a great deal with my setup. Be thankful you aren't using SLES10 SP2. Performance, you should be able to hit 200-210 meg/s write performance with RAID10 and around 150-160 with RAID6. Read performance for both usually pegged at 215-220meg/sec sustained. This is of course with sequential data. couple of goofy things I did notice in multipath.conf: 1. features 1 queue_if_no_path and no_path_retry conflict 2. rr_min_io has the largest effect on performance in multipath. Play with different options (8/16/32/64) depending on your data sizes. You can reconfigure on the fly using multipathd -k and the reconfigure command. I am currently using 16 as I have lots of small data chunks and I want lots of io in flight. iscsid.conf node.session.cmds_max = 1024 node.session.queue_depth = 64 (96 works good here as well if you have small data sizes) node.conn[0].timeo.noop_out_interval = 15 node.conn[0].timeo.noop_out_timeout = 45 The above are my settings, Mike's probably work better. Couple of performance inhibiters I found...Make sure you have flow control set on the switch. TSO/SG/TX and RX offloading work by assembly packets into 64k chunks prior to sending...This is death to performance. If you have a beefy CPU turn them off like this: ethtool -K eth4 rx off ethtool -K eth4 tso off ethtool -K eth4 tx off ethtool -K eth4 sg off Tune your TCP stack for gigabit performance for sure. On Jul 22, 12:01 pm, Hoot, Joseph joe.h...@itec.suny.edu wrote: Mike or others, Do you or others think that my ping/CHAP issue that I'm having -- aka sessions resetting (logging out) could be related to the more recent ping timeout and maxcmds patch that has been worked on and discussed? I am attempting to get my test systems updated to test this, but I figured that I'd toss it out there to see if they might be related. Thanks, Joe -Original Message- From: open-iscsi@googlegroups.com [mailto:open-is...@googlegroups.com] On Behalf Of Mike Christie Sent: Tuesday, July 14, 2009 6:01 PM To: open-iscsi@googlegroups.com Subject: Re: iscsiadm -m iface + routing On 07/13/2009 09:20 AM, Hoot, Joseph wrote: Mike, Just as an FYI (in case you were most curious about this issue) I've narrowed this issue down to something with CHAP. On my EqualLogic, if I disable CHAP, I can't reproduce this issue. So I did the following. I after upgrading to the latest OEL 5.3 release of the iscsi-initiator, I could still reproduce the problem. Therefore, I did the following: 1) Setup another test environment using the same hardware (physical different hardware, but all same firmware, models, etc..) 2) presented a new test volume from EqualLogic 3) ran the ping test (ping -Ieth2 192.168.0.19 ping -Ieth3 192.168.0.19). 4) I couldn't reproduce the issue. 5) I checked what the difference were-- CHAP the only difference. 6) So I turned on CHAP authentication to the volume. 7) rm -rf /var/lib/iscsi/nodes/* /var/lib/iscsi/send_targets/* 8) rediscovered targets (after modifying /etc/iscsi/iscsid.conf with CHAP information) node.session.auth.authmethod = CHAP node.session.auth.username = mychapuserhere node.session.auth.password = mypasshere 9) ran the same ping test and was able to get iscsi sessions to fail within 2 minutes. 10) I wanted to prove that CHAP was the issue. So I logout out of all iscsi sessions. 11) I disabled CHAP on the EqualLogic 12) rediscovery targets and re-logged in to the sessions (without CHAP authentication) 13) ran the ping tests and couldn't break it after 30 minutes. 14) added CHAP again and was able to break the sessions within 2 minutes. So definitely something odd with CHAP (my guess, either in open-iscsi code or EqualLogic code). I've asked Roger Lopez, from Dell, to attempt to reproduce this in his lab. He has EqualLogic and Oracle VM Servers. Oracle developers that I'm working with don't currently have access to an EqualLogic, but they are attempting to reproduce this with their iSCSI equipment as well. I'm going to setup port mirroring on our switch and run tcpdumps to see what I can get. This is strange because CHAP does not come into play in the normal IO path. When we login we will do authentication with CHAP, but after that succeeds it nevers comes up when doing IO. It would only come up again when the session/connection is dropped/disconnected and we relogin. For the relogin we will do the CHAP authentication again. Maybe some memory error where chap values are overwriting some other memory. There was one recent fix by Dell, where when using CHAP they could segfault iscsid. Here is the updated RPM that I am working on for RHEL 5.4 that has the fix:
Re: Tuning iscsi read performance with multipath Redhat 5.3 / SLES 10 SP2 / Oracle Linux / Equallogic
As an update:
new observed behavior:
- RAW disk read performance is phenomenal (200meg/sec)
- Ext3 performance is 100meg/sec and tps in iostat aren't going about
800 (50k with raw disk).
Some added info:
- This system has an oracle database on it and it's tuned for huge
pages..etc (see sysctl posted above)
On Apr 24, 12:07 pm, jnantel nan...@hotmail.com wrote:
If you recall my thread on tuning performance for writes. Now I am
attempting to squeeze as much read performance as I can from my
current setup. I've read a lot of the previous threads, and there has
been mention of miracle settings that resolved slow reads vs
writes. Unfortunately, most posts reference the effects and not the
changes. If I were tuning for read performance in the 4k to 128k
block range what would the best way to go about it?
Observed behavior:
- Read performance seems to be capped out at 110meg/sec
- Write performance I get upwards of 190meg/sec
Tuning options I'll be trying:
block alignment (stride)
Receiving buffers
multipath min io changes
iscsi cmd depth
Hardware:
2 x Cisco 3750 with 32gig interconnect
2 x Dell R900 with 128gig ram and 1 broadcom Quad (5709) and 2 dual
port intels (pro 1000/MT)
2 x Dell Equallogic PS5000XV with 15 x SAS in raid 10 config
multipath.conf:
device {
vendor EQLOGIC
product 100E-00
path_grouping_policy multibus
getuid_callout /sbin/scsi_id -g -u -s /block/%n
features 1 queue_if_no_path
path_checker readsector0
failback immediate
path_selector round-robin 0
rr_min_io 128
rr_weight priorities
}
iscsi settings:
node.tpgt = 1
node.startup = automatic
iface.hwaddress = default
iface.iscsi_ifacename = ieth10
iface.net_ifacename = eth10
iface.transport_name = tcp
node.discovery_address = 10.1.253.10
node.discovery_port = 3260
node.discovery_type = send_targets
node.session.initial_cmdsn = 0
node.session.initial_login_retry_max = 4
node.session.cmds_max = 1024
node.session.queue_depth = 128
node.session.auth.authmethod = None
node.session.timeo.replacement_timeout = 120
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.host_reset_timeout = 60
node.session.iscsi.FastAbort = Yes
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 2
node.session.iscsi.MaxConnections = 1
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.conn[0].address = 10.1.253.10
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.noop_out_interval = 10
node.conn[0].timeo.noop_out_timeout = 30
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.HeaderDigest = None,CRC32C
node.conn[0].iscsi.DataDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
/etc/sysctl.conf
net.core.rmem_default= 65536
net.core.rmem_max=2097152
net.core.wmem_default = 65536
net.core.wmem_max = 262144
net.ipv4.tcp_mem= 98304 131072 196608
net.ipv4.tcp_window_scaling=1
#
# Additional options for Oracle database server
#ORACLE
kernel.panic = 2
kernel.panic_on_oops = 1
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=524288
net.core.wmem_max=524288
fs.aio-max-nr=524288
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
open-iscsi group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~--~~~~--~~--~--~---
Re: Time of Log-In
Are you aware of them fixing the Broadcom iscsi offload cards to support Jumbo frames? We have 2 5709s we turfed because of it. On Apr 16, 11:18 am, Mike Christie micha...@cs.wisc.edu wrote: Ulrich Windl wrote: Hello, while thinking about an udev/multipath timing problem with device discovery, I wondered how difficult it would be to record and the report time of session establihment (i.e. log-in). iscsiadm -m session -P 3 does not show that. Would that tinme be related to SID? No. I can add it though. I am still busy with work stuff trying to finish adding bnx2i and making sure cxgb3i is ok. When that stuff gets finished I will work on all your and eveyrone else's requests more. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Multipath + iscsi + SLES10 SP2 / REDHAT 5.3 / Oracle Linux 5 update 3
Accross my SAN, tuned system:
ping -I eth2 -s 9000 10.1.253.48
PING 10.1.253.48 (10.1.253.48) from 10.1.253.48 eth2: 9000(9028) bytes
of data.
9008 bytes from 10.1.253.48: icmp_seq=1 ttl=64 time=0.074 ms
9008 bytes from 10.1.253.48: icmp_seq=2 ttl=64 time=0.013 ms
9008 bytes from 10.1.253.48: icmp_seq=3 ttl=64 time=0.012 ms
9008 bytes from 10.1.253.48: icmp_seq=4 ttl=64 time=0.011 ms
9008 bytes from 10.1.253.48: icmp_seq=5 ttl=64 time=0.012 ms
9008 bytes from 10.1.253.48: icmp_seq=6 ttl=64 time=0.012 ms
9008 bytes from 10.1.253.48: icmp_seq=7 ttl=64 time=0.012 ms
9008 bytes from 10.1.253.48: icmp_seq=8 ttl=64 time=0.011 ms
9008 bytes from 10.1.253.48: icmp_seq=9 ttl=64 time=0.012 ms
TSO, TCP checksum offload and things like that seem to have a big
effect on latency. If you look at how things like TSO work, their
intention is to save you CPU overhead...in my case I don't care about
overhead I've got 24 cores.
On Apr 17, 3:25 am, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de
wrote:
On 16 Apr 2009 at 13:59, jnantel wrote:
FINAL RESULTS *
First of all I'd thank Mike Christie for all his help. Mike I'll
tapping your brain again for some read performance help.
This for the benefit of anyone using the Dell Equallogic PS5000XV
PS5000E with SLES10 SP2 / Redhat 5.3 / Centos 5.3 / Oracle Linux +
Multipath ( MPIO ) and open-iscsi ( iscsi ). Sorry about weird
formatting, making sure this is going get hit for people that were in
my predicament.
When seeing your settings, I wonder what your network latency for jumbo
frames is
(e.g using ping). The timing is dependent on packet sizes. Here is what I
have if
everything is connected to one switch (and both ends are handling normal iSCSI
traffic at the same time), started from Domain-0 of a XEN-virtualized machine
that
has 77 users logged on:
# ping -s 9000 172.20.76.1
PING 172.20.76.1 (172.20.76.1) 9000(9028) bytes of data.
9008 bytes from 172.20.76.1: icmp_seq=1 ttl=64 time=1.90 ms
9008 bytes from 172.20.76.1: icmp_seq=2 ttl=64 time=1.38 ms
9008 bytes from 172.20.76.1: icmp_seq=3 ttl=64 time=1.39 ms
9008 bytes from 172.20.76.1: icmp_seq=4 ttl=64 time=1.40 ms
9008 bytes from 172.20.76.1: icmp_seq=5 ttl=64 time=1.56 ms
9008 bytes from 172.20.76.1: icmp_seq=6 ttl=64 time=1.52 ms
9008 bytes from 172.20.76.1: icmp_seq=7 ttl=64 time=1.39 ms
9008 bytes from 172.20.76.1: icmp_seq=8 ttl=64 time=1.40 ms
9008 bytes from 172.20.76.1: icmp_seq=9 ttl=64 time=1.55 ms
9008 bytes from 172.20.76.1: icmp_seq=10 ttl=64 time=1.38 ms
--- 172.20.76.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9000ms
rtt min/avg/max/mdev = 1.384/1.491/1.900/0.154 ms
# ping 172.20.76.1
PING 172.20.76.1 (172.20.76.1) 56(84) bytes of data.
64 bytes from 172.20.76.1: icmp_seq=1 ttl=64 time=0.253 ms
64 bytes from 172.20.76.1: icmp_seq=2 ttl=64 time=0.214 ms
64 bytes from 172.20.76.1: icmp_seq=3 ttl=64 time=0.223 ms
64 bytes from 172.20.76.1: icmp_seq=4 ttl=64 time=0.214 ms
64 bytes from 172.20.76.1: icmp_seq=5 ttl=64 time=0.215 ms
64 bytes from 172.20.76.1: icmp_seq=6 ttl=64 time=0.208 ms
64 bytes from 172.20.76.1: icmp_seq=7 ttl=64 time=0.270 ms
64 bytes from 172.20.76.1: icmp_seq=8 ttl=64 time=0.313 ms
--- 172.20.76.1 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 6996ms
rtt min/avg/max/mdev = 0.208/0.238/0.313/0.039 ms
I think large queues are more important if the roundtrip delay is high. ANd
don't
forget that queue sizes are per device or session, so is uses some RAM.
Regards,
Ulrich
As from this thread my issue was amazingly slow performance with
sequential writes with my multipath, around 35 meg/s, configuration
when measured with IOMETER. First things first... THROW OUT IOMETER
FOR LINUX , it has problems with queue depth. With that said, with
default iscsi and multipath setup we saw between 60-80meg/sec
performance with multipath. In essence it was slower than single
interface in certain block sizes. When I got done my write performance
was pushing 180-190meg/sec with blocks as small as 4k ( sequential
write test using dt).
Here are my tweaks:
After making any multipath changes do multipath -F then multipath
otherwise your changes won't take effect.
/etc/multipath.conf
device {
vendor EQLOGIC
product 100E-00
path_grouping_policy multibus
getuid_callout /sbin/scsi_id -g -u -s /block/%n
features 1 queue_if_no_path --- important
path_checker readsector0
failback immediate
path_selector round-robin 0
rr_min_io 512 important, only works with large queue
depth and cms in iscsi.conf
rr_weight priorities
}
/etc/iscsi/iscsi.conf ( restarting iscsi seems to apply the configs
fine)
# To control how many commands the session will queue set
# node.session.cmds_max to an integer between 2 and 2048 that is also
Re: open-iscsi created only one disk but missing 2 more
What Target are you using? I beat my head on a issue when I first setup regarding access permissions. If you don't have the permissions right on the array, open-iscsi will let you see it but not log in. Another thing you may want to check, if you are sharing luns between systems, some arrays require you to explicitly allow that behavior. And last but not least, if you have a backend and a front end connection make sure you are connecting to the array with the right interface. Make sure one connection does not have a matching MTU *jumbo frames with the array it can cause strange behavior. On Apr 15, 10:27 pm, sundar mahadevan sundarmahadeva...@gmail.com wrote: I tried the same connecting another hard drive of 20 G with 3 logical volumes namely: asm 17G , ocr 924M and vote 760M. iscsiadm -m node -T iqn.2001-04.com.ubuntu:scsi.disk.vg1.ocr -p 192.168.20.22 -l iscsiadm -m node -T iqn.2001-04.com.ubuntu:scsi.disk.vg1.vote -p 192.168.20.22 -l iscsiadm -m node -T iqn.2001-04.com.ubuntu:scsi.disk.vg1.asm -p 192.168.20.22 -l Looks like it only created the first one: ocr Here is the log from /var/log/syslog Apr 15 21:10:17 sunny2 kernel: [ 1603.409561] Loading iSCSI transport class v2.0-870. Apr 15 21:10:17 sunny2 kernel: [ 1603.486634] iscsi: registered transport (tcp) Apr 15 21:10:18 sunny2 kernel: [ 1603.848719] iscsi: registered transport (iser) Apr 15 21:10:18 sunny2 iscsid: iSCSI logger with pid=5912 started! Apr 15 21:10:18 sunny2 kernel: [ 1604.408284] scsi2 : iSCSI Initiator over TCP/IP Apr 15 21:10:19 sunny2 iscsid: transport class version 2.0-870. iscsid version 2.0-865 Apr 15 21:10:19 sunny2 iscsid: iSCSI daemon with pid=5914 started! Apr 15 21:10:19 sunny2 iscsid: Could not verify connection 1:2. Dropping event. Apr 15 21:10:19 sunny2 iscsid: Could not verify connection 1:2. Dropping event. Apr 15 21:10:20 sunny2 kernel: [ 1606.064265] scsi3 : iSCSI Initiator over TCP/IP Apr 15 21:10:21 sunny2 iscsid: Could not verify connection 2:3. Dropping event. Apr 15 21:10:21 sunny2 iscsid: Could not verify connection 2:3. Dropping event. Apr 15 21:10:21 sunny2 kernel: [ 1607.592257] scsi4 : iSCSI Initiator over TCP/IP Apr 15 21:10:22 sunny2 iscsid: Could not verify connection 3:4. Dropping event. Apr 15 21:10:22 sunny2 iscsid: Could not verify connection 3:4. Dropping event. Apr 15 21:10:23 sunny2 kernel: [ 1609.120249] scsi5 : iSCSI Initiator over TCP/IP Apr 15 21:10:24 sunny2 iscsid: Could not verify connection 4:5. Dropping event. Apr 15 21:10:24 sunny2 iscsid: Could not verify connection 4:5. Dropping event. Apr 15 21:15:00 sunny2 kernel: [ 1886.664257] scsi6 : iSCSI Initiator over TCP/IP Apr 15 21:15:01 sunny2 iscsid: Could not verify connection 5:6. Dropping event. Apr 15 21:15:01 sunny2 kernel: [ 1886.946886] scsi 6:0:0:0: Direct-Access IET VIRTUAL-DISK 0 PQ: 0 ANSI: 4 Apr 15 21:15:01 sunny2 kernel: [ 1886.952267] sd 6:0:0:0: [sdb] 1892352 512-byte hardware sectors (969 MB) Apr 15 21:15:01 sunny2 kernel: [ 1886.956338] sd 6:0:0:0: [sdb] Write Protect is off Apr 15 21:15:01 sunny2 kernel: [ 1886.956365] sd 6:0:0:0: [sdb] Mode Sense: 77 00 00 08 Apr 15 21:15:01 sunny2 kernel: [ 1886.964101] sd 6:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA Apr 15 21:15:01 sunny2 kernel: [ 1886.976089] sd 6:0:0:0: [sdb] 1892352 512-byte hardware sectors (969 MB) Apr 15 21:15:01 sunny2 kernel: [ 1886.986197] sd 6:0:0:0: [sdb] Write Protect is off Apr 15 21:15:01 sunny2 kernel: [ 1886.986228] sd 6:0:0:0: [sdb] Mode Sense: 77 00 00 08 Apr 15 21:15:01 sunny2 kernel: [ 1887.82] sd 6:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA Apr 15 21:15:01 sunny2 kernel: [ 1887.002741] sdb: unknown partition table Apr 15 21:15:01 sunny2 kernel: [ 1887.042501] sd 6:0:0:0: [sdb] Attached SCSI disk Apr 15 21:15:01 sunny2 kernel: [ 1887.043773] sd 6:0:0:0: Attached scsi generic sg1 type 0 Apr 15 21:15:02 sunny2 iscsid: connection5:0 is operational now Apr 15 21:15:20 sunny2 iscsid: Could not get host for sid 5. Apr 15 21:15:20 sunny2 iscsid: could not get host_no for session 6. Apr 15 21:15:20 sunny2 iscsid: could not find session info for session5 Apr 15 21:15:20 sunny2 iscsid: session [iqn.2001-04.com.ubuntu:scsi.disk.vg1.vote,192.168.20.22,3260] already running. Apr 15 21:15:28 sunny2 iscsid: Nop-out timedout after 15 seconds on connection 5:0 state (3). Dropping session. Apr 15 21:15:28 sunny2 iscsid: connection5:0 is operational after recovery (1 attempts) Apr 15 21:15:33 sunny2 iscsid: Could not get host for sid 5. Apr 15 21:15:33 sunny2 iscsid: could not get host_no for session 6. Apr 15 21:15:33 sunny2 iscsid: could not find session info for session5 Apr 15 21:15:33 sunny2 iscsid: session [iqn.2001-04.com.ubuntu:scsi.disk.vg1.asm,192.168.20.22,3260] already running. Apr 15 21:15:54 sunny2 iscsid: Nop-out timedout after 15 seconds on
Re: Multipath + iscsi + SLES10 SP2 / REDHAT 5.3 / Oracle Linux 5 update 3
FINAL RESULTS *
First of all I'd thank Mike Christie for all his help. Mike I'll
tapping your brain again for some read performance help.
This for the benefit of anyone using the Dell Equallogic PS5000XV
PS5000E with SLES10 SP2 / Redhat 5.3 / Centos 5.3 / Oracle Linux +
Multipath ( MPIO ) and open-iscsi ( iscsi ). Sorry about weird
formatting, making sure this is going get hit for people that were in
my predicament.
As from this thread my issue was amazingly slow performance with
sequential writes with my multipath, around 35 meg/s, configuration
when measured with IOMETER. First things first... THROW OUT IOMETER
FOR LINUX , it has problems with queue depth. With that said, with
default iscsi and multipath setup we saw between 60-80meg/sec
performance with multipath. In essence it was slower than single
interface in certain block sizes. When I got done my write performance
was pushing 180-190meg/sec with blocks as small as 4k ( sequential
write test using dt).
Here are my tweaks:
After making any multipath changes do multipath -F then multipath
otherwise your changes won't take effect.
/etc/multipath.conf
device {
vendor EQLOGIC
product 100E-00
path_grouping_policy multibus
getuid_callout /sbin/scsi_id -g -u -s /block/%n
features 1 queue_if_no_path--- important
path_checker readsector0
failback immediate
path_selector round-robin 0
rr_min_io 512 important, only works with large queue
depth and cms in iscsi.conf
rr_weight priorities
}
/etc/iscsi/iscsi.conf ( restarting iscsi seems to apply the configs
fine)
# To control how many commands the session will queue set
# node.session.cmds_max to an integer between 2 and 2048 that is also
# a power of 2. The default is 128.
node.session.cmds_max = 1024
# To control the device's queue depth set node.session.queue_depth
# to a value between 1 and 128. The default is 32.
node.session.queue_depth = 128
Other changes I've made are basic gigabit network tuning for large
transfers and turning off some congestion functions, some scheduler
changes (noop is amazing for sub 4k blocks but awful for 4meg chunks
or higher). I've turned off TSO on the network cards, apparently it's
not supported with jumbo frames and actually slows down performance.
dc1stgdb14:~ # ethtool -k eth7
Offload parameters for eth7:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
dc1stgdb14:~ # ethtool -k eth10
Offload parameters for eth10:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
dc1stgdb14:~ #
On Apr 13, 4:36 pm, jnantel nan...@hotmail.com wrote:
I am having a major issue with multipath + iscsi writeperformance
with anything random or any sequential write with data sizes smaller
than 4meg (128k 64k 32k 16k 8k). With 32k block size, I am able to
get a maximum throughput of 33meg/s write. Myperformancegets cut by
a third with each smaller size, with 4k blocks giving me a whopping
4meg/s combined throughput. Now bumping the data size up to 32meg
gets me 160meg/sec throughput, and 64 gives me 190meg/s and finally to
top it out 128meg gives me 210megabytes/sec. My question is what
factors would limit myperformancein the 4-128k range?
Some basics about myperformancelab:
2 identical 1 gigabit paths (2 dual port intel pro 1000 MTs) in
separate pcie slots.
Hardware:
2 x Dell R900 6 quad core, 128gig ram, 2 x Dual port Intel Pro MT
Cisco 3750s with 32gigabit stackwise interconnect
2 x Dell Equallogic PS5000XV arrays
1 x Dell Equallogic PS5000E arrays
Operating systems
SLES 10 SP2 , RHEL5 Update 3, Oracle Linux 5 update 3
/etc/mutipath.conf
defaults {
udev_dir /dev
polling_interval 10
selector round-robin 0
path_grouping_policy multibus
getuid_callout /sbin/scsi_id -g -u -s /block/%n
prio_callout /bin/true
path_checker readsector0
features 1 queue_if_no_path
rr_min_io 10
max_fds 8192
# rr_weight priorities
failback immediate
# no_path_retry fail
# user_friendly_names yes
/etc/iscsi/iscsi.conf (non default values)
node.session.timeo.replacement_timeout = 15
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 30
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.MaxXmitDataSegmentLength = 262144
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 65536
Scheduler:
cat /sys/block/sdb/queue/scheduler
[noop] anticipatory deadline cfq
cat /sys/block/sdc/queue/scheduler
[noop] anticipatory deadline
Re: Multipath + iscsi + SLES10 SP2 / REDHAT 5.3 / Oracle Linux 5 update 3
Well I've got some disconcerting news on this issue. No changes at
any level alter the 34/meg throughput I get. I flushed multipath, blew
away /var/lib/iscsi just in case. I also verified in /var/lib/iscsi
the options got set. RHEL53 took my renice no problem.
Some observations:
Single interface iscsi gives me the exact same 34meg/sec
Going with 2 interfaces it gives me 17meg/sec each interface
Going with 4 interfaces it gives me 8meg/sec...etc..etc..etc.
I can't seem to set node.conn[0].iscsi.MaxXmitDataSegmentLength =
262144 in a way that actually gets used.
node.session.iscsi.MaxConnections = 1can't find any docs on this,
doubtful it is relevant.
iscsiadm -m session -P 3 still gives me the default 65536 for xmit
segment.
The Equallogic has all its interfaces on the same SAN network, this is
contrary to most implementations of multipath I've done. This is the
vendor recommended deployment.
Whatever is choking performance its consistently choking it down to
the same level.
On Apr 13, 5:33 pm, Mike Christie micha...@cs.wisc.edu wrote:
jnantel wrote:
I am having a major issue with multipath + iscsi write performance
with anything random or any sequential write with data sizes smaller
than 4meg (128k 64k 32k 16k 8k). With 32k block size, I am able to
get a maximum throughput of 33meg/s write. My performance gets cut by
a third with each smaller size, with 4k blocks giving me a whopping
4meg/s combined throughput. Now bumping the data size up to 32meg
gets me 160meg/sec throughput, and 64 gives me 190meg/s and finally to
top it out 128meg gives me 210megabytes/sec. My question is what
factors would limit my performance in the 4-128k range?
I think linux is just not so good with smaller IO sizes like 4K. I do
not see good performance with Fibre Channel or iscsi.
64K+ should be fine, but you want to get lots of 64K+ IOs in flight. If
you run iostat or blktrace you should see more than 1 IO in flight. If
while the test is running if you
cat /sys/class/scsi_host/hostX/host_busy
you should also see lots of IO running.
What limits the number of IO? On the iscsi initiator side, it could be
params like node.session.cmds_max or node.session.queue_depth. For a
decent target like the ones you have I would increase
node.session.cmds_max to 1024 and increase node.session.queue_depth to 128.
What IO tool are you using? Are you doing direct IO or are you doing
file system IO? If you just use something like dd with bs=64K then you
are not going to get lots of IO running. I think you will get 1 64K IO
in flight, so throughput is not going to be high. If you use something
like disktest
disktest -PT -T30 -h1 -K128 -B64k -ID /dev/sdb
you should see a lot of IOs (depends on merging).
If you were using dd with bs=128m then that IO is going to get broken
down into lots of smaller IOs (probably around 256K), and so the pipe is
nice and full.
Another thing I noticed in RHEL is if you increase the nice value of the
iscsi threads it will increase write perforamnce sometimes. So for RHEL
or Oracle do
ps -u root | grep scsi_wq
Then patch the scsi_wq_%HOST_ID with the iscsiadm -m session -P 3 Host
Number. And then renive the thread to -20.
Also check the logs and make sure you do not see any conn error messages.
And then what do you get when running the IO test to the individual
iscsi disks instead of the dm one? Is there any difference? You might
want to change the rr_min_io. If you are sending smaller IOs then
rr_min_io of 10 is probably too small. The path is not going to get lots
of nice large IOs like you would want.
Some basics about my performance lab:
2 identical 1 gigabit paths (2 dual port intel pro 1000 MTs) in
separate pcie slots.
Hardware:
2 x Dell R900 6 quad core, 128gig ram, 2 x Dual port Intel Pro MT
Cisco 3750s with 32gigabit stackwise interconnect
2 x Dell Equallogic PS5000XV arrays
1 x Dell Equallogic PS5000E arrays
Operating systems
SLES 10 SP2 , RHEL5 Update 3, Oracle Linux 5 update 3
/etc/mutipath.conf
defaults {
udev_dir /dev
polling_interval 10
selector round-robin 0
path_grouping_policy multibus
getuid_callout /sbin/scsi_id -g -u -s /block/%n
prio_callout /bin/true
path_checker readsector0
features 1 queue_if_no_path
rr_min_io 10
max_fds 8192
# rr_weight priorities
failback immediate
# no_path_retry fail
# user_friendly_names yes
/etc/iscsi/iscsi.conf (non default values)
node.session.timeo.replacement_timeout = 15
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 30
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.iscsi.FirstBurstLength = 262144
Re: equallogic - load balancing and xfs
I've had a similar issue with SLES10 SP2
This is what my multipathd config looks like now:
efaults {
udev_dir/dev
polling_interval10
selectorround-robin 0
# path_grouping_policymultibus
# getuid_callout /lib/udev/scsi_id -g -u -s /block/%n
# prioconst
# path_checkerdirectio
no_path_retry 5
features1 queue_if_no_path
rr_min_io 10
max_fds 8192
rr_weight priorities
failbackimmediate
# no_path_retry 5
user_friendly_names no
}
no_path_retry has to be commented out in order for the
queue_if_no_path to work
I've also played with the no-op timeouts as well:
node.conn[0].timeo.noop_out_interval = 10
node.conn[0].timeo.noop_out_timeout = 30
On Apr 13, 9:23 pm, Matthew Kent m...@bravenet.com wrote:
On Mon, 2009-04-13 at 17:28 -0500, Mike Christie wrote:
Matthew Kent wrote:
On Mon, 2009-04-13 at 15:44 -0500, Mike Christie wrote:
Matthew Kent wrote:
Can anyone suggest a timeout I might be hitting or a setting I'm
missing?
The run down:
- EqualLogic target
- CentOS 5.2 client
You will want to upgrade that to 5.3 when you can. The iscsi code in
there fixes a bug where the initiator dropped the session when it should
not.
Will do, probably Wednesday night and we'll see if this goes away. I'll
be sure to follow up for the archives.
- xfs lvm iscsi
During a period of high load the EqualLogic decides to load balance:
INFO 4/13/09 12:08:29 AM eql3 iSCSI session to target
'20.20.20.31:3260,
iqn.2001-05.com.equallogic:0-8a0906-b7f6d3801-2b2000d0f5347d9a-foo' from
initiator '20.20.20.92:51274, iqn.1994-05.com.redhat:a62ba20db72' was
closed. Load balancing request was received on the array.
So is this what you get in the EQL log when it decides to load balance
the initiator and send us to a different portal?
Yes, a straight copy from event log in the java web interface.
INFO 4/13/09 12:08:31 AM eql3 iSCSI login to target
'20.20.20.32:3260,
iqn.2001-05.com.equallogic:0-8a0906-b7f6d3801-2b2000d0f5347d9a-foo' from
initiator '20.20.20.92:44805, iqn.1994-05.com.redhat:a62ba20db72'
successful, using standard frame length.
on the client see I get:
Apr 13 00:08:29 moo kernel: [4576850.161324] sd 5:0:0:0: SCSI error:
return code = 0x0002
Apr 13 00:08:29 moo kernel: [4576850.161330] end_request: I/O error, dev
sdc, sector 113287552
Apr 13 00:08:32 moo kernel: [4576852.470879] I/O error in filesystem
(dm-10) meta-data dev dm-10 block 0x6c0a000
Are you using dm-multipath over iscsi? Does this load balance issue
affect all the paths at the same time? What is your multipath
no_path_retry value? I think you might want to set that higher to avoid
the FS from getting IO errors at this time if all paths are affected at
the same time.
Not using multipath on this one.
Do you have xfs on sdc or is there something like LVM or RAID on top of sdc?
That is really strange then. 0x0002 is DID_BUS_BUSY. The iscsi
initiator layer would return this when the target does its load
balancing. The initiator does this to ask he scsi layer to retry the IO.
If dm-multipath was used then it is failed to the multipath layer right
away. If dm-multipath is not used then we get 5 retries so we should not
see the error if there was only the one rebalancing at the time. If
there was a bunch of load rebalancing within a couple minutes then it
makes sense.
Yeah xfs on top of lvm, no multipath.
Logs only show the one load balancing request around that time.
Funny thing is this system, and the load balancing etc, has been going
error free for months now, but the last couple days it's flared up right
around the time of some log rotation and heavy i/o.
We'll see what happens after the centos 5.3 upgrade. We'll also be
upgrading the firmware on all the equallogics to the latest version.
--
Matthew Kent \ SA \ bravenet.com
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
open-iscsi group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~--~~~~--~~--~--~---
Re: Tuning iSCSI between Linux and NetAPP
I use a netapp filer...where are these values set? Host or Array? iscsi.iswt.max_ios_per_session 64 iscsi.max_connections_per_session 16 iscsi.max_ios_per_session64 On Apr 14, 6:40 am, benoit plessis plessis.ben...@gmail.com wrote: First i would ask why the hell ? The netapp filer is a very good CIFS/SMB share server. Using it as an iSCSI target -- which is not is primary function (netapp filer are more NAS than SAN) -- will only create limitations (unable to resize volume on the fly, unable to use wafl attributes to store windows security acl, ...) with no visible gain ... Also your server seem very overkill to me, i must hope it won't have to be just a samba=iscsi interface ... For iSCSI and netapp in general, first make sure that you have at least 10% of free space inside the volume, and 10% of free space inside the aggregate or else perf could suffer and more important you won't be able to launch the reallocate process (defrag). The following is the recommended netapp/iscsi optimisations, however open-iscsi doesn't support multiple connections per session now (iirc), so the best way to have parallel access is to use multipath iscsi.iswt.max_ios_per_session 64 iscsi.max_connections_per_session 16 iscsi.max_ios_per_session 64 2009/4/14 Frank Bonnet f.bon...@esiee.fr Hello I'm setting up a Samba server that will use iSCSI to access some shares on a NetAPP filer ( FAS 2050 ) I would like to know if some of you has already build such configuration and if there are some tricks to optimize it. The Linux server is a HP Proliant quad CPU and runs Debian Lenny, it has 16 Gb of RAM. Thanks a lot. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Multipath + iscsi + SLES10 SP2 / REDHAT 5.3 / Oracle Linux 5 update 3
iometer 32k write 0 read 0 randoms Equallogic is using this in their lab iozone with -I option and various settings dd + iostat On Apr 14, 1:57 pm, Mike Christie micha...@cs.wisc.edu wrote: Bart Van Assche wrote: On Mon, Apr 13, 2009 at 10:33 PM, Mike Christie micha...@cs.wisc.edu wrote: I think linux is just not so good with smaller IO sizes like 4K. I do not see good performance with Fibre Channel or iscsi. Can you elaborate on the above ? I have already measured a throughput of more than 60 MB/s when using the SRP protocol over an InfiniBand network with a block size of 4 KB blocks, which is definitely not bad. How does that compare to Windows or Solaris? Is that a 10 gig link? What tool were you using and what command did you run? I will try to replicate it here and see what I get. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Open iSCSI Performance on IBM
Have you made any headway with this issue? I'm having a write issue that seems to share some similarities with yours. On Apr 13, 8:14 am, Gonçalo Borges borges.gonc...@gmail.com wrote: Hi... Is /apoio04/b1 a scsi/iscsi disk or is it LVM/DM/RAID on top of a iscsi/scsi disk? /apoio04/ is a RAID1 of two disks accessible via iscsi (in the following tests, I changed the mount point from /apoio04/ to /iscsi04- lun0/ but they are exactly the same). Could you set the IO scheduler to noop echo noop /sys/block/sdX/queue/scheduler and see if that makes a difference. I checked the definition and I have [r...@core06 ~]# cat /sys/block/sdh/queue/scheduler noop anticipatory deadline [cfq] Now I've changed to [r...@core06 ~]# cat /sys/block/sdh/queue/scheduler [noop] anticipatory deadline cfq and I've run the tests again. This is what I got: [r...@core06 ~]# dd if=/dev/zero of=/iscsi04-lun0/b1 bs=64k count=125000 125000+0 records in 125000+0 records out 819200 bytes (8.2 GB) copied, 470.332 seconds, 17.4 MB/s [r...@core06 ~]# dd if=/dev/zero of=/iscsi04-lun0/b2 bs=128k count=62500 62500+0 records in 62500+0 records out 819200 bytes (8.2 GB) copied, 470.973 seconds, 17.4 MB/s Basically, the performance didn't increase :( And then also run iscsiadm -m session -P 3 [r...@core06 ~]# iscsiadm -m session -P 3 iSCSI Transport Class version 2.0-724 iscsiadm version 2.0-868 Target: iqn.1992-01.com.lsi:1535.600a0b80003ad11c490ade2d Current Portal: 10.131.2.14:3260,1 Persistent Portal: 10.131.2.14:3260,1 ** Interface: ** Iface Name: default Iface Transport: tcp Iface Initiatorname: iqn.1994-05.com.redhat:8c56e324f294 Iface IPaddress: 10.131.4.6 Iface HWaddress: default Iface Netdev: default SID: 37 iSCSI Connection State: LOGGED IN iSCSI Session State: Unknown Internal iscsid Session State: NO CHANGE Negotiated iSCSI params: HeaderDigest: None DataDigest: None MaxRecvDataSegmentLength: 131072 MaxXmitDataSegmentLength: 65536 FirstBurstLength: 8192 MaxBurstLength: 262144 ImmediateData: Yes InitialR2T: Yes MaxOutstandingR2T: 1 Attached SCSI devices: Host Number: 38 State: running scsi38 Channel 00 Id 0 Lun: 0 scsi38 Channel 00 Id 0 Lun: 1 scsi38 Channel 00 Id 0 Lun: 2 scsi38 Channel 00 Id 0 Lun: 3 scsi38 Channel 00 Id 0 Lun: 4 scsi38 Channel 00 Id 0 Lun: 5 scsi38 Channel 00 Id 0 Lun: 31 Current Portal: 10.131.2.13:3260,1 Persistent Portal: 10.131.2.13:3260,1 ** Interface: ** Iface Name: default Iface Transport: tcp Iface Initiatorname: iqn.1994-05.com.redhat:8c56e324f294 Iface IPaddress: 10.131.4.6 Iface HWaddress: default Iface Netdev: default SID: 38 iSCSI Connection State: LOGGED IN iSCSI Session State: Unknown Internal iscsid Session State: NO CHANGE Negotiated iSCSI params: HeaderDigest: None DataDigest: None MaxRecvDataSegmentLength: 131072 MaxXmitDataSegmentLength: 65536 FirstBurstLength: 8192 MaxBurstLength: 262144 ImmediateData: Yes InitialR2T: Yes MaxOutstandingR2T: 1 Attached SCSI devices: Host Number: 39 State: running scsi39 Channel 00 Id 0 Lun: 0 scsi39 Channel 00 Id 0 Lun: 1 scsi39 Channel 00 Id 0 Lun: 2 scsi39 Channel 00 Id 0 Lun: 3 scsi39 Channel 00 Id 0 Lun: 4 scsi39 Channel 00 Id 0 Lun: 5 scsi39 Channel 00 Id 0 Lun: 31 Current Portal: 10.131.2.16:3260,2 Persistent Portal: 10.131.2.16:3260,2 ** Interface: ** Iface Name: default Iface Transport: tcp
Multipath + iscsi + SLES10 SP2 / REDHAT 5.3 / Oracle Linux 5 update 3
I am having a major issue with multipath + iscsi write performance
with anything random or any sequential write with data sizes smaller
than 4meg (128k 64k 32k 16k 8k). With 32k block size, I am able to
get a maximum throughput of 33meg/s write. My performance gets cut by
a third with each smaller size, with 4k blocks giving me a whopping
4meg/s combined throughput. Now bumping the data size up to 32meg
gets me 160meg/sec throughput, and 64 gives me 190meg/s and finally to
top it out 128meg gives me 210megabytes/sec. My question is what
factors would limit my performance in the 4-128k range?
Some basics about my performance lab:
2 identical 1 gigabit paths (2 dual port intel pro 1000 MTs) in
separate pcie slots.
Hardware:
2 x Dell R900 6 quad core, 128gig ram, 2 x Dual port Intel Pro MT
Cisco 3750s with 32gigabit stackwise interconnect
2 x Dell Equallogic PS5000XV arrays
1 x Dell Equallogic PS5000E arrays
Operating systems
SLES 10 SP2 , RHEL5 Update 3, Oracle Linux 5 update 3
/etc/mutipath.conf
defaults {
udev_dir/dev
polling_interval10
selectorround-robin 0
path_grouping_policymultibus
getuid_callout /sbin/scsi_id -g -u -s /block/%n
prio_callout/bin/true
path_checkerreadsector0
features 1 queue_if_no_path
rr_min_io 10
max_fds 8192
# rr_weight priorities
failbackimmediate
# no_path_retry fail
# user_friendly_names yes
/etc/iscsi/iscsi.conf (non default values)
node.session.timeo.replacement_timeout = 15
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 30
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.MaxXmitDataSegmentLength = 262144
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 65536
Scheduler:
cat /sys/block/sdb/queue/scheduler
[noop] anticipatory deadline cfq
cat /sys/block/sdc/queue/scheduler
[noop] anticipatory deadline cfq
Command outputs:
iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-724
iscsiadm version 2.0-868
Target: iqn.2001-05.com.equallogic:0-8a0906-2c82dfd03-64c000cfe2249e37-
dc1stgdb15-sas-raid6
Current Portal: 10.1.253.13:3260,1
Persistent Portal: 10.1.253.10:3260,1
**
Interface:
**
Iface Name: ieth1
Iface Transport: tcp
Iface Initiatorname: iqn.2005-04.com.linux:dc1stgdb15
Iface IPaddress: 10.1.253.148
Iface HWaddress: default
Iface Netdev: eth1
SID: 3
iSCSI Connection State: LOGGED IN
iSCSI Session State: Unknown
Internal iscsid Session State: NO CHANGE
Negotiated iSCSI params:
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 65536
FirstBurstLength: 65536
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: No
MaxOutstandingR2T: 1
Attached SCSI devices:
Host Number: 5 State: running
scsi5 Channel 00 Id 0 Lun: 0
Attached scsi disk sdb State: running
Current Portal: 10.1.253.12:3260,1
Persistent Portal: 10.1.253.10:3260,1
**
Interface:
**
Iface Name: ieth2
Iface Transport: tcp
Iface Initiatorname: iqn.2005-04.com.linux:dc1stgdb15
Iface IPaddress: 10.1.253.48
Iface HWaddress: default
Iface Netdev: eth2
SID: 4
iSCSI Connection State: LOGGED IN
iSCSI Session State: Unknown
Internal iscsid Session State: NO CHANGE
Negotiated iSCSI params:
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 65536
FirstBurstLength: 65536
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: No
MaxOutstandingR2T: 1
Attached SCSI devices:
