/var/lib/iscsi/static
What's the relevance of /var/lib/iscsi/static? I'm going to make an assumption an say that's where we can define static(permanent) target volumes that do not require discovery and will automatically logged in at boot? -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: iscsiadm -m iface + routing
Hi Joseph, we also have Dell equallogics in our shop. I can tell you A LOT about them and how to setup with multipath/iscsi..etc. Mike Christie is an incredible resource for sure, and he helped me a great deal with my setup. Be thankful you aren't using SLES10 SP2. Performance, you should be able to hit 200-210 meg/s write performance with RAID10 and around 150-160 with RAID6. Read performance for both usually pegged at 215-220meg/sec sustained. This is of course with sequential data. couple of goofy things I did notice in multipath.conf: 1. features "1 queue_if_no_path" and no_path_retry conflict 2. rr_min_io has the largest effect on performance in multipath. Play with different options (8/16/32/64) depending on your data sizes. You can reconfigure on the fly using "multipathd -k" and the "reconfigure" command. I am currently using 16 as I have lots of small data chunks and I want lots of io in flight. iscsid.conf node.session.cmds_max = 1024 node.session.queue_depth = 64 (96 works good here as well if you have small data sizes) node.conn[0].timeo.noop_out_interval = 15 node.conn[0].timeo.noop_out_timeout = 45 The above are my settings, Mike's probably work better. Couple of performance inhibiters I found...Make sure you have flow control set on the switch. TSO/SG/TX and RX offloading work by assembly packets into 64k chunks prior to sending...This is death to performance. If you have a beefy CPU turn them off like this: ethtool -K eth4 rx off ethtool -K eth4 tso off ethtool -K eth4 tx off ethtool -K eth4 sg off Tune your TCP stack for gigabit performance for sure. On Jul 22, 12:01 pm, "Hoot, Joseph" wrote: > Mike or others, > > Do you or others think that my "ping/CHAP" issue that I'm having -- aka > sessions resetting (logging out) could be related to the more recent > "ping timeout" and maxcmds patch that has been worked on and discussed? > > I am attempting to get my test systems updated to test this, but I > figured that I'd toss it out there to see if they might be related. > > Thanks, > Joe > > -Original Message- > From: open-iscsi@googlegroups.com [mailto:open-is...@googlegroups.com] > > On Behalf Of Mike Christie > Sent: Tuesday, July 14, 2009 6:01 PM > To: open-iscsi@googlegroups.com > Subject: Re: iscsiadm -m iface + routing > > On 07/13/2009 09:20 AM, Hoot, Joseph wrote: > > Mike, > > > Just as an FYI (in case you were most curious about this issue) I've > > narrowed this issue down to something with CHAP. On my EqualLogic, if > I > > disable CHAP, I can't reproduce this issue. > > > So I did the following. I after upgrading to the latest OEL 5.3 > release > > of the iscsi-initiator, I could still reproduce the problem. > Therefore, > > I did the following: > > > 1) Setup another test environment using the same hardware (physical > > different hardware, but all same firmware, models, etc..) > > 2) presented a new test volume from EqualLogic > > 3) ran the ping test (ping -Ieth2 192.168.0.19& ping -Ieth3 > > 192.168.0.19). > > 4) I couldn't reproduce the issue. > > 5) I checked what the difference were-- CHAP the only difference. > > 6) So I turned on CHAP authentication to the volume. > > 7) rm -rf /var/lib/iscsi/nodes/* /var/lib/iscsi/send_targets/* > > 8) rediscovered targets (after modifying /etc/iscsi/iscsid.conf with > > CHAP information) > > > node.session.auth.authmethod = CHAP > > node.session.auth.username = mychapuserhere > > node.session.auth.password = mypasshere > > > 9) ran the same ping test and was able to get iscsi sessions to fail > > within 2 minutes. > > 10) I wanted to prove that CHAP was the issue. So I logout out of all > > iscsi sessions. > > 11) I disabled CHAP on the EqualLogic > > 12) rediscovery targets and re-logged in to the sessions (without CHAP > > authentication) > > 13) ran the ping tests and couldn't break it after 30 minutes. > > 14) added CHAP again and was able to break the sessions within 2 > > minutes. > > > So definitely something odd with CHAP (my guess, either in open-iscsi > > code or EqualLogic code). I've asked Roger Lopez, from Dell, to > attempt > > to reproduce this in his lab. He has EqualLogic and Oracle VM > Servers. > > Oracle developers that I'm working with don't currently have access to > > an EqualLogic, but they are attempting to reproduce this with their > > iSCSI equipment as well. I'm going to setup port mirroring on our > > switch and run tcpdumps to see what I can get. > > This is strange because CHAP does not come into play in the normal IO > path. When we login we will do authentication with CHAP, but after that > succeeds it nevers comes up when doing IO. It would only come up again > when the session/connection is dropped/disconnected and we relogin. For > the relogin we will do the CHAP authentication again. > > Maybe some memory error where chap values are overwriting some other > memory. > > There was one recent fix by Dell, where when using CHAP they could > segfault iscsid. Here is the
Re: Tuning iscsi read performance with multipath Redhat 5.3 / SLES 10 SP2 / Oracle Linux / Equallogic
Just an update to this issue, it's still persisting. 210meg/sec raw
reads with 100-110meg/sec throughput on filesystem reads. I'm a bit
perplexed. Setting stride and aligning sectors in my mind is to help
with write performance not read performance. So I'm left with a
couple of questions. What does ext3 add to the equation for reads
that might cause this behavior.
Block device
File table
request merging (I'd like to tune this, but I've not seen any tunable
parameters here)
I'm missing a lot more obviously.
On Apr 24, 6:02 pm, jnantel wrote:
> Donald, thanks for the reply. This issue has me baffled. I can goof
> with the read ahead all I want but it has no effect on the performance
> with a filesystem. I must be missing a key buffer section that is
> starving my filesystem reads.
>
> Here is the output from iostat -k 5 during artificially generated read
> (dd if=/fs/disk0/testfile of=/dev/null bs=32k -c=100)
>
> This is reading a file residing on the ext3 filesystem on my raid6
> volume. Keep in mind I am using multipath:
>
> Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
> sdc 115.17 52090.22 0.00 260972 0
> sdd 0.00 0.00 0.00 0 0
> sde 109.78 49694.21 0.00 248968 0
> dm-0 249.30 101784.43 0.00 509940 0
>
> Same volume reading from the device itself( dd if=/dev/mapper/raid6
> of=/dev/null bs=32k -c=100):
>
> Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
> sdc 418.80 106905.60 0.00 534528 0
> sdd 0.00 0.00 0.00 0 0
> sde 901.80 106950.40 0.00 534752 0
> dm-0 53452.80 213811.20 0.00 1069056 0
>
> More detailed on ext3 performance
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.40 0.00 0.40 0.00 4.00
> 20.00 0.00 0.00 0.00 0.00
> sdb 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00
> sdc 12.40 0.20 110.80 0.40 50215.20 2.40
> 903.19 1.00 8.97 4.81 53.52
> sdd 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00
> sde 11.20 0.00 104.00 0.00 47205.60 0.00
> 907.80 0.91 8.77 4.65 48.32
> dm-0 0.00 0.00 238.40 0.60 97375.20 2.40
> 814.88 2.08 8.70 4.18 100.00
>
> On Apr 24, 3:14 pm, Donald Williams wrote:
>
> > Have you tried increasing the disk readahead value?
> > #blockdev --setra X /dev/
>
> > The default is 256. Use --getra to see current setting.
>
> > Setting it too high will probably hurt your database performance. Since
> > databases tend to be random, not sequential.
>
> > Don
>
> > On Fri, Apr 24, 2009 at 11:07 AM, jnantel wrote:
>
> > > If you recall my thread on tuning performance for writes. Now I am
> > > attempting to squeeze as much read performance as I can from my
> > > current setup. I've read a lot of the previous threads, and there has
> > > been mention of "miracle" settings that resolved slow reads vs
> > > writes. Unfortunately, most posts reference the effects and not the
> > > changes. If I were tuning for read performance in the 4k to 128k
> > > block range what would the best way to go about it?
>
> > > Observed behavior:
> > > - Read performance seems to be capped out at 110meg/sec
> > > - Write performance I get upwards of 190meg/sec
>
> > > Tuning options I'll be trying:
> > > block alignment (stride)
> > > Receiving buffers
> > > multipath min io changes
> > > iscsi cmd depth
>
> > > Hardware:
> > > 2 x Cisco 3750 with 32gig interconnect
> > > 2 x Dell R900 with 128gig ram and 1 broadcom Quad (5709) and 2 dual
> > > port intels (pro 1000/MT)
> > > 2 x Dell Equallogic PS5000XV with 15 x SAS in raid 10 config
>
> > > multipath.conf:
>
> > > device {
> > > vendor "EQLOGIC"
> > > product "100E-00"
> > > path_grouping_policy multibus
> > > getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
> > > features "1 queue_if_no_path"
> > > path_checker readsector0
> > > failback immediate
>
Re: Kpartx and logical DM partition
Let me clarify. In the context of an iscsi + multipath environment. On Apr 28, 3:10 pm, jnantel wrote: > What is the benefit of using a partition created with kpartx over the > device itself? I've tested using both, however, I don't have a clear > understanding of what it buys. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Kpartx and logical DM partition
What is the benefit of using a partition created with kpartx over the device itself? I've tested using both, however, I don't have a clear understanding of what it buys. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Tuning iscsi read performance with multipath Redhat 5.3 / SLES 10 SP2 / Oracle Linux / Equallogic
Donald, thanks for the reply. This issue has me baffled. I can goof
with the read ahead all I want but it has no effect on the performance
with a filesystem. I must be missing a key buffer section that is
starving my filesystem reads.
Here is the output from iostat -k 5 during artificially generated read
(dd if=/fs/disk0/testfile of=/dev/null bs=32k -c=100)
This is reading a file residing on the ext3 filesystem on my raid6
volume. Keep in mind I am using multipath:
Device:tpskB_read/skB_wrtn/skB_readkB_wrtn
sdc 115.17 52090.22 0.00 260972 0
sdd 0.00 0.00 0.00 0 0
sde 109.78 49694.21 0.00 248968 0
dm-0249.30101784.43 0.00 509940 0
Same volume reading from the device itself( dd if=/dev/mapper/raid6
of=/dev/null bs=32k -c=100):
Device:tpskB_read/skB_wrtn/skB_readkB_wrtn
sdc 418.80106905.60 0.00 534528 0
sdd 0.00 0.00 0.00 0 0
sde 901.80106950.40 0.00 534752 0
dm-0 53452.80213811.20 0.001069056 0
More detailed on ext3 performance
Device: rrqm/s wrqm/s r/s w/srkB/swkB/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.400.000.40 0.00 4.00
20.00 0.000.00 0.00 0.00
sdb 0.00 0.000.000.00 0.00 0.00
0.00 0.000.00 0.00 0.00
sdc 12.40 0.20 110.800.40 50215.20 2.40
903.19 1.008.97 4.81 53.52
sdd 0.00 0.000.000.00 0.00 0.00
0.00 0.000.00 0.00 0.00
sde 11.20 0.00 104.000.00 47205.60 0.00
907.80 0.918.77 4.65 48.32
dm-0 0.00 0.00 238.400.60 97375.20 2.40
814.88 2.088.70 4.18 100.00
On Apr 24, 3:14 pm, Donald Williams wrote:
> Have you tried increasing the disk readahead value?
> #blockdev --setra X /dev/
>
> The default is 256. Use --getra to see current setting.
>
> Setting it too high will probably hurt your database performance. Since
> databases tend to be random, not sequential.
>
> Don
>
> On Fri, Apr 24, 2009 at 11:07 AM, jnantel wrote:
>
> > If you recall my thread on tuning performance for writes. Now I am
> > attempting to squeeze as much read performance as I can from my
> > current setup. I've read a lot of the previous threads, and there has
> > been mention of "miracle" settings that resolved slow reads vs
> > writes. Unfortunately, most posts reference the effects and not the
> > changes. If I were tuning for read performance in the 4k to 128k
> > block range what would the best way to go about it?
>
> > Observed behavior:
> > - Read performance seems to be capped out at 110meg/sec
> > - Write performance I get upwards of 190meg/sec
>
> > Tuning options I'll be trying:
> > block alignment (stride)
> > Receiving buffers
> > multipath min io changes
> > iscsi cmd depth
>
> > Hardware:
> > 2 x Cisco 3750 with 32gig interconnect
> > 2 x Dell R900 with 128gig ram and 1 broadcom Quad (5709) and 2 dual
> > port intels (pro 1000/MT)
> > 2 x Dell Equallogic PS5000XV with 15 x SAS in raid 10 config
>
> > multipath.conf:
>
> > device {
> > vendor "EQLOGIC"
> > product "100E-00"
> > path_grouping_policy multibus
> > getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
> > features "1 queue_if_no_path"
> > path_checker readsector0
> > failback immediate
> > path_selector "round-robin 0"
> > rr_min_io 128
> > rr_weight priorities
> > }
>
> > iscsi settings:
>
> > node.tpgt = 1
> > node.startup = automatic
> > iface.hwaddress = default
> > iface.iscsi_ifacename = ieth10
> > iface.net_ifacename = eth10
> > iface.transport_name = tcp
> > node.discovery_address = 10.1.253.10
> > node.discovery_port = 3260
> > node.discovery_type = send_targets
> > node.session.initial_cmdsn = 0
> > node.session.initial_login_retry_max = 4
> > node.session.cmds_max = 1024
> > node.session.queue_depth = 128
> > node.session.auth.authmethod = None
> > node.session.timeo.replacement_timeout = 120
> > node.session.err_timeo.abort_timeout = 15
> > node.session.err_timeo.lu_reset_timeout = 30
> > node.session.err_timeo.host_reset_timeout = 60
> > node.session.iscsi.FastAbort =
Re: Tuning iscsi read performance with multipath Redhat 5.3 / SLES 10 SP2 / Oracle Linux / Equallogic
We are running with Async IO right now, it's yielding better results with multipath. On Apr 24, 5:06 pm, Konrad Rzeszutek wrote: > On Fri, Apr 24, 2009 at 02:14:43PM -0400, Donald Williams wrote: > > Have you tried increasing the disk readahead value? > > #blockdev --setra X /dev/ > > > The default is 256. Use --getra to see current setting. > > > Setting it too high will probably hurt your database performance. Since > > databases tend to be random, not sequential. > > I would think that the databases would open the disks with O_DIRECT > bypassing the block cache (And hence the disk readahead value isn't used > at all). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Tuning iscsi read performance with multipath Redhat 5.3 / SLES 10 SP2 / Oracle Linux / Equallogic
As an update:
new observed behavior:
- RAW disk read performance is phenomenal (200meg/sec)
- Ext3 performance is 100meg/sec and tps in iostat aren't going about
800 (50k with raw disk).
Some added info:
- This system has an oracle database on it and it's tuned for huge
pages..etc (see sysctl posted above)
On Apr 24, 12:07 pm, jnantel wrote:
> If you recall my thread on tuning performance for writes. Now I am
> attempting to squeeze as much read performance as I can from my
> current setup. I've read a lot of the previous threads, and there has
> been mention of "miracle" settings that resolved slow reads vs
> writes. Unfortunately, most posts reference the effects and not the
> changes. If I were tuning for read performance in the 4k to 128k
> block range what would the best way to go about it?
>
> Observed behavior:
> - Read performance seems to be capped out at 110meg/sec
> - Write performance I get upwards of 190meg/sec
>
> Tuning options I'll be trying:
> block alignment (stride)
> Receiving buffers
> multipath min io changes
> iscsi cmd depth
>
> Hardware:
> 2 x Cisco 3750 with 32gig interconnect
> 2 x Dell R900 with 128gig ram and 1 broadcom Quad (5709) and 2 dual
> port intels (pro 1000/MT)
> 2 x Dell Equallogic PS5000XV with 15 x SAS in raid 10 config
>
> multipath.conf:
>
> device {
> vendor "EQLOGIC"
> product "100E-00"
> path_grouping_policy multibus
> getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
> features "1 queue_if_no_path"
> path_checker readsector0
> failback immediate
> path_selector "round-robin 0"
> rr_min_io 128
> rr_weight priorities
>
> }
>
> iscsi settings:
>
> node.tpgt = 1
> node.startup = automatic
> iface.hwaddress = default
> iface.iscsi_ifacename = ieth10
> iface.net_ifacename = eth10
> iface.transport_name = tcp
> node.discovery_address = 10.1.253.10
> node.discovery_port = 3260
> node.discovery_type = send_targets
> node.session.initial_cmdsn = 0
> node.session.initial_login_retry_max = 4
> node.session.cmds_max = 1024
> node.session.queue_depth = 128
> node.session.auth.authmethod = None
> node.session.timeo.replacement_timeout = 120
> node.session.err_timeo.abort_timeout = 15
> node.session.err_timeo.lu_reset_timeout = 30
> node.session.err_timeo.host_reset_timeout = 60
> node.session.iscsi.FastAbort = Yes
> node.session.iscsi.InitialR2T = No
> node.session.iscsi.ImmediateData = Yes
> node.session.iscsi.FirstBurstLength = 262144
> node.session.iscsi.MaxBurstLength = 16776192
> node.session.iscsi.DefaultTime2Retain = 0
> node.session.iscsi.DefaultTime2Wait = 2
> node.session.iscsi.MaxConnections = 1
> node.session.iscsi.MaxOutstandingR2T = 1
> node.session.iscsi.ERL = 0
> node.conn[0].address = 10.1.253.10
> node.conn[0].port = 3260
> node.conn[0].startup = manual
> node.conn[0].tcp.window_size = 524288
> node.conn[0].tcp.type_of_service = 0
> node.conn[0].timeo.logout_timeout = 15
> node.conn[0].timeo.login_timeout = 15
> node.conn[0].timeo.auth_timeout = 45
> node.conn[0].timeo.noop_out_interval = 10
> node.conn[0].timeo.noop_out_timeout = 30
> node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
> node.conn[0].iscsi.HeaderDigest = None,CRC32C
> node.conn[0].iscsi.DataDigest = None
> node.conn[0].iscsi.IFMarker = No
> node.conn[0].iscsi.OFMarker = No
>
> /etc/sysctl.conf
>
> net.core.rmem_default= 65536
> net.core.rmem_max=2097152
> net.core.wmem_default = 65536
> net.core.wmem_max = 262144
> net.ipv4.tcp_mem= 98304 131072 196608
> net.ipv4.tcp_window_scaling=1
>
> #
> # Additional options for Oracle database server
> #ORACLE
> kernel.panic = 2
> kernel.panic_on_oops = 1
> net.ipv4.ip_local_port_range = 1024 65000
> net.core.rmem_default=262144
> net.core.wmem_default=262144
> net.core.rmem_max=524288
> net.core.wmem_max=524288
> fs.aio-max-nr=524288
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~--~~~~--~~--~--~---
Tuning iscsi read performance with multipath Redhat 5.3 / SLES 10 SP2 / Oracle Linux / Equallogic
If you recall my thread on tuning performance for writes. Now I am
attempting to squeeze as much read performance as I can from my
current setup. I've read a lot of the previous threads, and there has
been mention of "miracle" settings that resolved slow reads vs
writes. Unfortunately, most posts reference the effects and not the
changes. If I were tuning for read performance in the 4k to 128k
block range what would the best way to go about it?
Observed behavior:
- Read performance seems to be capped out at 110meg/sec
- Write performance I get upwards of 190meg/sec
Tuning options I'll be trying:
block alignment (stride)
Receiving buffers
multipath min io changes
iscsi cmd depth
Hardware:
2 x Cisco 3750 with 32gig interconnect
2 x Dell R900 with 128gig ram and 1 broadcom Quad (5709) and 2 dual
port intels (pro 1000/MT)
2 x Dell Equallogic PS5000XV with 15 x SAS in raid 10 config
multipath.conf:
device {
vendor "EQLOGIC"
product "100E-00"
path_grouping_policy multibus
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
features "1 queue_if_no_path"
path_checker readsector0
failback immediate
path_selector "round-robin 0"
rr_min_io 128
rr_weight priorities
}
iscsi settings:
node.tpgt = 1
node.startup = automatic
iface.hwaddress = default
iface.iscsi_ifacename = ieth10
iface.net_ifacename = eth10
iface.transport_name = tcp
node.discovery_address = 10.1.253.10
node.discovery_port = 3260
node.discovery_type = send_targets
node.session.initial_cmdsn = 0
node.session.initial_login_retry_max = 4
node.session.cmds_max = 1024
node.session.queue_depth = 128
node.session.auth.authmethod = None
node.session.timeo.replacement_timeout = 120
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.host_reset_timeout = 60
node.session.iscsi.FastAbort = Yes
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 2
node.session.iscsi.MaxConnections = 1
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.conn[0].address = 10.1.253.10
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.noop_out_interval = 10
node.conn[0].timeo.noop_out_timeout = 30
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.HeaderDigest = None,CRC32C
node.conn[0].iscsi.DataDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
/etc/sysctl.conf
net.core.rmem_default= 65536
net.core.rmem_max=2097152
net.core.wmem_default = 65536
net.core.wmem_max = 262144
net.ipv4.tcp_mem= 98304 131072 196608
net.ipv4.tcp_window_scaling=1
#
# Additional options for Oracle database server
#ORACLE
kernel.panic = 2
kernel.panic_on_oops = 1
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=524288
net.core.wmem_max=524288
fs.aio-max-nr=524288
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~--~~~~--~~--~--~---
Re: Multipath + iscsi + SLES10 SP2 / REDHAT 5.3 / Oracle Linux 5 update 3
Accross my SAN, tuned system:
ping -I eth2 -s 9000 10.1.253.48
PING 10.1.253.48 (10.1.253.48) from 10.1.253.48 eth2: 9000(9028) bytes
of data.
9008 bytes from 10.1.253.48: icmp_seq=1 ttl=64 time=0.074 ms
9008 bytes from 10.1.253.48: icmp_seq=2 ttl=64 time=0.013 ms
9008 bytes from 10.1.253.48: icmp_seq=3 ttl=64 time=0.012 ms
9008 bytes from 10.1.253.48: icmp_seq=4 ttl=64 time=0.011 ms
9008 bytes from 10.1.253.48: icmp_seq=5 ttl=64 time=0.012 ms
9008 bytes from 10.1.253.48: icmp_seq=6 ttl=64 time=0.012 ms
9008 bytes from 10.1.253.48: icmp_seq=7 ttl=64 time=0.012 ms
9008 bytes from 10.1.253.48: icmp_seq=8 ttl=64 time=0.011 ms
9008 bytes from 10.1.253.48: icmp_seq=9 ttl=64 time=0.012 ms
TSO, TCP checksum offload and things like that seem to have a big
effect on latency. If you look at how things like TSO work, their
intention is to save you CPU overhead...in my case I don't care about
overhead I've got 24 cores.
On Apr 17, 3:25 am, "Ulrich Windl"
wrote:
> On 16 Apr 2009 at 13:59, jnantel wrote:
>
>
>
> > FINAL RESULTS *
> > First of all I'd thank Mike Christie for all his help. Mike I'll
> > tapping your brain again for some read performance help.
>
> > This for the benefit of anyone using the Dell Equallogic PS5000XV
> > PS5000E with SLES10 SP2 / Redhat 5.3 / Centos 5.3 / Oracle Linux +
> > Multipath ( MPIO ) and open-iscsi ( iscsi ). Sorry about weird
> > formatting, making sure this is going get hit for people that were in
> > my predicament.
>
> When seeing your settings, I wonder what your network latency for jumbo
> frames is
> (e.g using ping). The timing is dependent on packet sizes. Here is what I
> have if
> everything is connected to one switch (and both ends are handling normal iSCSI
> traffic at the same time), started from Domain-0 of a XEN-virtualized machine
> that
> has 77 users logged on:
>
> # ping -s 9000 172.20.76.1
> PING 172.20.76.1 (172.20.76.1) 9000(9028) bytes of data.
> 9008 bytes from 172.20.76.1: icmp_seq=1 ttl=64 time=1.90 ms
> 9008 bytes from 172.20.76.1: icmp_seq=2 ttl=64 time=1.38 ms
> 9008 bytes from 172.20.76.1: icmp_seq=3 ttl=64 time=1.39 ms
> 9008 bytes from 172.20.76.1: icmp_seq=4 ttl=64 time=1.40 ms
> 9008 bytes from 172.20.76.1: icmp_seq=5 ttl=64 time=1.56 ms
> 9008 bytes from 172.20.76.1: icmp_seq=6 ttl=64 time=1.52 ms
> 9008 bytes from 172.20.76.1: icmp_seq=7 ttl=64 time=1.39 ms
> 9008 bytes from 172.20.76.1: icmp_seq=8 ttl=64 time=1.40 ms
> 9008 bytes from 172.20.76.1: icmp_seq=9 ttl=64 time=1.55 ms
> 9008 bytes from 172.20.76.1: icmp_seq=10 ttl=64 time=1.38 ms
>
> --- 172.20.76.1 ping statistics ---
> 10 packets transmitted, 10 received, 0% packet loss, time 9000ms
> rtt min/avg/max/mdev = 1.384/1.491/1.900/0.154 ms
> # ping 172.20.76.1
> PING 172.20.76.1 (172.20.76.1) 56(84) bytes of data.
> 64 bytes from 172.20.76.1: icmp_seq=1 ttl=64 time=0.253 ms
> 64 bytes from 172.20.76.1: icmp_seq=2 ttl=64 time=0.214 ms
> 64 bytes from 172.20.76.1: icmp_seq=3 ttl=64 time=0.223 ms
> 64 bytes from 172.20.76.1: icmp_seq=4 ttl=64 time=0.214 ms
> 64 bytes from 172.20.76.1: icmp_seq=5 ttl=64 time=0.215 ms
> 64 bytes from 172.20.76.1: icmp_seq=6 ttl=64 time=0.208 ms
> 64 bytes from 172.20.76.1: icmp_seq=7 ttl=64 time=0.270 ms
> 64 bytes from 172.20.76.1: icmp_seq=8 ttl=64 time=0.313 ms
>
> --- 172.20.76.1 ping statistics ---
> 8 packets transmitted, 8 received, 0% packet loss, time 6996ms
> rtt min/avg/max/mdev = 0.208/0.238/0.313/0.039 ms
>
> I think large queues are more important if the roundtrip delay is high. ANd
> don't
> forget that queue sizes are per device or session, so is uses some RAM.
>
> Regards,
> Ulrich
>
>
>
> > As from this thread my issue was amazingly slow performance with
> > sequential writes with my multipath, around 35 meg/s, configuration
> > when measured with IOMETER. First things first... THROW OUT IOMETER
> > FOR LINUX , it has problems with queue depth. With that said, with
> > default iscsi and multipath setup we saw between 60-80meg/sec
> > performance with multipath. In essence it was slower than single
> > interface in certain block sizes. When I got done my write performance
> > was pushing 180-190meg/sec with blocks as small as 4k ( sequential
> > write test using "dt").
>
> > Here are my tweaks:
>
> > After making any multipath changes do "multipath -F" then "multipath"
> > otherwise your changes won't take effect.
>
> > /etc/multipath.conf
>
> > device {
> > vendor "EQLOGIC"
> > product "100E-00"
> > path_grouping_policy multibus
> > getuid_callout "/sbin/scsi_id
Re: Time of Log-In
Are you aware of them fixing the Broadcom iscsi offload cards to support Jumbo frames? We have 2 5709s we turfed because of it. On Apr 16, 11:18 am, Mike Christie wrote: > Ulrich Windl wrote: > > Hello, > > > while thinking about an udev/multipath timing problem with device > > discovery, I > > wondered how difficult it would be to record and the report time of session > > establihment (i.e. log-in). "iscsiadm -m session -P 3" does not show that. > > Would > > that tinme be related to "SID"? > > No. > > I can add it though. I am still busy with work stuff trying to finish > adding bnx2i and making sure cxgb3i is ok. When that stuff gets finished > I will work on all your and eveyrone else's requests more. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Multipath + iscsi + SLES10 SP2 / REDHAT 5.3 / Oracle Linux 5 update 3
FINAL RESULTS *
First of all I'd thank Mike Christie for all his help. Mike I'll
tapping your brain again for some read performance help.
This for the benefit of anyone using the Dell Equallogic PS5000XV
PS5000E with SLES10 SP2 / Redhat 5.3 / Centos 5.3 / Oracle Linux +
Multipath ( MPIO ) and open-iscsi ( iscsi ). Sorry about weird
formatting, making sure this is going get hit for people that were in
my predicament.
As from this thread my issue was amazingly slow performance with
sequential writes with my multipath, around 35 meg/s, configuration
when measured with IOMETER. First things first... THROW OUT IOMETER
FOR LINUX , it has problems with queue depth. With that said, with
default iscsi and multipath setup we saw between 60-80meg/sec
performance with multipath. In essence it was slower than single
interface in certain block sizes. When I got done my write performance
was pushing 180-190meg/sec with blocks as small as 4k ( sequential
write test using "dt").
Here are my tweaks:
After making any multipath changes do "multipath -F" then "multipath"
otherwise your changes won't take effect.
/etc/multipath.conf
device {
vendor "EQLOGIC"
product "100E-00"
path_grouping_policy multibus
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
features "1 queue_if_no_path" < --- important
path_checker readsector0
failback immediate
path_selector "round-robin 0"
rr_min_io 512 < important, only works with large queue
depth and cms in iscsi.conf
rr_weight priorities
}
/etc/iscsi/iscsi.conf ( restarting iscsi seems to apply the configs
fine)
# To control how many commands the session will queue set
# node.session.cmds_max to an integer between 2 and 2048 that is also
# a power of 2. The default is 128.
node.session.cmds_max = 1024
# To control the device's queue depth set node.session.queue_depth
# to a value between 1 and 128. The default is 32.
node.session.queue_depth = 128
Other changes I've made are basic gigabit network tuning for large
transfers and turning off some congestion functions, some scheduler
changes (noop is amazing for sub 4k blocks but awful for 4meg chunks
or higher). I've turned off TSO on the network cards, apparently it's
not supported with jumbo frames and actually slows down performance.
dc1stgdb14:~ # ethtool -k eth7
Offload parameters for eth7:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
dc1stgdb14:~ # ethtool -k eth10
Offload parameters for eth10:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
dc1stgdb14:~ #
On Apr 13, 4:36 pm, jnantel wrote:
> I am having a major issue with multipath + iscsi writeperformance
> with anything random or any sequential write with data sizes smaller
> than 4meg (128k 64k 32k 16k 8k). With 32k block size, I am able to
> get a maximum throughput of 33meg/s write. Myperformancegets cut by
> a third with each smaller size, with 4k blocks giving me a whopping
> 4meg/s combined throughput. Now bumping the data size up to 32meg
> gets me 160meg/sec throughput, and 64 gives me 190meg/s and finally to
> top it out 128meg gives me 210megabytes/sec. My question is what
> factors would limit myperformancein the 4-128k range?
>
> Some basics about myperformancelab:
>
> 2 identical 1 gigabit paths (2 dual port intel pro 1000 MTs) in
> separate pcie slots.
>
> Hardware:
> 2 x Dell R900 6 quad core, 128gig ram, 2 x Dual port Intel Pro MT
> Cisco 3750s with 32gigabit stackwise interconnect
> 2 x Dell Equallogic PS5000XV arrays
> 1 x Dell Equallogic PS5000E arrays
>
> Operating systems
> SLES 10 SP2 , RHEL5 Update 3, Oracle Linux 5 update 3
>
> /etc/mutipath.conf
>
> defaults {
> udev_dir /dev
> polling_interval 10
> selector "round-robin 0"
> path_grouping_policy multibus
> getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
> prio_callout /bin/true
> path_checker readsector0
> features "1 queue_if_no_path"
> rr_min_io 10
> max_fds 8192
> # rr_weight priorities
> failback immediate
> # no_path_retry fail
> # user_friendly_names yes
>
> /etc/iscsi/iscsi.conf (non default values)
>
> node.session.timeo.replacement_timeout = 15
> node.conn[0].timeo.noop_out_interval = 5
> node.conn[0].timeo.noop_out_timeout = 30
> node.session.cmds_max = 128
> node.session.queue_depth = 32
> node.session.iscsi.FirstBurstLength = 262144
> node
Re: open-iscsi created only one disk but missing 2 more
What Target are you using? I beat my head on a issue when I first setup regarding access permissions. If you don't have the permissions right on the array, open-iscsi will let you see it but not log in. Another thing you may want to check, if you are sharing luns between systems, some arrays require you to explicitly allow that behavior. And last but not least, if you have a backend and a front end connection make sure you are connecting to the array with the right interface. Make sure one connection does not have a matching MTU *jumbo frames with the array it can cause strange behavior. On Apr 15, 10:27 pm, sundar mahadevan wrote: > I tried the same connecting another hard drive of 20 G with 3 logical > volumes namely: asm 17G , ocr 924M and vote 760M. > > iscsiadm -m node -T iqn.2001-04.com.ubuntu:scsi.disk.vg1.ocr -p 192.168.20.22 > -l > iscsiadm -m node -T iqn.2001-04.com.ubuntu:scsi.disk.vg1.vote -p > 192.168.20.22 -l > iscsiadm -m node -T iqn.2001-04.com.ubuntu:scsi.disk.vg1.asm -p 192.168.20.22 > -l > > Looks like it only created the first one: ocr > > Here is the log from /var/log/syslog > Apr 15 21:10:17 sunny2 kernel: [ 1603.409561] Loading iSCSI transport > class v2.0-870. > Apr 15 21:10:17 sunny2 kernel: [ 1603.486634] iscsi: registered transport > (tcp) > Apr 15 21:10:18 sunny2 kernel: [ 1603.848719] iscsi: registered transport > (iser) > Apr 15 21:10:18 sunny2 iscsid: iSCSI logger with pid=5912 started! > Apr 15 21:10:18 sunny2 kernel: [ 1604.408284] scsi2 : iSCSI Initiator > over TCP/IP > Apr 15 21:10:19 sunny2 iscsid: transport class version 2.0-870. iscsid > version 2.0-865 > Apr 15 21:10:19 sunny2 iscsid: iSCSI daemon with pid=5914 started! > Apr 15 21:10:19 sunny2 iscsid: Could not verify connection 1:2. Dropping > event. > Apr 15 21:10:19 sunny2 iscsid: Could not verify connection 1:2. Dropping > event. > Apr 15 21:10:20 sunny2 kernel: [ 1606.064265] scsi3 : iSCSI Initiator > over TCP/IP > Apr 15 21:10:21 sunny2 iscsid: Could not verify connection 2:3. Dropping > event. > Apr 15 21:10:21 sunny2 iscsid: Could not verify connection 2:3. Dropping > event. > Apr 15 21:10:21 sunny2 kernel: [ 1607.592257] scsi4 : iSCSI Initiator > over TCP/IP > Apr 15 21:10:22 sunny2 iscsid: Could not verify connection 3:4. Dropping > event. > Apr 15 21:10:22 sunny2 iscsid: Could not verify connection 3:4. Dropping > event. > Apr 15 21:10:23 sunny2 kernel: [ 1609.120249] scsi5 : iSCSI Initiator > over TCP/IP > Apr 15 21:10:24 sunny2 iscsid: Could not verify connection 4:5. Dropping > event. > Apr 15 21:10:24 sunny2 iscsid: Could not verify connection 4:5. Dropping > event. > Apr 15 21:15:00 sunny2 kernel: [ 1886.664257] scsi6 : iSCSI Initiator > over TCP/IP > Apr 15 21:15:01 sunny2 iscsid: Could not verify connection 5:6. Dropping > event. > Apr 15 21:15:01 sunny2 kernel: [ 1886.946886] scsi 6:0:0:0: > Direct-Access IET VIRTUAL-DISK 0 PQ: 0 ANSI: 4 > Apr 15 21:15:01 sunny2 kernel: [ 1886.952267] sd 6:0:0:0: [sdb] > 1892352 512-byte hardware sectors (969 MB) > Apr 15 21:15:01 sunny2 kernel: [ 1886.956338] sd 6:0:0:0: [sdb] Write > Protect is off > Apr 15 21:15:01 sunny2 kernel: [ 1886.956365] sd 6:0:0:0: [sdb] Mode > Sense: 77 00 00 08 > Apr 15 21:15:01 sunny2 kernel: [ 1886.964101] sd 6:0:0:0: [sdb] Write > cache: disabled, read cache: enabled, doesn't support DPO or FUA > Apr 15 21:15:01 sunny2 kernel: [ 1886.976089] sd 6:0:0:0: [sdb] > 1892352 512-byte hardware sectors (969 MB) > Apr 15 21:15:01 sunny2 kernel: [ 1886.986197] sd 6:0:0:0: [sdb] Write > Protect is off > Apr 15 21:15:01 sunny2 kernel: [ 1886.986228] sd 6:0:0:0: [sdb] Mode > Sense: 77 00 00 08 > Apr 15 21:15:01 sunny2 kernel: [ 1887.82] sd 6:0:0:0: [sdb] Write > cache: disabled, read cache: enabled, doesn't support DPO or FUA > Apr 15 21:15:01 sunny2 kernel: [ 1887.002741] sdb: unknown partition table > Apr 15 21:15:01 sunny2 kernel: [ 1887.042501] sd 6:0:0:0: [sdb] > Attached SCSI disk > Apr 15 21:15:01 sunny2 kernel: [ 1887.043773] sd 6:0:0:0: Attached > scsi generic sg1 type 0 > Apr 15 21:15:02 sunny2 iscsid: connection5:0 is operational now > Apr 15 21:15:20 sunny2 iscsid: Could not get host for sid 5. > Apr 15 21:15:20 sunny2 iscsid: could not get host_no for session 6. > Apr 15 21:15:20 sunny2 iscsid: could not find session info for session5 > Apr 15 21:15:20 sunny2 iscsid: session > [iqn.2001-04.com.ubuntu:scsi.disk.vg1.vote,192.168.20.22,3260] already > running. > Apr 15 21:15:28 sunny2 iscsid: Nop-out timedout after 15 seconds on > connection 5:0 state (3). Dropping session. > Apr 15 21:15:28 sunny2 iscsid: connection5:0 is operational after > recovery (1 attempts) > Apr 15 21:15:33 sunny2 iscsid: Could not get host for sid 5. > Apr 15 21:15:33 sunny2 iscsid: could not get host_no for session 6. > Apr 15 21:15:33 sunny2 iscsid: could not find session info for session5 > Apr 15 21:15:33 sunny2 iscsid: session > [iqn.2001-04.com.ubuntu:scsi.disk.vg1.asm,192.168.20.22,3260] already > running. > Apr 15 2
Re: Multipath + iscsi + SLES10 SP2 / REDHAT 5.3 / Oracle Linux 5 update 3
iometer 32k write 0 read 0 randoms Equallogic is using this in their lab iozone with -I option and various settings dd + iostat On Apr 14, 1:57 pm, Mike Christie wrote: > Bart Van Assche wrote: > > On Mon, Apr 13, 2009 at 10:33 PM, Mike Christie > > wrote: > >> I think linux is just not so good with smaller IO sizes like 4K. I do > >> not see good performance with Fibre Channel or iscsi. > > > Can you elaborate on the above ? I have already measured a throughput > > of more than 60 MB/s when using the SRP protocol over an InfiniBand > > network with a block size of 4 KB blocks, which is definitely not bad. > > How does that compare to Windows or Solaris? > > Is that a 10 gig link? > > What tool were you using and what command did you run? I will try to > replicate it here and see what I get. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Multipath + iscsi + SLES10 SP2 / REDHAT 5.3 / Oracle Linux 5 update 3
An addition to this, I seem to be getting the following error when I login: Apr 14 08:24:54 dc1stgdb15 iscsid: received iferror -38 Apr 14 08:24:54 dc1stgdb15 last message repeated 2 times Apr 14 08:24:54 dc1stgdb15 iscsid: connection2:0 is operational now Apr 14 08:24:54 dc1stgdb15 iscsid: received iferror -38 Apr 14 08:24:54 dc1stgdb15 last message repeated 2 times Apr 14 08:24:54 dc1stgdb15 iscsid: connection1:0 is operational now On Apr 14, 1:12 pm, jnantel wrote: > Well I've got some disconcerting news on this issue. No changes at > any level alter the 34/meg throughput I get. I flushed multipath, blew > away /var/lib/iscsi just in case. I also verified in /var/lib/iscsi > the options got set. RHEL53 took my renice no problem. > > Some observations: > Single interface iscsi gives me the exact same 34meg/sec > Going with 2 interfaces it gives me 17meg/sec each interface > Going with 4 interfaces it gives me 8meg/sec...etc..etc..etc. > I can't seem to set node.conn[0].iscsi.MaxXmitDataSegmentLength = > 262144 in a way that actually gets used. > node.session.iscsi.MaxConnections = 1 can't find any docs on this, > doubtful it is relevant. > > iscsiadm -m session -P 3 still gives me the default 65536 for xmit > segment. > > The Equallogic has all its interfaces on the same SAN network, this is > contrary to most implementations of multipath I've done. This is the > vendor recommended deployment. > > Whatever is choking performance its consistently choking it down to > the same level. > > On Apr 13, 5:33 pm, Mike Christie wrote: > > > jnantel wrote: > > > > I am having a major issue with multipath + iscsi write performance > > > with anything random or any sequential write with data sizes smaller > > > than 4meg (128k 64k 32k 16k 8k). With 32k block size, I am able to > > > get a maximum throughput of 33meg/s write. My performance gets cut by > > > a third with each smaller size, with 4k blocks giving me a whopping > > > 4meg/s combined throughput. Now bumping the data size up to 32meg > > > gets me 160meg/sec throughput, and 64 gives me 190meg/s and finally to > > > top it out 128meg gives me 210megabytes/sec. My question is what > > > factors would limit my performance in the 4-128k range? > > > I think linux is just not so good with smaller IO sizes like 4K. I do > > not see good performance with Fibre Channel or iscsi. > > > 64K+ should be fine, but you want to get lots of 64K+ IOs in flight. If > > you run iostat or blktrace you should see more than 1 IO in flight. If > > while the test is running if you > > cat /sys/class/scsi_host/hostX/host_busy > > you should also see lots of IO running. > > > What limits the number of IO? On the iscsi initiator side, it could be > > params like node.session.cmds_max or node.session.queue_depth. For a > > decent target like the ones you have I would increase > > node.session.cmds_max to 1024 and increase node.session.queue_depth to 128. > > > What IO tool are you using? Are you doing direct IO or are you doing > > file system IO? If you just use something like dd with bs=64K then you > > are not going to get lots of IO running. I think you will get 1 64K IO > > in flight, so throughput is not going to be high. If you use something > > like disktest > > disktest -PT -T30 -h1 -K128 -B64k -ID /dev/sdb > > > you should see a lot of IOs (depends on merging). > > > If you were using dd with bs=128m then that IO is going to get broken > > down into lots of smaller IOs (probably around 256K), and so the pipe is > > nice and full. > > > Another thing I noticed in RHEL is if you increase the nice value of the > > iscsi threads it will increase write perforamnce sometimes. So for RHEL > > or Oracle do > > > ps -u root | grep scsi_wq > > > Then patch the scsi_wq_%HOST_ID with the iscsiadm -m session -P 3 Host > > Number. And then renive the thread to -20. > > > Also check the logs and make sure you do not see any conn error messages. > > > And then what do you get when running the IO test to the individual > > iscsi disks instead of the dm one? Is there any difference? You might > > want to change the rr_min_io. If you are sending smaller IOs then > > rr_min_io of 10 is probably too small. The path is not going to get lots > > of nice large IOs like you would want. > > > > Some basics about my performance lab: > > > > 2 identical 1 gigabit paths (2 dual port intel pro 1000 MTs) in > > > separate pcie slots. > > > > Hardware: > > > 2 x Dell R900 6 quad core, 128gig ram, 2 x Dual
Re: Tuning iSCSI between Linux and NetAPP
I use a netapp filer...where are these values set? Host or Array?
iscsi.iswt.max_ios_per_session 64
iscsi.max_connections_per_session 16
iscsi.max_ios_per_session64
On Apr 14, 6:40 am, benoit plessis wrote:
> First i would ask why the hell ?
>
> The netapp filer is a very good CIFS/SMB share server. Using it as an iSCSI
> target -- which is not is primary
> function (netapp filer are more NAS than SAN) -- will only create
> limitations (unable to resize volume on the fly,
> unable to use wafl attributes to store windows security acl, ...) with no
> visible gain ...
>
> Also your server seem very overkill to me, i must hope it won't have to be
> just a samba<=>iscsi interface ...
>
> For iSCSI and netapp in general, first make sure that you have at least
> 10% of free space inside the volume, and 10% of free space inside the
> aggregate or else perf could
> suffer and more important you won't be able to launch the reallocate process
> ("defrag").
>
> The following is the recommended netapp/iscsi optimisations, however
> open-iscsi doesn't support multiple
> connections per session now (iirc), so the best way to have parallel access
> is to use multipath
>
> iscsi.iswt.max_ios_per_session 64
> iscsi.max_connections_per_session 16
> iscsi.max_ios_per_session 64
>
> 2009/4/14 Frank Bonnet
>
>
>
> > Hello
>
> > I'm setting up a Samba server that will use iSCSI to access
> > some shares on a NetAPP filer ( FAS 2050 )
>
> > I would like to know if some of you has already build such
> > configuration and if there are some tricks to optimize it.
>
> > The Linux server is a HP Proliant quad CPU and runs
> > Debian Lenny, it has 16 Gb of RAM.
>
> > Thanks a lot.
>
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~--~~~~--~~--~--~---
Re: equallogic - load balancing and xfs
I've had a similar issue with SLES10 SP2
This is what my multipathd config looks like now:
efaults {
udev_dir/dev
polling_interval10
selector"round-robin 0"
# path_grouping_policymultibus
# getuid_callout "/lib/udev/scsi_id -g -u -s /block/%n"
# prioconst
# path_checkerdirectio
no_path_retry 5
features"1 queue_if_no_path"
rr_min_io 10
max_fds 8192
rr_weight priorities
failbackimmediate
# no_path_retry 5
user_friendly_names no
}
no_path_retry has to be commented out in order for the
queue_if_no_path to work
I've also played with the no-op timeouts as well:
node.conn[0].timeo.noop_out_interval = 10
node.conn[0].timeo.noop_out_timeout = 30
On Apr 13, 9:23 pm, Matthew Kent wrote:
> On Mon, 2009-04-13 at 17:28 -0500, Mike Christie wrote:
> > Matthew Kent wrote:
> > > On Mon, 2009-04-13 at 15:44 -0500, Mike Christie wrote:
> > >> Matthew Kent wrote:
> > >>> Can anyone suggest a timeout I might be hitting or a setting I'm
> > >>> missing?
>
> > >>> The run down:
>
> > >>> - EqualLogic target
> > >>> - CentOS 5.2 client
> > >> You will want to upgrade that to 5.3 when you can. The iscsi code in
> > >> there fixes a bug where the initiator dropped the session when it should
> > >> not.
>
> > > Will do, probably Wednesday night and we'll see if this goes away. I'll
> > > be sure to follow up for the archives.
>
> > >>> - xfs > lvm > iscsi
>
> > >>> During a period of high load the EqualLogic decides to load balance:
>
> > >>> INFO 4/13/09 12:08:29 AM eql3 iSCSI session to target
> > >>> '20.20.20.31:3260,
> > >>> iqn.2001-05.com.equallogic:0-8a0906-b7f6d3801-2b2000d0f5347d9a-foo' from
> > >>> initiator '20.20.20.92:51274, iqn.1994-05.com.redhat:a62ba20db72' was
> > >>> closed. Load balancing request was received on the array.
>
> > >> So is this what you get in the EQL log when it decides to load balance
> > >> the initiator and send us to a different portal?
>
> > > Yes, a straight copy from event log in the java web interface.
>
> > >>> INFO 4/13/09 12:08:31 AM eql3 iSCSI login to target
> > >>> '20.20.20.32:3260,
> > >>> iqn.2001-05.com.equallogic:0-8a0906-b7f6d3801-2b2000d0f5347d9a-foo' from
> > >>> initiator '20.20.20.92:44805, iqn.1994-05.com.redhat:a62ba20db72'
> > >>> successful, using standard frame length.
>
> > >>> on the client see I get:
>
> > >>> Apr 13 00:08:29 moo kernel: [4576850.161324] sd 5:0:0:0: SCSI error:
> > >>> return code = 0x0002
>
> > >>> Apr 13 00:08:29 moo kernel: [4576850.161330] end_request: I/O error, dev
> > >>> sdc, sector 113287552
>
> > >>> Apr 13 00:08:32 moo kernel: [4576852.470879] I/O error in filesystem
> > >>> ("dm-10") meta-data dev dm-10 block 0x6c0a000
> > >> Are you using dm-multipath over iscsi? Does this load balance issue
> > >> affect all the paths at the same time? What is your multipath
> > >> no_path_retry value? I think you might want to set that higher to avoid
> > >> the FS from getting IO errors at this time if all paths are affected at
> > >> the same time.
>
> > > Not using multipath on this one.
>
> > Do you have xfs on sdc or is there something like LVM or RAID on top of sdc?
>
> > That is really strange then. 0x0002 is DID_BUS_BUSY. The iscsi
> > initiator layer would return this when the target does its load
> > balancing. The initiator does this to ask he scsi layer to retry the IO.
> > If dm-multipath was used then it is failed to the multipath layer right
> > away. If dm-multipath is not used then we get 5 retries so we should not
> > see the error if there was only the one rebalancing at the time. If
> > there was a bunch of load rebalancing within a couple minutes then it
> > makes sense.
>
> Yeah xfs on top of lvm, no multipath.
>
> Logs only show the one load balancing request around that time.
>
> Funny thing is this system, and the load balancing etc, has been going
> error free for months now, but the last couple days it's flared up right
> around the time of some log rotation and heavy i/o.
>
> We'll see what happens after the centos 5.3 upgrade. We'll also be
> upgrading the firmware on all the equallogics to the latest version.
> --
> Matthew Kent \ SA \ bravenet.com
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~--~~~~--~~--~--~---
Re: Multipath + iscsi + SLES10 SP2 / REDHAT 5.3 / Oracle Linux 5 update 3
Well I've got some disconcerting news on this issue. No changes at
any level alter the 34/meg throughput I get. I flushed multipath, blew
away /var/lib/iscsi just in case. I also verified in /var/lib/iscsi
the options got set. RHEL53 took my renice no problem.
Some observations:
Single interface iscsi gives me the exact same 34meg/sec
Going with 2 interfaces it gives me 17meg/sec each interface
Going with 4 interfaces it gives me 8meg/sec...etc..etc..etc.
I can't seem to set node.conn[0].iscsi.MaxXmitDataSegmentLength =
262144 in a way that actually gets used.
node.session.iscsi.MaxConnections = 1can't find any docs on this,
doubtful it is relevant.
iscsiadm -m session -P 3 still gives me the default 65536 for xmit
segment.
The Equallogic has all its interfaces on the same SAN network, this is
contrary to most implementations of multipath I've done. This is the
vendor recommended deployment.
Whatever is choking performance its consistently choking it down to
the same level.
On Apr 13, 5:33 pm, Mike Christie wrote:
> jnantel wrote:
>
> > I am having a major issue with multipath + iscsi write performance
> > with anything random or any sequential write with data sizes smaller
> > than 4meg (128k 64k 32k 16k 8k). With 32k block size, I am able to
> > get a maximum throughput of 33meg/s write. My performance gets cut by
> > a third with each smaller size, with 4k blocks giving me a whopping
> > 4meg/s combined throughput. Now bumping the data size up to 32meg
> > gets me 160meg/sec throughput, and 64 gives me 190meg/s and finally to
> > top it out 128meg gives me 210megabytes/sec. My question is what
> > factors would limit my performance in the 4-128k range?
>
> I think linux is just not so good with smaller IO sizes like 4K. I do
> not see good performance with Fibre Channel or iscsi.
>
> 64K+ should be fine, but you want to get lots of 64K+ IOs in flight. If
> you run iostat or blktrace you should see more than 1 IO in flight. If
> while the test is running if you
> cat /sys/class/scsi_host/hostX/host_busy
> you should also see lots of IO running.
>
> What limits the number of IO? On the iscsi initiator side, it could be
> params like node.session.cmds_max or node.session.queue_depth. For a
> decent target like the ones you have I would increase
> node.session.cmds_max to 1024 and increase node.session.queue_depth to 128.
>
> What IO tool are you using? Are you doing direct IO or are you doing
> file system IO? If you just use something like dd with bs=64K then you
> are not going to get lots of IO running. I think you will get 1 64K IO
> in flight, so throughput is not going to be high. If you use something
> like disktest
> disktest -PT -T30 -h1 -K128 -B64k -ID /dev/sdb
>
> you should see a lot of IOs (depends on merging).
>
> If you were using dd with bs=128m then that IO is going to get broken
> down into lots of smaller IOs (probably around 256K), and so the pipe is
> nice and full.
>
> Another thing I noticed in RHEL is if you increase the nice value of the
> iscsi threads it will increase write perforamnce sometimes. So for RHEL
> or Oracle do
>
> ps -u root | grep scsi_wq
>
> Then patch the scsi_wq_%HOST_ID with the iscsiadm -m session -P 3 Host
> Number. And then renive the thread to -20.
>
> Also check the logs and make sure you do not see any conn error messages.
>
> And then what do you get when running the IO test to the individual
> iscsi disks instead of the dm one? Is there any difference? You might
> want to change the rr_min_io. If you are sending smaller IOs then
> rr_min_io of 10 is probably too small. The path is not going to get lots
> of nice large IOs like you would want.
>
>
>
> > Some basics about my performance lab:
>
> > 2 identical 1 gigabit paths (2 dual port intel pro 1000 MTs) in
> > separate pcie slots.
>
> > Hardware:
> > 2 x Dell R900 6 quad core, 128gig ram, 2 x Dual port Intel Pro MT
> > Cisco 3750s with 32gigabit stackwise interconnect
> > 2 x Dell Equallogic PS5000XV arrays
> > 1 x Dell Equallogic PS5000E arrays
>
> > Operating systems
> > SLES 10 SP2 , RHEL5 Update 3, Oracle Linux 5 update 3
>
> > /etc/mutipath.conf
>
> > defaults {
> > udev_dir /dev
> > polling_interval 10
> > selector "round-robin 0"
> > path_grouping_policy multibus
> > getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
> > prio_callout /bin/true
> > path_checker readsector0
> > features "1 queue_if_no_path"
> > rr_min_io 10
> >
Multipath + iscsi + SLES10 SP2 / REDHAT 5.3 / Oracle Linux 5 update 3
I am having a major issue with multipath + iscsi write performance
with anything random or any sequential write with data sizes smaller
than 4meg (128k 64k 32k 16k 8k). With 32k block size, I am able to
get a maximum throughput of 33meg/s write. My performance gets cut by
a third with each smaller size, with 4k blocks giving me a whopping
4meg/s combined throughput. Now bumping the data size up to 32meg
gets me 160meg/sec throughput, and 64 gives me 190meg/s and finally to
top it out 128meg gives me 210megabytes/sec. My question is what
factors would limit my performance in the 4-128k range?
Some basics about my performance lab:
2 identical 1 gigabit paths (2 dual port intel pro 1000 MTs) in
separate pcie slots.
Hardware:
2 x Dell R900 6 quad core, 128gig ram, 2 x Dual port Intel Pro MT
Cisco 3750s with 32gigabit stackwise interconnect
2 x Dell Equallogic PS5000XV arrays
1 x Dell Equallogic PS5000E arrays
Operating systems
SLES 10 SP2 , RHEL5 Update 3, Oracle Linux 5 update 3
/etc/mutipath.conf
defaults {
udev_dir/dev
polling_interval10
selector"round-robin 0"
path_grouping_policymultibus
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout/bin/true
path_checkerreadsector0
features "1 queue_if_no_path"
rr_min_io 10
max_fds 8192
# rr_weight priorities
failbackimmediate
# no_path_retry fail
# user_friendly_names yes
/etc/iscsi/iscsi.conf (non default values)
node.session.timeo.replacement_timeout = 15
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 30
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.MaxXmitDataSegmentLength = 262144
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 65536
Scheduler:
cat /sys/block/sdb/queue/scheduler
[noop] anticipatory deadline cfq
cat /sys/block/sdc/queue/scheduler
[noop] anticipatory deadline cfq
Command outputs:
iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-724
iscsiadm version 2.0-868
Target: iqn.2001-05.com.equallogic:0-8a0906-2c82dfd03-64c000cfe2249e37-
dc1stgdb15-sas-raid6
Current Portal: 10.1.253.13:3260,1
Persistent Portal: 10.1.253.10:3260,1
**
Interface:
**
Iface Name: ieth1
Iface Transport: tcp
Iface Initiatorname: iqn.2005-04.com.linux:dc1stgdb15
Iface IPaddress: 10.1.253.148
Iface HWaddress: default
Iface Netdev: eth1
SID: 3
iSCSI Connection State: LOGGED IN
iSCSI Session State: Unknown
Internal iscsid Session State: NO CHANGE
Negotiated iSCSI params:
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 65536
FirstBurstLength: 65536
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: No
MaxOutstandingR2T: 1
Attached SCSI devices:
Host Number: 5 State: running
scsi5 Channel 00 Id 0 Lun: 0
Attached scsi disk sdb State: running
Current Portal: 10.1.253.12:3260,1
Persistent Portal: 10.1.253.10:3260,1
**
Interface:
**
Iface Name: ieth2
Iface Transport: tcp
Iface Initiatorname: iqn.2005-04.com.linux:dc1stgdb15
Iface IPaddress: 10.1.253.48
Iface HWaddress: default
Iface Netdev: eth2
SID: 4
iSCSI Connection State: LOGGED IN
iSCSI Session State: Unknown
Internal iscsid Session State: NO CHANGE
Negotiated iSCSI params:
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 65536
FirstBurstLength: 65536
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: No
MaxOutstandingR2T: 1
Attached SCSI devices:
*
Re: Open iSCSI Performance on IBM
Have you made any headway with this issue? I'm having a write issue that seems to share some similarities with yours. On Apr 13, 8:14 am, Gonçalo Borges wrote: > Hi... > > > Is /apoio04/b1 a scsi/iscsi disk or is it LVM/DM/RAID on top of a > > iscsi/scsi disk? > > /apoio04/ is a RAID1 of two disks accessible via iscsi (in the > following tests, I changed the mount point from /apoio04/ to /iscsi04- > lun0/ but they are exactly the same). > > > > > Could you set the IO scheduler to noop > > echo noop > /sys/block/sdX/queue/scheduler and see if that makes a > > difference. > > I checked the definition and I have > > [r...@core06 ~]# cat /sys/block/sdh/queue/scheduler > noop anticipatory deadline [cfq] > > Now I've changed to > > [r...@core06 ~]# cat /sys/block/sdh/queue/scheduler > [noop] anticipatory deadline cfq > > and I've run the tests again. This is what I got: > > [r...@core06 ~]# dd if=/dev/zero of=/iscsi04-lun0/b1 bs=64k > count=125000 > 125000+0 records in > 125000+0 records out > 819200 bytes (8.2 GB) copied, 470.332 seconds, 17.4 MB/s > > [r...@core06 ~]# dd if=/dev/zero of=/iscsi04-lun0/b2 bs=128k > count=62500 > 62500+0 records in > 62500+0 records out > 819200 bytes (8.2 GB) copied, 470.973 seconds, 17.4 MB/s > > Basically, the performance didn't increase :( > > > And then also run > > iscsiadm -m session -P 3 > > [r...@core06 ~]# iscsiadm -m session -P 3 > iSCSI Transport Class version 2.0-724 > iscsiadm version 2.0-868 > Target: iqn.1992-01.com.lsi:1535.600a0b80003ad11c490ade2d > Current Portal: 10.131.2.14:3260,1 > Persistent Portal: 10.131.2.14:3260,1 > ** > Interface: > ** > Iface Name: default > Iface Transport: tcp > Iface Initiatorname: iqn.1994-05.com.redhat:8c56e324f294 > Iface IPaddress: 10.131.4.6 > Iface HWaddress: default > Iface Netdev: default > SID: 37 > iSCSI Connection State: LOGGED IN > iSCSI Session State: Unknown > Internal iscsid Session State: NO CHANGE > > Negotiated iSCSI params: > > HeaderDigest: None > DataDigest: None > MaxRecvDataSegmentLength: 131072 > MaxXmitDataSegmentLength: 65536 > FirstBurstLength: 8192 > MaxBurstLength: 262144 > ImmediateData: Yes > InitialR2T: Yes > MaxOutstandingR2T: 1 > > Attached SCSI devices: > > Host Number: 38 State: running > scsi38 Channel 00 Id 0 Lun: 0 > scsi38 Channel 00 Id 0 Lun: 1 > scsi38 Channel 00 Id 0 Lun: 2 > scsi38 Channel 00 Id 0 Lun: 3 > scsi38 Channel 00 Id 0 Lun: 4 > scsi38 Channel 00 Id 0 Lun: 5 > scsi38 Channel 00 Id 0 Lun: 31 > Current Portal: 10.131.2.13:3260,1 > Persistent Portal: 10.131.2.13:3260,1 > ** > Interface: > ** > Iface Name: default > Iface Transport: tcp > Iface Initiatorname: iqn.1994-05.com.redhat:8c56e324f294 > Iface IPaddress: 10.131.4.6 > Iface HWaddress: default > Iface Netdev: default > SID: 38 > iSCSI Connection State: LOGGED IN > iSCSI Session State: Unknown > Internal iscsid Session State: NO CHANGE > > Negotiated iSCSI params: > > HeaderDigest: None > DataDigest: None > MaxRecvDataSegmentLength: 131072 > MaxXmitDataSegmentLength: 65536 > FirstBurstLength: 8192 > MaxBurstLength: 262144 > ImmediateData: Yes > InitialR2T: Yes > MaxOutstandingR2T: 1 > > Attached SCSI devices: > > Host Number: 39 State: running > scsi39 Channel 00 Id 0 Lun: 0 > scsi39 Channel 00 Id 0 Lun: 1 > scsi39 Channel 00 Id 0 Lun: 2 > scsi39 Channel 00 Id 0 Lun: 3 > scsi39 Channel 00 Id 0 Lun: 4 > scsi39 Channel 00 Id 0 Lun: 5 > scsi39 Channel 00 Id 0 Lun: 31 > Current Portal: 10.131.2.16:3260,2 > Persistent Portal: 10.131.2.16:3260,2 > ** > Interface: >
