An addition to this, I seem to be getting the following error when I
login:
Apr 14 08:24:54 dc1stgdb15 iscsid: received iferror -38
Apr 14 08:24:54 dc1stgdb15 last message repeated 2 times
Apr 14 08:24:54 dc1stgdb15 iscsid: connection2:0 is operational now
Apr 14 08:24:54 dc1stgdb15 iscsid: received iferror -38
Apr 14 08:24:54 dc1stgdb15 last message repeated 2 times
Apr 14 08:24:54 dc1stgdb15 iscsid: connection1:0 is operational now
On Apr 14, 1:12 pm, jnantel <nan...@hotmail.com> wrote:
> Well I've got some disconcerting news on this issue. No changes at
> any level alter the 34/meg throughput I get. I flushed multipath, blew
> away /var/lib/iscsi just in case. I also verified in /var/lib/iscsi
> the options got set. RHEL53 took my renice no problem.
>
> Some observations:
> Single interface iscsi gives me the exact same 34meg/sec
> Going with 2 interfaces it gives me 17meg/sec each interface
> Going with 4 interfaces it gives me 8meg/sec...etc..etc..etc.
> I can't seem to set node.conn[0].iscsi.MaxXmitDataSegmentLength =
> 262144 in a way that actually gets used.
> node.session.iscsi.MaxConnections = 1 can't find any docs on this,
> doubtful it is relevant.
>
> iscsiadm -m session -P 3 still gives me the default 65536 for xmit
> segment.
>
> The Equallogic has all its interfaces on the same SAN network, this is
> contrary to most implementations of multipath I've done. This is the
> vendor recommended deployment.
>
> Whatever is choking performance its consistently choking it down to
> the same level.
>
> On Apr 13, 5:33 pm, Mike Christie <micha...@cs.wisc.edu> wrote:
>
> > jnantel wrote:
>
> > > I am having a major issue with multipath + iscsi write performance
> > > with anything random or any sequential write with data sizes smaller
> > > than 4meg (128k 64k 32k 16k 8k). With 32k block size, I am able to
> > > get a maximum throughput of 33meg/s write. My performance gets cut by
> > > a third with each smaller size, with 4k blocks giving me a whopping
> > > 4meg/s combined throughput. Now bumping the data size up to 32meg
> > > gets me 160meg/sec throughput, and 64 gives me 190meg/s and finally to
> > > top it out 128meg gives me 210megabytes/sec. My question is what
> > > factors would limit my performance in the 4-128k range?
>
> > I think linux is just not so good with smaller IO sizes like 4K. I do
> > not see good performance with Fibre Channel or iscsi.
>
> > 64K+ should be fine, but you want to get lots of 64K+ IOs in flight. If
> > you run iostat or blktrace you should see more than 1 IO in flight. If
> > while the test is running if you
> > cat /sys/class/scsi_host/hostX/host_busy
> > you should also see lots of IO running.
>
> > What limits the number of IO? On the iscsi initiator side, it could be
> > params like node.session.cmds_max or node.session.queue_depth. For a
> > decent target like the ones you have I would increase
> > node.session.cmds_max to 1024 and increase node.session.queue_depth to 128.
>
> > What IO tool are you using? Are you doing direct IO or are you doing
> > file system IO? If you just use something like dd with bs=64K then you
> > are not going to get lots of IO running. I think you will get 1 64K IO
> > in flight, so throughput is not going to be high. If you use something
> > like disktest
> > disktest -PT -T30 -h1 -K128 -B64k -ID /dev/sdb
>
> > you should see a lot of IOs (depends on merging).
>
> > If you were using dd with bs=128m then that IO is going to get broken
> > down into lots of smaller IOs (probably around 256K), and so the pipe is
> > nice and full.
>
> > Another thing I noticed in RHEL is if you increase the nice value of the
> > iscsi threads it will increase write perforamnce sometimes. So for RHEL
> > or Oracle do
>
> > ps -u root | grep scsi_wq
>
> > Then patch the scsi_wq_%HOST_ID with the iscsiadm -m session -P 3 Host
> > Number. And then renive the thread to -20.
>
> > Also check the logs and make sure you do not see any conn error messages.
>
> > And then what do you get when running the IO test to the individual
> > iscsi disks instead of the dm one? Is there any difference? You might
> > want to change the rr_min_io. If you are sending smaller IOs then
> > rr_min_io of 10 is probably too small. The path is not going to get lots
> > of nice large IOs like you would want.
>
> > > Some basics about my performance lab:
>
> > > 2 identical 1 gigabit paths (2 dual port intel pro 1000 MTs) in
> > > separate pcie slots.
>
> > > Hardware:
> > > 2 x Dell R900 6 quad core, 128gig ram, 2 x Dual port Intel Pro MT
> > > Cisco 3750s with 32gigabit stackwise interconnect
> > > 2 x Dell Equallogic PS5000XV arrays
> > > 1 x Dell Equallogic PS5000E arrays
>
> > > Operating systems
> > > SLES 10 SP2 , RHEL5 Update 3, Oracle Linux 5 update 3
>
> > > /etc/mutipath.conf
>
> > > defaults {
> > > udev_dir /dev
> > > polling_interval 10
> > > selector "round-robin 0"
> > > path_grouping_policy multibus
> > > getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
> > > prio_callout /bin/true
> > > path_checker readsector0
> > > features "1 queue_if_no_path"
> > > rr_min_io 10
> > > max_fds 8192
> > > # rr_weight priorities
> > > failback immediate
> > > # no_path_retry fail
> > > # user_friendly_names yes
>
> > > /etc/iscsi/iscsi.conf (non default values)
>
> > > node.session.timeo.replacement_timeout = 15
> > > node.conn[0].timeo.noop_out_interval = 5
> > > node.conn[0].timeo.noop_out_timeout = 30
> > > node.session.cmds_max = 128
> > > node.session.queue_depth = 32
> > > node.session.iscsi.FirstBurstLength = 262144
> > > node.session.iscsi.MaxBurstLength = 16776192
> > > node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
> > > node.conn[0].iscsi.MaxXmitDataSegmentLength = 262144
>
> > > discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 65536
>
> > > Scheduler:
>
> > > cat /sys/block/sdb/queue/scheduler
> > > [noop] anticipatory deadline cfq
> > > cat /sys/block/sdc/queue/scheduler
> > > [noop] anticipatory deadline cfq
>
> > > Command outputs:
>
> > > iscsiadm -m session -P 3
> > > iSCSI Transport Class version 2.0-724
> > > iscsiadm version 2.0-868
> > > Target: iqn.2001-05.com.equallogic:0-8a0906-2c82dfd03-64c000cfe2249e37-
> > > dc1stgdb15-sas-raid6
> > > Current Portal: 10.1.253.13:3260,1
> > > Persistent Portal: 10.1.253.10:3260,1
> > > **********
> > > Interface:
> > > **********
> > > Iface Name: ieth1
> > > Iface Transport: tcp
> > > Iface Initiatorname: iqn.2005-04.com.linux:dc1stgdb15
> > > Iface IPaddress: 10.1.253.148
> > > Iface HWaddress: default
> > > Iface Netdev: eth1
> > > SID: 3
> > > iSCSI Connection State: LOGGED IN
> > > iSCSI Session State: Unknown
> > > Internal iscsid Session State: NO CHANGE
> > > ************************
> > > Negotiated iSCSI params:
> > > ************************
> > > HeaderDigest: None
> > > DataDigest: None
> > > MaxRecvDataSegmentLength: 262144
> > > MaxXmitDataSegmentLength: 65536
> > > FirstBurstLength: 65536
> > > MaxBurstLength: 262144
> > > ImmediateData: Yes
> > > InitialR2T: No
> > > MaxOutstandingR2T: 1
> > > ************************
> > > Attached SCSI devices:
> > > ************************
> > > Host Number: 5 State: running
> > > scsi5 Channel 00 Id 0 Lun: 0
> > > Attached scsi disk sdb State: running
> > > Current Portal: 10.1.253.12:3260,1
> > > Persistent Portal: 10.1.253.10:3260,1
> > > **********
> > > Interface:
> > > **********
> > > Iface Name: ieth2
> > > Iface Transport: tcp
> > > Iface Initiatorname: iqn.2005-04.com.linux:dc1stgdb15
> > > Iface IPaddress: 10.1.253.48
> > > Iface HWaddress: default
> > > Iface Netdev: eth2
> > > SID: 4
> > > iSCSI Connection State: LOGGED IN
> > > iSCSI Session State: Unknown
> > > Internal iscsid Session State: NO CHANGE
> > > ************************
> > > Negotiated iSCSI params:
> > > ************************
> > > HeaderDigest: None
> > > DataDigest: None
> > > MaxRecvDataSegmentLength: 262144
> > > MaxXmitDataSegmentLength: 65536
> > > FirstBurstLength: 65536
> > > MaxBurstLength: 262144
> > > ImmediateData: Yes
> > > InitialR2T: No
> > > MaxOutstandingR2T: 1
> > > ************************
> > > Attached SCSI devices:
> > > ************************
> > > Host Number: 6 State: running
> > > scsi6 Channel 00 Id 0 Lun: 0
> > > Attached scsi disk sdc State: running
>
> > > Jonathan Nantel
>
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~----------~----~----~----~------~----~------~--~---
