Here are some things to look at. (NOTE: I'm definitely not a ZFS expert.)
The most common low hanging fruit, in iSCSI performance tweaks, are in
the networking stack setup and the following tweak.
http://www.opensolaris.org/jive/thread.jspa?messageID=95566𗕎
1) In your testing your using dd. If I'm not mistaken that is a single threaded
IO tool, ie. only one outstanding IO at a time. So you are dependent on the
latency of that single command. iSCSI is known to have a higher latency
that direct attach storage or fibre-channel. You have a good block size of
64K but I'm not sure how ZFS is breaking up that size. You might check your
performance with a different tool like iometer or vdbench.
1) Are there Errors. Checked /var/adm/messages for any type of errors
during your IO run. Check netstat for network errors and collisions.
2) Simplify the Problem. Take ZFS out of the picture. I'm not trying to
point a finger at ZFS but you could get a lot of useful data by taking it out
of the equation. Instead run your iSCSI target off a ramdisk/tmp, ufs, or
svm (preferred in that order.) Do you see a performance improvement to
the level you expect?
> A couple years ago I helped someone out with a NFS->iSCSI->ZFS performance
> problem. When we dug into the problem all the layers had their own issues
> and when put together they all magnified each others problems. In that case
> NFS has its problem of each file write it has to write the data, metadata,
> and other junk. Instead of a single write. Thats great for data sharing but
> bad on performance. ZFS then had its problem of needing to protect each
> one of the NFS writes with a SCSI SYNC CACHE command. Which is great
> but again adds a lot of overhead. Then you layer this on iSCSI. iSCSI is a
> great technology but in general its down fall is latency. Its best used
> with applications needing lower latency. It can handle either low or high
> bandwidth. There are ways to lower iSCSI's latency, but thats a much larger
> topic and most of those approaches are not supported with Solaris.
3) Evaluate Performance at the Different Layers.
a) It sounds like you already did the test of ZFS to direct storage vs iSCSI.
b) You might want to quickly do some sanity check of the network
performance with netperf.
c) iSCSI is a pretty dumb layer. It doesn't add much loss from what I have
seen. The loss tends to be in the other layers around it. !OR! iSCSI
errors
or poor configuration. Poor configuration:
i) iSCSI cmdsn windowing. This is a pretty old post but it still pretty
much
applies, http://blogs.sun.com/dweibel/entry/iscsi_kernel_visibilty.
You want to make sure the maxcmdsn windows is not sitting less that
current cmdsn. Otherwise the initiator is just sitting around doing
nothing.
I have also seen targets with a cmdsn window of 1, which is pretty sad
(unless its a tape device.)
ii) iSCSI command windows. You can check the initiators pending and active
queue counts to see if you have a large pending queue. Thats a sign of a
problem. The depending queue should run very low if not at 0. I thought
I blogged about getting at that data a couple years ago but I can't find
the
post now.
iii) Window size.
http://www.opensolaris.org/jive/thread.jspa?messageID=95566𗕎
In general those are the most common iSCSI performance issues, excluding
errors.
This message posted from opensolaris.org
_______________________________________________
storage-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/storage-discuss
