This sounds like the Nagle algorithm is getting in the way. I found this thread which describes almost the exact behavior you mentioned:
http://www.opensolaris.org/jive/thread.jspa?messageID=92345 I haven't tried this but according to the thread you can set TCP_NODELAY (disabling the Nagle algorithm) in iscsi.conf by adding "tcp-nodelay=1". Patrick Sullivan wrote: > When I write to a raw device (/dev/rdsk) connected through the Solaris iSCSI > Software initiator, I am seeing poor performance until I have more IO threads > than there are connections to the volume. > > See the following output for an example when using Sun's MPxIO: > > # iscsiadm list target -v > Target: > iqn.2001-05.com.equallogic:6-8a0900-0a33d1d01-c436e05358444ac7-v1 > Alias: - > TPGT: 0 > ISID: 4000002a0001 > Connections: 1 > CID: 0 > IP address (Local): 172.19.51.69:32775 > IP address (Peer): 172.19.50.53:3260 > > Target: > iqn.2001-05.com.equallogic:6-8a0900-0a33d1d01-c436e05358444ac7-v1 > Alias: - > TPGT: 0 > ISID: 4000002a0000 > Connections: 1 > CID: 0 > IP address (Local): 172.19.51.67:32800 > IP address (Peer): 172.19.50.53:3260 > > > Now checking performance I show the following: > > # ps -ef | grep dd > root 2732 2658 0 16:32:45 pts/1 0:02 dd if=/dev/zero > of=/dev/rdsk/c4t0690A018D0D1330AC74A445853E036C4d0s6 bs=64k > root 2721 2658 0 16:32:06 pts/1 0:02 dd if=/dev/zero > of=/dev/rdsk/c4t0690A018D0D1330AC74A445853E036C4d0s6 bs=64k > # iostat -dxz 2 > extended device statistics > device r/s w/s kr/s kw/s wait actv svc_t %w %b > sd4 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0 0 > sd14 0.2 0.0 1.6 0.0 0.0 0.0 7.5 0 0 > ssd0 0.0 0.2 0.2 11.1 0.0 0.0 17.8 0 0 > extended device statistics > device r/s w/s kr/s kw/s wait actv svc_t %w %b > ssd0 0.0 14.0 0.0 896.9 0.0 2.0 142.6 0 100 > extended device statistics > device r/s w/s kr/s kw/s wait actv svc_t %w %b > ssd0 0.0 11.0 0.0 704.0 0.0 2.0 181.7 0 100 > extended device statistics > device r/s w/s kr/s kw/s wait actv svc_t %w %b > ssd0 0.0 11.0 0.0 704.1 0.0 2.0 181.7 0 100 > ^C > > > Now, starting up a third dd to the same volume: > # ps -ef | grep dd > root 2732 2658 1 16:32:45 pts/1 0:02 dd if=/dev/zero > of=/dev/rdsk/c4t0690A018D0D1330AC74A445853E036C4d0s6 bs=64k > root 2721 2658 1 16:32:06 pts/1 0:02 dd if=/dev/zero > of=/dev/rdsk/c4t0690A018D0D1330AC74A445853E036C4d0s6 bs=64k > root 2771 2658 0 16:35:04 pts/1 0:00 dd if=/dev/zero > of=/dev/rdsk/c4t0690A018D0D1330AC74A445853E036C4d0s6 bs=64k > > > Checking performance yields: > [EMAIL PROTECTED]/]# iostat -dxz 2 > extended device statistics > device r/s w/s kr/s kw/s wait actv svc_t %w %b > sd4 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0 0 > sd14 0.2 0.0 1.6 0.0 0.0 0.0 7.5 0 0 > ssd0 0.0 0.2 0.2 12.1 0.0 0.0 17.2 0 0 > extended device statistics > device r/s w/s kr/s kw/s wait actv svc_t %w %b > ssd0 0.0 944.2 0.0 60431.0 0.0 2.9 3.1 0 100 > extended device statistics > device r/s w/s kr/s kw/s wait actv svc_t %w %b > ssd0 0.0 902.9 0.0 57787.7 0.0 2.9 3.2 0 100 > extended device statistics > device r/s w/s kr/s kw/s wait actv svc_t %w %b > ssd0 0.0 841.8 0.0 53876.3 0.0 2.9 3.5 0 100 > > I have reproduced this when not using Solaris MPxIO and I can provide the > same data as above if necessary. I have also tried this using dd to write to > different partitions and I have witnessed the same behavior. > > I believe I am seeing this issue also when using Solaris Volume manager. If I > connect to two separate volumes and I configure these whole volumes as > mirrors of each other, they sync at around 400-500k. If I divide these same > volumes into two separate slices and mirror both slices, I see the > performance increase to at least 30MB/sec. Also, if I connect to both of > these same volumes from a Linux server and I run a single threaded > performance test (using the same dd command), I am seeing performance of > 100MB/sec to the volume (looking through iostat -dx also). > > I have engaged Sun support, but nothing has come of that yet. I updated to > the Solaris 10 recommended patch cluster at the beginning of last week, and I > have also updated my OBP on my Sun Fire v250 server to whatever the latest > release was at the same time I updated the patch cluster (4.17.3 I think). > > Any clues as to what the issue is that I am seeing? > > Thanks. > > Patrick > > > This message posted from opensolaris.org > _______________________________________________ > storage-discuss mailing list > [email protected] > http://mail.opensolaris.org/mailman/listinfo/storage-discuss > > _______________________________________________ storage-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/storage-discuss
