Carsten Aulbert schrieb: > Hi Thomas, > > Thomas Maier-Komor wrote: > >> Carsten, >> >> the summary looks like you are using mbuffer. Can you elaborate on what >> options you are passing to mbuffer? Maybe changing the blocksize to be >> consistent with the recordsize of the zpool could improve performance. >> Is the buffer running full or is it empty most of the time? Are you sure >> that the network connection is 10Gb/s all the way through from machine >> to machine? > > Well spotted :) > > right now plain mbuffer with plenty of buffer (-m 2048M) on both ends > and I have not seen any buffer exceeding the 10% watermark level. The > network connection are via Neterion XFrame II Sun Fire NICs then via CX4 > cables to our core switch where both boxes are directly connected > (WovenSystmes EFX1000). netperf tells me that the TCP performance is > close to 7.5 GBit/s duplex and if I use > > cat /dev/zero | mbuffer | socat ---> socat | mbuffer > /dev/null > > I easily see speeds of about 350-400 MB/s so I think the network is fine. > > Cheers > > Carsten > _______________________________________________ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
I don't know socat or what benefit it gives you, but have you tried using mbuffer to send and receive directly (options -I and -O)? Additionally, try to set the block size of mbuffer to the recordsize of zfs (usually 128k): receiver$ mbuffer -I sender:10000 -s 128k -m 2048M | zfs receive sender$ zfs send blabla | mbuffer -s 128k -m 2048M -O receiver:10000 As transmitting from /dev/zero to /dev/null is at a rate of 350MB/s, I guess, you are really hitting the maximum speed of your zpool. From my understanding, I'd guess sending is always slower than receiving, because reads are random and writes are sequential. So it should be quite normal that mbuffer's buffer doesn't really see a lot of usage. Cheers, Thomas _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
