On 3/26/2012 6:44 AM, Conrad Wood wrote:
On Sat, 2012-03-24 at 11:45 -0400, Doug Hughes wrote:
On 3/24/2012 6:42 AM, Conrad Wood wrote:
Mostly I wonder if it is atall possible to get such speeds over QDR. Are
you in a position where you could perhaps run a "cp" followed by "sync"
on an infiniband attached storage system?
indeed it is. Many of the larger integrated vendors are moving to QDR
based SANs for that reason. speeds in excess of 6GBytes/sec and up on
sequential loads are not uncommon. (with the appropriate back-end storage)
The sequential speed of Infiniband QDR is fantastic, but our clients (I
guess rightly) complain about the speed of 'cp'. The same 'cp' on the
local array runs lightning fast.
That's what I don't understand: Why is the same sequence of commands so
extraordinarily different on local vs remote storage?
I don't think I have mentioned the sequence of commands in details, it
is:
MOUNTPOINT=/tmp/[randomstuff]
mbytes=4G
1. mkdir ${MOUNTPOINT}
2. mkfs -t ext4 -F ${MOUNTPOINT}
3. dd if=/dev/zero of=${MOUNTPOINT}/randomfile.bin bs=1M \
count=$mbytes oflag=direct
4. sync
5. STARTTIME=time()
6. cp ${MOUNTPOINT}/randomfile.bin ${MOUNTPOINT}/randomfile2.bin
7. sync
8. umount ${MOUNTPOINT}
9. ENDTIME=time()
and now the speed is calculated by simply ($ENDTIME-$STARTTIME)/$mbytes
This sequence is run 5x in parallel on different volumes and the sum of
all tests is the total throughput.
The total sum is>1000 MByte/s on the storage server locally, but only
~150MByte/s over SRP.
There are many things possibly wrong.. Here are some things to try.,
1) use strace to compare your cp to your dd and see what they are doing.
(cp does do some weird stuff on Linux). Without that, how can you be
sure the problem isn't on the file that you are opening for read or
other problem? Use the flags to include system call timing measurements
(no man page near, but I think it's -T)
2) write a simple perl (or other) program that writes a 1MB buffer to
your storage 200 times into a file and measure that. (don't open a local
file for this, it will taint your results)
3) try iser instead of SRP.
4) turn on IP over IB and see how NFS works over there.
There are many further diagnostics to be done, but I assure you this is
not a Linux IB bandwidth or latency issue. We use IB extensively (but
not SRP at this time)
_______________________________________________
Tech mailing list
[email protected]
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
