Hello,
I have performed more precise performance measures with the MPI-IO
interface. The bandwidth when writing data gets degraded when the size
of the data is above some threshold.
I measured this bandwidth with different settings: TroveSyncData is
disabled, TroveMethod is set to alt-io and the IO scheduler is cfq. The
versions are: OrangeFS 2.8.5, MPICH 1.4.1p1, IOR 2.10.3, kernel
2.6.32-5-amd64 (debian squeeze), ext3 file system and Gigabit Ethernet.
There is a significant variability of the write bandwidth for files of
size 200 MB, whereas it is stable when the size is 10 MB. However, when
I use pvfs2-cp with a file of size 2GB, the bandwidth is close to 100 MB/s:
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev
write 92.06 88.43 91.61 1.06
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 94.67 35.72 68.07 26.17
dd if=/dev/zero of=/tmp/test bs=1000000 count=2000
pvfs2-cp -t /tmp/test /mnt/pvfs2/test
Wrote 2000000000 bytes in 19.527506 seconds. 97.674973 MB/seconds
I provided more measures where the defaults are alternatively changed.
The performance variability with large file with MPI-IO seems related to
the defaults settings for TroveSyncData and TroveMethod. When changing
one of those, the performance is even worse, but it is stable.
Additionally, the IO scheduler does not seems to have a significant impact.
Does anyone has idea about this issue (low and unstable bandwidth when
writing large file with MPI-IO)?
Regards,
Louis-Claude Canon
<Defaults>
UnexpectedRequests 50
EventLogging none
EnableTracing no
LogStamp datetime
BMIModules bmi_tcp
FlowModules flowproto_multiqueue
PerfUpdateInterval 1000
ServerJobBMITimeoutSecs 30
ServerJobFlowTimeoutSecs 30
ClientJobBMITimeoutSecs 300
ClientJobFlowTimeoutSecs 300
ClientRetryLimit 5
ClientRetryDelayMilliSecs 2000
PrecreateBatchSize 0,32,512,32,32,32,0
PrecreateLowThreshold 0,16,256,16,16,16,0
DataStorageSpace /tmp/pvfs2-storage
MetadataStorageSpace /tmp/pvfs2-storage
LogFile /tmp/pvfs2-server.log
</Defaults>
<Aliases>
Alias paradent-19.rennes.grid5000.fr
tcp://paradent-19.rennes.grid5000.fr:3334
Alias paradent-2.rennes.grid5000.fr
tcp://paradent-2.rennes.grid5000.fr:3334
</Aliases>
<Filesystem>
Name pvfs2-fs
ID 1396726978
RootHandle 1048576
FileStuffing yes
<MetaHandleRanges>
Range paradent-19.rennes.grid5000.fr 3-2305843009213693953
Range paradent-2.rennes.grid5000.fr
2305843009213693954-4611686018427387904
</MetaHandleRanges>
<DataHandleRanges>
Range paradent-19.rennes.grid5000.fr
4611686018427387905-6917529027641081855
Range paradent-2.rennes.grid5000.fr
6917529027641081856-9223372036854775806
</DataHandleRanges>
<StorageHints>
TroveSyncMeta yes
TroveSyncData no
TroveMethod alt-aio
</StorageHints>
</Filesystem>
Defaults: TroveSyncData no and TroveMethod alt-io and cfq
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev
write 92.06 88.43 91.61 1.06
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 94.67 35.72 68.07 26.17
dd if=/dev/zero of=/tmp/test bs=1000000 count=2000
pvfs2-cp -t /tmp/test /mnt/pvfs2/test
Wrote 2000000000 bytes in 19.527506 seconds. 97.674973 MB/seconds
TroveSyncData yes
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 45.29 39.86 42.30 1.58
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 45.53 44.09 44.88 0.48
pvfs2-cp -t /tmp/test /mnt/pvfs2/test
Wrote 2000000000 bytes in 44.841938 seconds. 42.534929 MB/seconds
TroveMethod directio
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 68.21 43.72 58.06 6.96
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 51.36 45.25 48.11 1.62
pvfs2-cp -t /tmp/test /mnt/pvfs2/test
Wrote 2000000000 bytes in 42.624171 seconds. 44.748052 MB/seconds
deadline
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 94.04 87.00 93.07 2.03
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 94.28 44.68 57.26 18.57
pvfs2-cp -t /tmp/test /mnt/pvfs2/test
Wrote 2000000000 bytes in 21.759281 seconds. 87.656785 MB/seconds
anticipatory
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 93.03 87.79 92.38 1.54
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 93.96 46.93 57.09 18.02
pvfs2-cp -t /tmp/test /mnt/pvfs2/test
Wrote 2000000000 bytes in 19.415972 seconds. 98.236062 MB/seconds
noop
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 90.97 78.92 87.59 3.34
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 94.19 40.79 70.24 24.15
pvfs2-cp -t /tmp/test /mnt/pvfs2/test
Wrote 2000000000 bytes in 18.625568 seconds. 102.404858 MB/seconds
--- Complete ---
TroveSyncData yes
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 45.29 39.86 42.30 1.58
read 92.11 88.27 91.28 1.14
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 45.53 44.09 44.88 0.48
read 94.66 94.40 94.53 0.08
TroveMethod directio
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 68.21 43.72 58.06 6.96
read 71.70 44.06 62.70 10.54
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 51.36 45.25 48.11 1.62
read 53.16 46.85 51.31 1.92
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 92.06 88.43 91.61 1.06
read 90.39 84.36 89.63 1.76
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 94.67 35.72 68.07 26.17
read 94.66 94.53 94.57 0.04
deadline
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 94.04 87.00 93.07 2.03
read 92.24 90.91 91.96 0.37
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 94.28 44.68 57.26 18.57
read 94.17 92.13 93.87 0.58
anticipatory
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 93.03 87.79 92.38 1.54
read 91.31 90.35 91.12 0.27
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 93.96 46.93 57.09 18.02
read 95.04 94.06 94.70 0.38
noop
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 90.97 78.92 87.59 3.34
read 91.21 90.14 90.91 0.32
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 94.19 40.79 70.24 24.15
read 94.16 94.01 94.09 0.05
TroveSyncData yes and deadline
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 46.36 40.19 44.56 2.02
read 88.91 84.93 87.04 1.39
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 46.33 45.48 45.86 0.29
read 94.30 94.10 94.21 0.06
TroveSyncData yes and anticipatory
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 45.53 39.79 42.10 1.86
read 91.36 89.32 91.02 0.58
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 46.50 45.08 45.80 0.36
read 94.25 94.00 94.14 0.08
TroveSyncData yes and noop
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000
write 46.63 40.31 43.49 2.41
read 91.37 90.53 91.10 0.22
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000
write 46.30 45.56 46.01 0.25
read 94.40 94.06 94.33 0.09
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
