Hi, The I/O performance of CNL (as measured with IOR) seems quite different for a shared file, compared to the same with separated files.
Here are some numbers on a smaller file system on XT system at ORNL. All files are striped to 72OSTs. I deliberately use a block size 8512m. 1. sample tests with separate files # aprun -n 32 -N 1 ~/benchmarks/IOR-2.9.1/src/C/IOR -a MPIIO -b 8512m -t 64m -d 1 -i 2 -w -r -g -F -o iortes Max Write: 9978.18 MiB/sec (10462.88 MB/sec) Max Read: 5612.78 MiB/sec (5885.43 MB/sec) 2. sample share file performance # aprun -n 32 -N 1 ~/benchmarks/IOR-2.9.1/src/C/IOR -a MPIIO -b 8512m -t 64m -d 1 -i 2 -w -r -g -o iortes Max Write: 6817.31 MiB/sec (7148.47 MB/sec) Max Read: 5591.98 MiB/sec (5863.62 MB/sec) In addition, using my experimental MPI-IO library, I noticed that enabling direct I/O can have various effects for I/O on CNL. 3. sample seprate files with direct I/O export MPIO_DIRECT_WRITE=true; export MPIO_DIRECT_READ=true; aprun -n 32 -N 1 ~/benchmarks/IOR-2.10.1/src/C/IOR -a MPIIO -b 8512m -t 64m -d 1 -i 2 -w -r -g -F -k -o lustre:iortest Max Write: 9353.66 MiB/sec (9808.03 MB/sec) Max Read: 8269.28 MiB/sec (8670.97 MB/sec) 4. sample share file performance with direct IO # export MPIO_DIRECT_WRITE=true; export MPIO_DIRECT_READ=true; aprun -n 32 -N 1 ~/benchmarks/IOR-2.10.1/src/C/IOR -a MPIIO -b 8512m -t 64m -d 1 -i 2 -w -r -g -k -o lustre:iortes Max Write: 9484.11 MiB/sec (9944.81 MB/sec) Max Read: 7929.63 MiB/sec (8314.81 MB/sec) It seems direct I/O helps quite a bit on the performance of parallel reads, but not on writes. The shared file mode appears to benefit more from direct write. While it is understandable that the client cache can play a big role here, I am not sure how it could help the share-file mode much better. Anybody can help with some explanations on the comparison between reads and writes and the same for shared-file and separated-files? Also let me know if I am not clear in my descriptions. -- Weikuan Yu <+> 1-865-574-7990 http://ft.ornl.gov/~wyu/ P.S.: What shown are the good numbers from several runs. So you may consider them as consistent results. _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
