Hi,
I've installed PVFS (orangeFS 2.8.7) on a small cluster (2 PVFS nodes, 28 compute nodes of 24 cores each, everything connected through infiniband but using an IP stack on top of it, so the protocol for PVFS is TCP), and I witness some strange performance behaviors with IOR (using ROMIO compiled against PVFS, no kernel support): IOR is started on 336 processes (14 nodes), writing 4MB/process in a single shared file using MPI-I/O (4MB transfer size also). It completes 100 iterations. First every time I start an instance of IOR, the first I/O operation is extremely slow. I'm guessing this is because ROMIO has to initialize everything, get the list of PVFS servers, etc. Is there a way to speed this up? Then, I set some delay between each iteration, to better reflect the behavior of an actual scientific application. When I set this delay to 90 sec, I get decreasing performance from one iteration to another: the first write shows 2 GB/s, and after 100 iterations, I get about 300 MB/s. Both the "open", "write" and "close" time are increasing. More precisely the completion time increases linearly with the iteration number. Finally if the delay is short (5 sec for example, or no delay at all), I get 2 GB/s for one operation, then barely 100 MB/s for the next, then 2 GB/s again, and so on. I tried different values for the delay: 90 seconds, 60, 30, 20, 10, 5 and 0. For 90, 60 and 30, I observe the first situation (throughput decreasing). For 20 and 10 the throughput is very variable, but we can notice some periodicity. With 5 seconds the periodicity is obvious: one slow, one fast, one slow, one fast, etc. When there is no delay at all, the periodicity is larger: 4 or 5 fast writes at around 2 GB/s, then 1 slow at 75 MB/s, and so on. Any idea how to explain this behavior? Thanks Matthieu Dorier PhD student at ENS Cachan Brittany and IRISA http://people.irisa.fr/Matthieu.Dorier
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
