Also something else: when I run IOR with 16MB/process (and still a transfer size of 4MB), the performance drops to 17 MB/s, most BMI jobs timeout...
Matthieu ----- Mail original ----- > De: "Matthieu Dorier" <[email protected]> > À: "pvfs2-users" <[email protected]> > Envoyé: Samedi 23 Mars 2013 15:31:22 > Objet: [Pvfs2-users] Strange performance behavior with IOR > Hi, > I've installed PVFS (orangeFS 2.8.7) on a small cluster (2 PVFS > nodes, 28 compute nodes of 24 cores each, everything connected > through infiniband but using an IP stack on top of it, so the > protocol for PVFS is TCP), and I witness some strange performance > behaviors with IOR (using ROMIO compiled against PVFS, no kernel > support): > IOR is started on 336 processes (14 nodes), writing 4MB/process in a > single shared file using MPI-I/O (4MB transfer size also). It > completes 100 iterations. > First every time I start an instance of IOR, the first I/O operation > is extremely slow. I'm guessing this is because ROMIO has to > initialize everything, get the list of PVFS servers, etc. Is there a > way to speed this up? > Then, I set some delay between each iteration, to better reflect the > behavior of an actual scientific application. When I set this delay > to 90 sec, I get decreasing performance from one iteration to > another: the first write shows 2 GB/s, and after 100 iterations, I > get about 300 MB/s. Both the "open", "write" and "close" time are > increasing. More precisely the completion time increases linearly > with the iteration number. > Finally if the delay is short (5 sec for example, or no delay at > all), I get 2 GB/s for one operation, then barely 100 MB/s for the > next, then 2 GB/s again, and so on. > I tried different values for the delay: 90 seconds, 60, 30, 20, 10, 5 > and 0. For 90, 60 and 30, I observe the first situation (throughput > decreasing). For 20 and 10 the throughput is very variable, but we > can notice some periodicity. With 5 seconds the periodicity is > obvious: one slow, one fast, one slow, one fast, etc. > When there is no delay at all, the periodicity is larger: 4 or 5 fast > writes at around 2 GB/s, then 1 slow at 75 MB/s, and so on. > Any idea how to explain this behavior? > Thanks > Matthieu Dorier > PhD student at ENS Cachan Brittany and IRISA > http://people.irisa.fr/Matthieu.Dorier > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
