Hi Matthieu,
I don't know what would cause the behavior that you are seeing, but I
might be able to help suggest some steps to debug the performance
problem. Could you perhaps share your pvfs configuration file and IOR
configuration file (or command line options)? Also, how much RAM and
what kind of storage do your two servers have? The periodic behavior
makes it sounds like maybe the servers are buffering up data and having
a hard time flushing it to disk, but that seems pretty extreme.
When you specify the file name for IOR, could you add a "pvfs2:" prefix
to the filename and see if that has any impact? It shouldn't matter,
but it will force ROMIO to skip file system detection and use the PVFS2
ROMIO driver directly just as a sanity check.
thanks!
-Phil
On 03/23/2013 12:23 PM, Matthieu Dorier wrote:
Also something else: when I run IOR with 16MB/process (and still a
transfer size of 4MB), the performance drops to 17 MB/s, most BMI jobs
timeout...
Matthieu
------------------------------------------------------------------------
*De: *"Matthieu Dorier" <[email protected]>
*À: *"pvfs2-users" <[email protected]>
*Envoyé: *Samedi 23 Mars 2013 15:31:22
*Objet: *[Pvfs2-users] Strange performance behavior with IOR
Hi,
I've installed PVFS (orangeFS 2.8.7) on a small cluster (2 PVFS
nodes, 28 compute nodes of 24 cores each, everything connected
through infiniband but using an IP stack on top of it, so the
protocol for PVFS is TCP), and I witness some strange performance
behaviors with IOR (using ROMIO compiled against PVFS, no kernel
support):
IOR is started on 336 processes (14 nodes), writing 4MB/process in
a single shared file using MPI-I/O (4MB transfer size also). It
completes 100 iterations.
First every time I start an instance of IOR, the first I/O
operation is extremely slow. I'm guessing this is because ROMIO
has to initialize everything, get the list of PVFS servers, etc.
Is there a way to speed this up?
Then, I set some delay between each iteration, to better reflect
the behavior of an actual scientific application. When I set this
delay to 90 sec, I get decreasing performance from one iteration
to another: the first write shows 2 GB/s, and after 100
iterations, I get about 300 MB/s. Both the "open", "write" and
"close" time are increasing. More precisely the completion time
increases linearly with the iteration number.
Finally if the delay is short (5 sec for example, or no delay at
all), I get 2 GB/s for one operation, then barely 100 MB/s for the
next, then 2 GB/s again, and so on.
I tried different values for the delay: 90 seconds, 60, 30, 20,
10, 5 and 0. For 90, 60 and 30, I observe the first situation
(throughput decreasing). For 20 and 10 the throughput is very
variable, but we can notice some periodicity. With 5 seconds the
periodicity is obvious: one slow, one fast, one slow, one fast, etc.
When there is no delay at all, the periodicity is larger: 4 or 5
fast writes at around 2 GB/s, then 1 slow at 75 MB/s, and so on.
Any idea how to explain this behavior?
Thanks
Matthieu Dorier
PhD student at ENS Cachan Brittany and IRISA
http://people.irisa.fr/Matthieu.Dorier
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
