Hi Uday, Phil had a few fixes in the recent release that should fix the "invalid termination path" messages that you are seeing. Could you upgrade your setups to the recent 1.3.2 release (or from CVS) and let us know if you are observing the same problems over a prolonged period like you had described. Thanks, PS: You can download 1.3.2 from ftp://ftp.parl.clemson.edu/pub/pvfs2/pvfs2-1.3.2.tar.gz Murali
On Wed, 7 Dec 2005, Uday Shingwekar wrote: > Hi Murali, > The workload has been heavy across 800+ clients. It has mostly > (over 90%) been read with very little write. The current (the server > being running normally) lsof output for the pvfs2-server shows 887 > entries with 805 being IPv4 (we have about that many client machines) > TCP connections. > > Thanks, > uday > > -----Original Message----- > From: Murali Vilayannur [mailto:[EMAIL PROTECTED] > Sent: Wednesday, December 07, 2005 9:15 AM > To: Uday Shingwekar > Cc: [email protected] > Subject: Re: [PVFS2-users] pvfs2-client-core memory usage keeps > increasing, pvfs2-server runs out of open file descriptors etc. > > Hi Uday, > Could you shed some light on what sort of workload(s) you were running > over the 4-5 day period after which this situation was encountered? > Also if you could do an lsof -p `pidof pvfs2-server` that might tell us > where/how all the fd's of the server are being used for.. > Thanks for the report! > Murali > > > Hi, > > We are noticing a few problems when running PVFS2 version > pre1.3.1. > > > > - pvfs2-client-core memory keeps increasing after using the > system for > > 4-5 days. We have seen it go as high as 900M, at which time, we have > to > > kill it and remount the file system. > > - The pvfs2 client log shows the following errors. The server > does not > > have any corresponding error. > > > > [E 23:31:07.186802] *** msgpairarray_completion_fn: msgpair failed, no > > retry: Connection reset by peer > > [E 23:31:07.186856] Error: state machine using an invalid termination > > path. > > > > - We have also seen an error in the pvfs2-server log indicating > the the > > pvfs2-server ran out of max open file descriptors. I was unable to > > capture the log message as the server clobbered the log (instead of > > appending to it) when I restarted the pvfs2 server. The kernel > > parameters for max open files are quite high. > > > > - One pvfs2-server in group of 8 keeps running at 99.9% cpu. > > Incidentally this is the same server which ran out of max open file > > descriptors. > > > > > > Has anyone seen these or similar errors? I would appreciate if anyone > > can share their experience in resolving these errors. > > > > thanks, > > uday > > > > > > > > > > > > _______________________________________________ > > PVFS2-users mailing list > > [email protected] > > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > > > > > > > _______________________________________________ PVFS2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
