Rob,
I did the following:
1. Check that the condition still exists. Try to read a 170MB file from
pvfs2:
# time cp /data/P1/some-file /dev/null
^C -- killed after 4 minutes
2. Remount /data/P1 and run the test again
# umount /data/P1
# mount /data/P1
# time cp /data/P1/some-file /dev/null
^C -- killed after 3m23s
-- the problem still exists.
3. Kill pvfs2-client-core
# ps auxww | grep pvfs2
....
# kill 45673
# time cp /data/P1/some-file /dev/null
real 0m38.448s
user 0m0.023s
sys 0m0.652s
-- faster than before, but still a lot slower than pvfs2-cp:
4. Run pvfs2-cp on the same file:
# pvfs2-cp -t /data/P1/some-file /dev/null
Wrote 169869851 bytes in 1.708348 seconds. 94.828753 MB/seconds
Do you want me to do the third test on your list and remove the kernel
module?
Hope it helps,
--andrew
On Feb 22, 2006, at 11:31 AM, Robert Latham wrote:
On Wed, Feb 15, 2006 at 05:38:56PM -0500, Andrew Pochinsky wrote:
Here's pvfs2-statfs output. There is, of course, a chance of a
hardware
problem, but the fact that pvfs2-cp works ten times faster than cp
suggests otherwise, in my opinion. I have rebooted both servers and
the
client, and after about 15 hours I again see Badness... messages in
dmesg and the system log. There is nothing new in the pvfs2 client log
file.
Ok, good. sorry, i misread the part where you explained everything is
running just fine through the system interface. That's good news.
And things run well for a while, meaning the servers, clients, and
kernel interface things are all configured correctly. Then everything
goes badly after 15 hours.
When PVFS2 gets into this bad state, is there any way to recover? You
could try 1: unmounting and remounting the file system; 2: killing
pvfs2-client-core (relying on pvfs2-client to restart it); 3: removing
the kernel module (after unmounting and stopping pvfs2-client) and
then reloading it.
All that will do is narrow down where the wierd state lies. If you do
all three things above and pvfs2 still performs slowly through the
VFS, then I'll have to think some more about what's going on.
Do other pvfs2 clients have this performance problem or is it isolated
to one client?
Can you tell me more about the workload you are using? Is most of
the activity coming from the login nodes or the compute nodes?
Thanks
==rob
--
Rob Latham
Mathematics and Computer Science Division A215 0178 EA2D B059 8CDF
Argonne National Labs, IL USA B29D F333 664A 4280 315B
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
