Sadly none of my test boxes will run 2.6 any more, but I have a theory
about what the problem might be here.
For some background, the pvfs2-server daemon does these steps in order
(among others): initializes BMI (networking), initializes Trove
(storage), and then finally starts processing requests.
In your case, two extra things are going on:
- the trove initialization may take a while, because it has to do a
conversion of the
format for all objects from v. 2.6 to 2.8, especially if it is also
switching to o_direct format at the same time.
- whichever server gets done first is going to immediately contact the
other servers in order to precreate handles for new files (a new feature
in 2.8)
I'm guessing that one server finished the trove conversion before the
others and started its pre-create requests. The other servers can't
answer yet (because they are still busy with trove), but since BMI is
already running the incoming precreate requests just get queued up on
the socket. When the slow server finally does try to service them, the
requests are way out of date and have since been retried by the fast server.
I'm not sure exactly what goes wrong from there, but if that's the
cause, the solution might be relatively simple. If you look in
pvfs2-server.c, you can take the block of code from
"BMI_initialize(...)" to "*server_status_flag |= SERVER_BMI_INIT;" and
try moving that whole block to _after_ the "*server_status_flag |=
SERVER_TROVE_INIT;" line that indicates that trove is done.
-Phil
On 03/30/2010 06:23 PM, Bart Taylor wrote:
I am having some problems upgrading existing file systems to 2.8.
After I finish the upgrade and start the file system, I cannot create
files. Simple commands like dd and cp stall until they timeout and
leave partial dirents like this:
[bat...@client t]$ dd if=/dev/zero of=/mnt/pvfs28/10MB.dat.6 bs=1M
count=10
dd: writing `/mnt/pvfs28/10MB.dat.6': Connection timed out
1+0 records in
0+0 records out
0 bytes (0 B) copied, 180.839 seconds, 0.0 kB/s
[r...@client ~]# ls -alh /mnt/pvfs28/
total 31M
drwxrwxrwt 1 root root 4.0K Mar 30 11:24 .
drwxr-xr-x 4 root root 4.0K Mar 23 13:38 ..
-rw-rw-r-- 1 batayl batayl 10M Mar 30 08:44 10MB.dat.1
-rw-rw-r-- 1 batayl batayl 10M Mar 30 08:44 10MB.dat.2
-rw-rw-r-- 1 batayl batayl 10M Mar 30 08:44 10MB.dat.3
?--------- ? ? ? ? ? 10MB.dat.5
drwxrwxrwx 1 root root 4.0K Mar 29 14:06 lost+found
This happens both on local disk and on network storage, but it only
happens if the upgraded file system starts up the first time using
directio. If it is started with alt-aio as the TroveMethod, everything
works as expected. It also only happens the first time the file system
is started; if I stop the server daemons and restart them, everything
operates as expected. I do have to kill -9 the server deamons, since
they will not exit gracefully.
My test is running on RHEL4 U8 i386 with kernel version 2.6.9-89.ELsmp
with two server nodes and one client. I was unable to recreate the
problem with a single server.
I attached verbose server logs from the time the daemon was started
after the upgrade until the client failed as well as client logs from
mount until the returned error. The cliffs notes are that one of the
servers logs as many unstuff requests as we have client retries
configured. The client fails at the end of the allotted retries. The
other server doesn't log anythign after starting.
Has anyone seen anything similar or know what might be going on?
Bart.
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
