On 10/11/2010 11:11 AM, Phil Carns wrote:
I'm not sure how you are getting there yet, but I am able to recreate
this state on my laptop with a single server and the pvfs2 admin
utilities.
The following shows the creation of 4 files. I then get the metadata
handle for one of them (via pvfs2-stat) and the fsid from the conf
file. Then pvfs2-remove-object is used to manually delete that
particular object, leaving the directory entry in place. pvfs2-lsplus
at that point starts showing the "Invalid type..." error and
pvfs2-stat gets no such file or directory.
-------------------------------------------------------------
[pca...@pcarns-laptop admin]$ ./pvfs2-cp /tmp/16mb.dat /mnt/pvfs2/a.dat
[pca...@pcarns-laptop admin]$ ./pvfs2-cp /tmp/16mb.dat /mnt/pvfs2/b.dat
[pca...@pcarns-laptop admin]$ ./pvfs2-cp /tmp/16mb.dat /mnt/pvfs2/d.dat
[pca...@pcarns-laptop admin]$ ./pvfs2-cp /tmp/16mb.dat /mnt/pvfs2/c.dat
[pca...@pcarns-laptop admin]$ ./pvfs2-lsplus -alh /mnt/pvfs2
drwxrwxrwx 1 pcarns pcarns 4.0K 2010-10-11 10:53 .
drwxrwxrwx 1 pcarns pcarns 4.0K 2010-10-11 10:53 .. (faked)
-rw-r--r-- 1 pcarns pcarns 16.0M 2010-10-11 10:53 a.dat
-rw-r--r-- 1 pcarns pcarns 16.0M 2010-10-11 10:53 b.dat
-rw-r--r-- 1 pcarns pcarns 16.0M 2010-10-11 10:53 c.dat
-rw-r--r-- 1 pcarns pcarns 16.0M 2010-10-11 10:53 d.dat
drwxrwxrwx 1 pcarns pcarns 4.0K 2010-10-11 10:52 lost+found
[pca...@pcarns-laptop admin]$ ./pvfs2-stat /mnt/pvfs2/b.dat
-------------------------------------------------------
File Name : /mnt/pvfs2/b.dat
Relative Name : /b.dat
fs ID : 513116601
Handle : 1048574
Mask : 704000177
Permissions : 644
Type : Regular File
Size : 16777216
Owner : 1000 (pcarns)
Group : 1000 (pcarns)
atime : 1286808818 (Mon Oct 11 10:53:38 2010)
mtime : 1286808818 (Mon Oct 11 10:53:38 2010)
ctime : 1286808818 (Mon Oct 11 10:53:38 2010)
datafiles : 1
flags : none
[pca...@pcarns-laptop server]$ cat simple.conf |grep ID
ID 513116601
[pca...@pcarns-laptop admin]$ ./pvfs2-remove-object -f 513116601 -o
1048574
Attempting to remove object 1048574,513116601
[pca...@pcarns-laptop admin]$ ./pvfs2-lsplus /mnt/pvfs2
[E 10:54:54.724633] Invalid type 2 in readdirplus
a.dat
b.dat
c.dat
d.dat
lost+found
[pca...@pcarns-laptop admin]$ ./pvfs2-stat /mnt/pvfs2/b.dat
PVFS_sys_lookup: No such file or directory (error class: 0)
Error stating [/mnt/pvfs2/b.dat]
----------------------------------------------------------------------
From there I can do a pvfs2-rm, but it doesn't really clean it up
properly. I think you might get different errors if you tried to do
this via the kernel module, FWIW:
[pca...@pcarns-laptop admin]$ ./pvfs2-rm /mnt/pvfs2/b.dat
Error: An error occurred while removing /mnt/pvfs2/b.dat
PVFS_sys_remove: No such file or directory (error class: 0)
[pca...@pcarns-laptop admin]$ ./pvfs2-ls -alh /mnt/pvfs2/
drwxrwxrwx 1 pcarns pcarns 4.0K 2010-10-11 11:00 .
drwxrwxrwx 1 pcarns pcarns 4.0K 2010-10-11 11:00 .. (faked)
-rw-r--r-- 1 pcarns pcarns 16.0M 2010-10-11 10:53 a.dat
Failed to get attributes on handle 1048574,513116601
Getattr failure: No such file or directory (error class: 0)
-rw-r--r-- 1 pcarns pcarns 16.0M 2010-10-11 10:53 c.dat
-rw-r--r-- 1 pcarns pcarns 16.0M 2010-10-11 10:53 d.dat
drwxrwxrwx 1 pcarns pcarns 4.0K 2010-10-11 10:52 lost+found
Whoops! I just realized that that last command was pvfs2-ls instead of
pvfs2-lsplus. Here is what I really get:
[pca...@pcarns-laptop admin]$ ./pvfs2-lsplus -alh /mnt/pvfs2/
[E 11:16:57.443365] Invalid type 2 in readdirplus
drwxrwxrwx 1 pcarns pcarns 4.0K 2010-10-11 11:00 .
drwxrwxrwx 1 pcarns pcarns 4.0K 2010-10-11 11:00 .. (faked)
-rw-r--r-- 1 pcarns pcarns 16.0M 2010-10-11 10:53 a.dat
---------- 1 root root 0 1969-12-31 19:00 b.dat
-rw-r--r-- 1 pcarns pcarns 16.0M 2010-10-11 10:53 c.dat
-rw-r--r-- 1 pcarns pcarns 16.0M 2010-10-11 10:53 d.dat
drwxrwxrwx 1 pcarns pcarns 4.0K 2010-10-11 10:52 lost+found
Now it looks more like your behavior. After attempting to remove the
file I still get "Invalid type..." but now the object shows up with 0'd
out attributes and a 1969 creation date.
-Phil
Maybe that example above can make it easier to reproduce in a
development environment? We can tackle the problem later about how
the metadata object disappeared in the first place, but for starters
it would probably be good to at least fix these two things:
- how to make pvfs2-rm safely remove what it can (even if via a
"force" option)
- how to get pvfs2-lsplus (and probably other utilities and/or kernel
module as well) to report a sane error message instead of the "Invalid
object" message
....
FYI, the 2.6 tree is pretty cranky to build on my machine at this
point. I attached a patch that sets more BDB flags that are necessary
for recent BDB releases, in case anyone else needs to build 2.6 on a
modern box. I also found that I had to use the "DBCacheType mmap"
option in the server config file. I can't get the kernel module
working, so I stuck to command line utilities.
-Phil
On 10/11/2010 09:23 AM, Phil Carns wrote:
Oh, Ok. The metadata object must be missing then. If you do a
normal pvfs2-ls (no -al options) does the file show up in the listing
without errors?
Maybe there is an initial problem (the metadata object missing for
some reason), followed by a secondary problem that a getattr on that
object is returning an empty attribute structure rather than
indicating that there is an error.
I'm going to setup a small 2.6 setup here and try destroying a
metafile (using remove-object, probably) to see if I can recreate the
symptoms that you are seeing.
-Phil
On 10/08/2010 04:05 PM, Bart Taylor wrote:
Hey Phil,
The pvfs2-stat call never actually made it past sys_lookup, so there
was no ref.handle to print out. Here is the output:
#pvfs2-stat /mnt/pvfs2/file1.txt
PVFS_sys_lookup: No such file or directory (error class: 0)
Error stating [/mnt/pvfs2/file1.txt]
Bart.
On Thu, Oct 7, 2010 at 3:13 PM, Phil Carns <[email protected]
<mailto:[email protected]>> wrote:
Hi Bart,
Can you run pvfs2-stat on one of the files, and also send along
the fs.conf file? pvfs2-stat might be helpful because it shows
the metadata handle value. We can compare that value to the
handle ranges in the conf file to narrow down whether it is
hitting a metadata object that has just been corrupted somehow,
or whether it really is hitting a datafile handle.
If pvfs2-stat fails to show any output, then maybe you can
modify pvfs2-stat.c to print the value of ref.handle right
before the sys_getattr() call.
thanks,
-Phil
On 10/07/2010 01:08 PM, Bart Taylor wrote:
Hey guys,
We are having an increasing number of files that cannot be
removed on our 2.6 file systems. When we run the pvfs2-lsplus
tool, the output on these files looks like this:
[E 15:14:05.798568] Invalid type 2 in readdirplus
---------- 1 root root 0 1969-12-31 18:00
File1
[E 15:14:17.712553] Invalid type 2 in readdirplus
---------- 1 root root 0 1969-12-31 18:00
File2
[E 15:14:24.799221] Invalid type 2 in readdirplus
[E 15:14:24.799257] Invalid type 2 in readdirplus
[E 15:14:24.799269] Invalid type 2 in readdirplus
---------- 1 root root 0 1969-12-31 18:00
File3.txt
---------- 1 root root 0 1969-12-31 18:00
File5.txt
---------- 1 root root 0 1969-12-31 18:00
File6.txt
The "Invalid type 2" message indicates that readdirplus is
returning datafile attributes mixed in with the directory
entries. That might explain why all of the attributes look like
default values, but I am not sure why those files are having
problems in the first place.
This gets noticed when someone tries to update, create, append,
etc. the file. Most operations seem to return "No such file or
directory" when trying to access those files. A standard
/bin/ls will return normally, but long listings fail. pvfs2-ls,
pvfs2-viewdist and pvfs2-validate return getattr failures.
pvfs2-rm returns output like this:
[E 09:42:30.669413] Error: failed removing one or more
datafiles associated with the meta handle 238502937
[E 09:42:30.669599] WARNING: PVFS_sys_remove() encountered an
error which may lead
to inconsistent state: No such file or directory
[E 09:42:30.669614] WARNING: PVFS2 fsck (if available) may be
needed.
Error: An error occurred while removing /mnt/pvfs2/file1.txt
PVFS_sys_remove: No such file or directory (error class: 0)
Removing these files is a manual process. These are the steps
we follow:
- Track down the file(s) that are causing the problems
- pvfs2-stat on the directory where the file resides
- Grab the FSID and handle from the output
- pvfs2-remove-object using the file name, directory handle,
and FSID
As more of these files start appearing, this process is
becoming slow and painful. It would be great if we could sort
out why these files are showing up like they are, but right now
I think a utility that could efficiently remove these files
without the legwork would be really helpful. Any idea what
might work based on what we are seeing?
I am not sure if the problem also exists in 2.8, but it may be
related to this issue mailed in by Jim in September:
http://www.beowulf-underground.org/pipermail/pvfs2-users/2010-September/003186.html
We are experiencing this issue as well.
Thanks,
Bart.
_______________________________________________
Pvfs2-developers mailing list
[email protected]
<mailto:[email protected]>
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
<mailto:[email protected]>
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
