Oh, Ok. The metadata object must be missing then. If you do a normal
pvfs2-ls (no -al options) does the file show up in the listing without
errors?
Maybe there is an initial problem (the metadata object missing for some
reason), followed by a secondary problem that a getattr on that object
is returning an empty attribute structure rather than indicating that
there is an error.
I'm going to setup a small 2.6 setup here and try destroying a metafile
(using remove-object, probably) to see if I can recreate the symptoms
that you are seeing.
-Phil
On 10/08/2010 04:05 PM, Bart Taylor wrote:
Hey Phil,
The pvfs2-stat call never actually made it past sys_lookup, so there
was no ref.handle to print out. Here is the output:
#pvfs2-stat /mnt/pvfs2/file1.txt
PVFS_sys_lookup: No such file or directory (error class: 0)
Error stating [/mnt/pvfs2/file1.txt]
Bart.
On Thu, Oct 7, 2010 at 3:13 PM, Phil Carns <[email protected]
<mailto:[email protected]>> wrote:
Hi Bart,
Can you run pvfs2-stat on one of the files, and also send along
the fs.conf file? pvfs2-stat might be helpful because it shows
the metadata handle value. We can compare that value to the
handle ranges in the conf file to narrow down whether it is
hitting a metadata object that has just been corrupted somehow, or
whether it really is hitting a datafile handle.
If pvfs2-stat fails to show any output, then maybe you can modify
pvfs2-stat.c to print the value of ref.handle right before the
sys_getattr() call.
thanks,
-Phil
On 10/07/2010 01:08 PM, Bart Taylor wrote:
Hey guys,
We are having an increasing number of files that cannot be
removed on our 2.6 file systems. When we run the pvfs2-lsplus
tool, the output on these files looks like this:
[E 15:14:05.798568] Invalid type 2 in readdirplus
---------- 1 root root 0 1969-12-31 18:00 File1
[E 15:14:17.712553] Invalid type 2 in readdirplus
---------- 1 root root 0 1969-12-31 18:00 File2
[E 15:14:24.799221] Invalid type 2 in readdirplus
[E 15:14:24.799257] Invalid type 2 in readdirplus
[E 15:14:24.799269] Invalid type 2 in readdirplus
---------- 1 root root 0 1969-12-31 18:00
File3.txt
---------- 1 root root 0 1969-12-31 18:00
File5.txt
---------- 1 root root 0 1969-12-31 18:00
File6.txt
The "Invalid type 2" message indicates that readdirplus is
returning datafile attributes mixed in with the directory
entries. That might explain why all of the attributes look like
default values, but I am not sure why those files are having
problems in the first place.
This gets noticed when someone tries to update, create, append,
etc. the file. Most operations seem to return "No such file or
directory" when trying to access those files. A standard /bin/ls
will return normally, but long listings fail. pvfs2-ls,
pvfs2-viewdist and pvfs2-validate return getattr failures.
pvfs2-rm returns output like this:
[E 09:42:30.669413] Error: failed removing one or more datafiles
associated with the meta handle 238502937
[E 09:42:30.669599] WARNING: PVFS_sys_remove() encountered an
error which may lead
to inconsistent state: No such file or directory
[E 09:42:30.669614] WARNING: PVFS2 fsck (if available) may be needed.
Error: An error occurred while removing /mnt/pvfs2/file1.txt
PVFS_sys_remove: No such file or directory (error class: 0)
Removing these files is a manual process. These are the steps we
follow:
- Track down the file(s) that are causing the problems
- pvfs2-stat on the directory where the file resides
- Grab the FSID and handle from the output
- pvfs2-remove-object using the file name, directory handle, and FSID
As more of these files start appearing, this process is becoming
slow and painful. It would be great if we could sort out why
these files are showing up like they are, but right now I think a
utility that could efficiently remove these files without the
legwork would be really helpful. Any idea what might work based
on what we are seeing?
I am not sure if the problem also exists in 2.8, but it may be
related to this issue mailed in by Jim in September:
http://www.beowulf-underground.org/pipermail/pvfs2-users/2010-September/003186.html
We are experiencing this issue as well.
Thanks,
Bart.
_______________________________________________
Pvfs2-developers mailing list
[email protected]
<mailto:[email protected]>
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
<mailto:[email protected]>
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
