Hi Bart,
Can you run pvfs2-stat on one of the files, and also send along the
fs.conf file? pvfs2-stat might be helpful because it shows the metadata
handle value. We can compare that value to the handle ranges in the
conf file to narrow down whether it is hitting a metadata object that
has just been corrupted somehow, or whether it really is hitting a
datafile handle.
If pvfs2-stat fails to show any output, then maybe you can modify
pvfs2-stat.c to print the value of ref.handle right before the
sys_getattr() call.
thanks,
-Phil
On 10/07/2010 01:08 PM, Bart Taylor wrote:
Hey guys,
We are having an increasing number of files that cannot be removed on
our 2.6 file systems. When we run the pvfs2-lsplus tool, the output on
these files looks like this:
[E 15:14:05.798568] Invalid type 2 in readdirplus
---------- 1 root root 0 1969-12-31 18:00 File1
[E 15:14:17.712553] Invalid type 2 in readdirplus
---------- 1 root root 0 1969-12-31 18:00 File2
[E 15:14:24.799221] Invalid type 2 in readdirplus
[E 15:14:24.799257] Invalid type 2 in readdirplus
[E 15:14:24.799269] Invalid type 2 in readdirplus
---------- 1 root root 0 1969-12-31 18:00 File3.txt
---------- 1 root root 0 1969-12-31 18:00 File5.txt
---------- 1 root root 0 1969-12-31 18:00 File6.txt
The "Invalid type 2" message indicates that readdirplus is returning
datafile attributes mixed in with the directory entries. That might
explain why all of the attributes look like default values, but I am
not sure why those files are having problems in the first place.
This gets noticed when someone tries to update, create, append, etc.
the file. Most operations seem to return "No such file or directory"
when trying to access those files. A standard /bin/ls will return
normally, but long listings fail. pvfs2-ls, pvfs2-viewdist and
pvfs2-validate return getattr failures. pvfs2-rm returns output like this:
[E 09:42:30.669413] Error: failed removing one or more datafiles
associated with the meta handle 238502937
[E 09:42:30.669599] WARNING: PVFS_sys_remove() encountered an error
which may lead
to inconsistent state: No such file or directory
[E 09:42:30.669614] WARNING: PVFS2 fsck (if available) may be needed.
Error: An error occurred while removing /mnt/pvfs2/file1.txt
PVFS_sys_remove: No such file or directory (error class: 0)
Removing these files is a manual process. These are the steps we follow:
- Track down the file(s) that are causing the problems
- pvfs2-stat on the directory where the file resides
- Grab the FSID and handle from the output
- pvfs2-remove-object using the file name, directory handle, and FSID
As more of these files start appearing, this process is becoming slow
and painful. It would be great if we could sort out why these files
are showing up like they are, but right now I think a utility that
could efficiently remove these files without the legwork would be
really helpful. Any idea what might work based on what we are seeing?
I am not sure if the problem also exists in 2.8, but it may be related
to this issue mailed in by Jim in September:
http://www.beowulf-underground.org/pipermail/pvfs2-users/2010-September/003186.html
We are experiencing this issue as well.
Thanks,
Bart.
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
