It is interesting to know that the problem is related to link files. I
have found a way to find and delete them all on back-end filesystems,
using a command like the one below.
find /brick/path -size 0b -perm 1000 -exec /bin/rm -v {} \;
I used this technique once in the past, before I realised what link
files were, after I rescued some data from a non-functioning GlusterFS
volume after some of my early experiments went wrong (- definitely not
recommended or supported). I deleted the link files from the backend
filesystems before transferring the files brick by brick.
This might be a solution to our current problem; the link files seem to
be regenerated again when the files are accessed, but I don't know what
other side effects there might be. Even if this procedure did work and
was safe, I suspect the effect would only be temporary. The problem
seems to affect files randomly, including old and newly created files.
The only work-around we have found is to to use NFS instead of GlusterFS
to mount the data. I am trying to stick with GlusterFS for our compute
servers for performance reasons, and we have found that moving problem
files to another location, using an NFS client, then moving them back
again, allows the files to be accessed normally on GlusterFS clients again.
-Dan.
On 18/01/2011 00:05, Diego M. Vadell wrote:
Hi
I had a similar problem: I couldn't move or read or CHMOD a file. Everything
returned Invalid Argument. I found the file in one of the storage nodes (where
one shouldn't be touching anything), with the full contents of the file, the
right permissions, etc. Then I found another file in another storage node, with
0 bytes and with permission "---------T". I deleted the last one and now I can
read/chmod the original file.
Hope this helps,
-- Diego
On Thursday 06 January 2011 07:26:01 Dan Bretherton wrote:
Hello All,
I have seen what appears to be the same problem when a user tried to
change the ownership of some files. The client log file entries were
like this:
[2011-01-05 17:10:02.222293] W [fuse-bridge.c:648:fuse_setattr_cbk]
glusterfs-fuse: 16210806: SETATTR()
/users/rle/INTERIM/INTERIM_2003_VOR850.nc => -1 (Invalid argument)
The error reported on the command line was "permission denied" I am
told, not "invalid argument" as with the file deletion problem.
Unfortunately I didn't get a chance set the log level to TRACE because
the user in question went to an NFS client to change the ownership
before telling me there had been a problem. However this might make it
easier to reproduce the problem, now that we know that it isn't
restricted to file deletion.
-Dan.
-
On 01/06/2011 09:07 AM, Thai. Ngo Bao wrote:
Dear All,
I'd like to confirm that I am having the same problem. I am using
glusterfs 3.1.1.
I did set volume diagnostics.client-log-level to TRACE, please see the
attached file for further information.
Thanks for your support.
~Thai
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Vijay Bellur
Sent: Wednesday, January 05, 2011 1:30 AM
To: Dan Bretherton
Cc: gluster-users
Subject: Re: [Gluster-users] Invalid argument on delete with GlusterFS
3.1.1 client
On Tuesday 04 January 2011 11:21 PM, Dan Bretherton wrote:
Hello Vijay,
There is nothing except routine "accepted client" messages in the
brick log files on the servers, and I can't see anything relevant in
the /var/log/messages files. On the client the only messages relating
to this problem were included in my original mailing list message. Are
there some other log files I should be looking at? I don't know how
to recreate this problem I'm afraid - it's just a case of waiting
until it crops up again. When it does I will try to persuade the
users to leave the files in place while I investigate further. When
the time comes what should I look for?
When it happens again, please set the glusterfs client log level to
TRACE through
#gluster volume set<volname> diagnostics.client-log-level TRACE,
perform the delete operation on such files and send across the client
log file to us.
You can revert back the log level to INFO after you are done with this.
Thanks,
Vijay
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
