Hello Mikko,
>From what we've seen, rename operations are not atomic through the cluster
>because of the hashing algorithm that will move data to a new brick after the
>rename.
To make it atomic, you need to use a feature that allows tweaking the regular
expression that is used to feed the hashing algorithm. With this, you can
decide which part of the path/filename will be used for the distribution hash,
and which part will be ignored. This allowed us to append a ".tmp" extension
that is ignored by the hashing. Consequently, renaming a file to and from
".tmp" ensures it is not moved around the bricks, thereby making it an atomic
operation.
The setting to use for this would be:
cluster.extra-hash-regex: "(.*)\\.tmp"
Cheers,
Laurent Chouinard
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Mikko Vatanen
Sent: 9 mai 2014 06:25
To: [email protected]
Subject: [Gluster-users] Repetitive rename() causes device or resource busy
errors
Hi!
Hello to all, as being noobie on this mailing list. :) I have been trying to
google and figure out this problem without success, so I am asking here for
some help.
As I have understood GlusterFS is should have atomic rename() operation. Am I
going wrong here?
I have some separate testing setups:
- 2 virtual servers, low performance, two bricks on each host
- 4 medium/high performance servers, few bricks on each host, 10Gbit
ethernet between
- Running RHEL 6.5 / kernel 2.6.32-431.11.2.el6.x86_64
Have tried the following versions (with out of the box settings):
- glusterfs 3.5.0 built on Apr 23 2014 12:53:57
- glusterfs 3.4.2 built on Jan 3 2014 12:38:06
Tested volume is simple 2 x replicated volume with default settings "volume
create storage_vol01 rep 2 transport tcp ...the bricks... "
In addition I have also tried disabling write/read caches on server side and
bunch of other parameters, with no change.
Test is run by mounting the glusterfs volume with fuse client on 2 separate
hosts and running test script on both hosts: http://pastebin.com/aAUhaswM
Script sleeps for a short time, if the directory given as first parameter
exists, renames it as the second directory. This is repeated forever.
For example:
host1: python test.py 1/test_directory 2/test_directory
host2: python test.py 2/test_directory 1/test_directory
Both client logs are showing the following messages:
[2014-05-09 07:34:41.227499] I [dht-layout.c:640:dht_layout_normalize]
2-ingest_vol-dht: found anomalies in /test/1/test_directory. holes=1 overlaps=0
[2014-05-09 07:34:41.229184] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 2-ingest_vol-client-2: remote
operation failed: File exists. Path: /test/1/test_directory
[2014-05-09 07:34:41.229240] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 2-ingest_vol-client-3: remote
operation failed: File exists. Path: /test/1/test_directory
After some time has passed other of the scripts raises an exception:
Traceback (most recent call last):
File "test.py", line 28, in <module>
main()
File "test.py", line 25, in main
os.rename(source_filename, destination_filename)
OSError: [Errno 16] Device or resource busy
This error is reproduced always on both setups after decreasing time.sleep()
parameter enough.
Any advice on options / tuning that I could try to resolve this problem?
Thanks in advance :)
-
Mikko Vatanen
Applications Specialist
Digital Preservation Services
CSC - IT Center for Science Ltd.
P.O. Box 405, FI-02101 Espoo, Finland
Mobile: + 358 50 381 2435
http://www.csc.fi/
_______________________________________________
Gluster-users mailing list
[email protected]
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://supercolony.gluster.org/mailman/listinfo/gluster-users
