Thanks for the reply Pranith.
Unfortunately I am not able to see the bug in the system. Can you give
me more detail to a workaround?
Much appreciated.
Regards,
John
On 02/24/2011 10:11 PM, Pranith Kumar. Karampuri wrote:
Rebalance and EPERM, looks like you are hitting
http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2369
Pranith.
----- Original Message -----
From: "John Lao"<[email protected]>
To: [email protected]
Sent: Thursday, February 24, 2011 11:55:52 PM
Subject: [Gluster-users] Experiencing errors after adding new nodes
Hi,
I had a 2 node distributed cluster running on 3.1.1 and I added 2 more nodes.
I then ran a rebalance on the cluster.
Now I am getting permission denied errors and I see the following in the client
logs:
[2011-02-24 09:59:10.210166] I [dht-common.c:369:dht_revalidate_cbk]
loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
[2011-02-24 09:59:11.851656] I [dht-common.c:369:dht_revalidate_cbk]
loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
[root@qe-loader1 glusterfs]# tail -100 mnt-qe-filer01.log
[2011-02-24 09:32:50.844211] I [dht-common.c:415:dht_revalidate_cbk]
loader-dht: mismatching layouts for /
[2011-02-24 09:32:50.845312] I [dht-layout.c:588:dht_layout_normalize]
loader-dht: found anomalies in /. holes=1 overlaps=0
[2011-02-24 09:32:50.846417] I [dht-layout.c:689:dht_layout_dir_mismatch]
loader-dht: subvol: loader-client-0; inode layout - 0 - 2147483646; disk layout
- 2147483646 - 3221225468
[2011-02-24 09:32:50.846434] I [dht-common.c:415:dht_revalidate_cbk]
loader-dht: mismatching layouts for /
[2011-02-24 09:32:50.846561] I [dht-layout.c:689:dht_layout_dir_mismatch]
loader-dht: subvol: loader-client-1; inode layout - 2147483647 - 4294967295;
disk layout - 3221225469 - 4294967295
[2011-02-24 09:32:50.846577] I [dht-common.c:415:dht_revalidate_cbk]
loader-dht: mismatching layouts for /
[2011-02-24 09:32:50.847650] I [dht-layout.c:588:dht_layout_normalize]
loader-dht: found anomalies in /. holes=1 overlaps=0
[2011-02-24 09:39:08.159175] I [fuse-bridge.c:3138:fuse_thread_proc] fuse:
unmounting /mnt/qe-filer01
[2011-02-24 09:39:08.160180] I [glusterfsd.c:672:cleanup_and_exit] glusterfsd:
shutting down
[2011-02-24 09:39:14.184554] W [io-stats.c:1644:init] loader: dangling volume.
check volfile
[2011-02-24 09:39:14.184599] W [dict.c:1204:data_to_str] dict: @data=(nil)
[2011-02-24 09:39:14.184614] W [dict.c:1204:data_to_str] dict: @data=(nil)
Given volfile:
+------------------------------------------------------------------------------+
1: volume loader-client-0
2: type protocol/client
3: option remote-host qe-filer03
4: option remote-subvolume /data1
5: option transport-type tcp
6: end-volume
7:
8: volume loader-client-1
9: type protocol/client
10: option remote-host qe-filer04
11: option remote-subvolume /data1
12: option transport-type tcp
13: end-volume
14:
15: volume loader-client-2
16: type protocol/client
17: option remote-host 172.24.8.25
18: option remote-subvolume /data1
19: option transport-type tcp
20: end-volume
21:
22: volume loader-client-3
23: type protocol/client
24: option remote-host qe-filer02
25: option remote-subvolume /data1
26: option transport-type tcp
27: end-volume
28:
29: volume loader-dht
30: type cluster/distribute
31: subvolumes loader-client-0 loader-client-1 loader-client-2
loader-client-3
32: end-volume
33:
34: volume loader-write-behind
35: type performance/write-behind
36: subvolumes loader-dht
37: end-volume
38:
39: volume loader-read-ahead
40: type performance/read-ahead
41: subvolumes loader-write-behind
42: end-volume
43:
44: volume loader-io-cache
45: type performance/io-cache
46: subvolumes loader-read-ahead
47: end-volume
48:
49: volume loader-quick-read
50: type performance/quick-read
51: subvolumes loader-io-cache
52: end-volume
53:
54: volume loader-stat-prefetch
55: type performance/stat-prefetch
56: subvolumes loader-quick-read
57: end-volume
58:
59: volume loader
60: type debug/io-stats
61: subvolumes loader-stat-prefetch
62: end-volume
+------------------------------------------------------------------------------+
[2011-02-24 09:39:18.132954] I
[client-handshake.c:993:select_server_supported_programs] loader-client-2:
Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-02-24 09:39:18.136077] I [client-handshake.c:829:client_setvolume_cbk]
loader-client-2: Connected to 172.24.8.25:24009, attached to remote volume
'/data1'.
[2011-02-24 09:39:18.136498] I
[client-handshake.c:993:select_server_supported_programs] loader-client-1:
Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-02-24 09:39:18.137083] I [client-handshake.c:829:client_setvolume_cbk]
loader-client-1: Connected to 172.24.8.222:24009, attached to remote volume
'/data1'.
[2011-02-24 09:39:18.140002] I
[client-handshake.c:993:select_server_supported_programs] loader-client-0:
Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-02-24 09:39:18.140599] I [client-handshake.c:829:client_setvolume_cbk]
loader-client-0: Connected to 172.24.8.221:24009, attached to remote volume
'/data1'.
[2011-02-24 09:39:18.144060] I
[client-handshake.c:993:select_server_supported_programs] loader-client-3:
Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-02-24 09:39:18.144689] I [client-handshake.c:829:client_setvolume_cbk]
loader-client-3: Connected to 172.24.8.220:24009, attached to remote volume
'/data1'.
[2011-02-24 09:39:18.150335] I [fuse-bridge.c:2817:fuse_init] glusterfs-fuse:
FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10
[2011-02-24 09:54:12.904354] W [fuse-bridge.c:888:fuse_unlink_cbk] glusterfs-fuse:
4094: UNLINK() / c9.lock => -1 (Permission denied)
[2011-02-24 09:54:34.716535] I [dht-common.c:369:dht_revalidate_cbk]
loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
[2011-02-24 09:54:36.470275] I [dht-common.c:369:dht_revalidate_cbk]
loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
[2011-02-24 09:55:24.588912] I [dht-common.c:369:dht_revalidate_cbk]
loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
[2011-02-24 09:55:26.180690] I [dht-common.c:369:dht_revalidate_cbk]
loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
[2011-02-24 09:55:57.816431] I [dht-common.c:369:dht_revalidate_cbk]
loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
[2011-02-24 09:55:59.242725] I [dht-common.c:369:dht_revalidate_cbk]
loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
[2011-02-24 09:56:54.92112] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse:
23558: / c9_refresh.lock => -1 (Permission denied)
[2011-02-24 09:57:01.792819] W [fuse-bridge.c:1510:fuse_create_cbk]
glusterfs-fuse: 23564: /Cloud9c9_refresh.lock => -1 (Permission denied)
[2011-02-24 09:57:31.812728] W [fuse-bridge.c:1510:fuse_create_cbk]
glusterfs-fuse: 23570: / c9_refresh.lock => -1 (Permission denied)
[2011-02-24 09:58:12.738834] W [fuse-bridge.c:1510:fuse_create_cbk]
glusterfs-fuse: 23582: / c9_refresh.lock => -1 (Permission denied)
[2011-02-24 09:59:10.210166] I [dht-common.c:369:dht_revalidate_cbk]
loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
[2011-02-24 09:59:11.851656] I [dht-common.c:369:dht_revalidate_cbk]
loader-dht: subvolume loader-client-3 returned -1 (Invalid argument)
I am running the rebalance again to see if it will fix the layout issue.
Any ideas why I'm seeing these errors after adding new nodes?
Regards,
John Lao
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
