Hi,
answer below.
Cheers,
Laurent DOUCHY.
System Administrator
On 9/1/11 8:34 PM, Mohit Anchlia wrote:
Can you also list the files in /etc/glusterd/peers on node 07 and node
05? Can you check if you are missing for node07 on node05?
[root@node00 ~]# ssh node05
Last login: Thu Sep 1 21:43:12 2011 from node00-0.isdc.unige.ch
[root@node05 ~]# ls -l /etc/glusterd/peers/
total 40
-rw-r--r-- 1 root root 72 Sep 1 21:46 212ce5a0-de51-4a98-9262-ae071c2d63a0
-rw-r--r-- 1 root root 72 Sep 1 21:46 3142fb9a-0a6b-46ec-9262-ede95e8f798a
-rw-r--r-- 1 root root 75 Sep 1 21:46 a35fb0a1-af35-4a04-b38a-434f68369508
-rw-r--r-- 1 root root 72 Sep 1 21:46 dd114546-5b94-4a62-9301-260703bf5707
-rw-r--r-- 1 root root 72 Sep 1 21:46 f73fee83-8d47-4f07-bfac-b8a8592eff04
[root@node05 ~]# ssh node07
root@node07's password:
Last login: Thu Sep 1 19:44:10 2011 from node00-0.isdc.unige.ch
[root@node07 ~]# ls -l /etc/glusterd/peers/
total 40
-rw-r--r-- 1 root root 72 Sep 1 19:47 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
-rw-r--r-- 1 root root 72 Sep 1 19:47 212ce5a0-de51-4a98-9262-ae071c2d63a0
-rw-r--r-- 1 root root 72 Sep 1 19:47 3142fb9a-0a6b-46ec-9262-ede95e8f798a
-rw-r--r-- 1 root root 72 Aug 31 17:02 dd114546-5b94-4a62-9301-260703bf5707
-rw-r--r-- 1 root root 72 Sep 1 19:47 f73fee83-8d47-4f07-bfac-b8a8592eff04
seems good.
You could also try and do detach again and remove everything from
peers and re-do steps and see if that helps.
I detach node05 and node06.
restart glusterd on all machines
delete files on /etc/glusterd/peer/
restart glusterd on all machine
the status of peer is good
add node05 and node06 to the glusterfs
restart glusterd on all machine
the status of peer is good
but:
[root@node07 ~]# gluster volume add-brick cluster node05:/gluster2
node06:/gluster2
Operation failed on node05
I can't understand, i install exactly on the same way each node, deploy
the same version of gluster but only 4 machines are working properly.
On Thu, Sep 1, 2011 at 11:01 AM, Laurent DOUCHY<[email protected]> wrote:
see below.
Cheers,
Laurent DOUCHY.
System Administrator
ISDC Data Centre for Astrophysics
16, ch. d'Ecogia
CH-1290 VERSOIX
Tel.: +41 (0)22 379 21 31
On 9/1/11 7:11 PM, Mohit Anchlia wrote:
Can you paste the error logs from node05 and the node you are using to
perform add.
the one I'm using to perform the add:
[2011-09-01 19:44:13.903186] I
[glusterd-handler.c:1305:glusterd_handle_add_brick] 0-glusterd: Received add
brick req
[2011-09-01 19:44:13.903274] I [glusterd-utils.c:243:glusterd_lock]
0-glusterd: Cluster lock held by a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 19:44:13.903284] I
[glusterd-handler.c:420:glusterd_op_txn_begin] 0-glusterd: Acquired local
lock
[2011-09-01 19:44:13.903795] I
[glusterd-rpc-ops.c:742:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received
ACC from uuid: dd114546-5b94-4a62-9301-260703bf5707
[2011-09-01 19:44:13.903834] I
[glusterd-rpc-ops.c:742:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received
ACC from uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
[2011-09-01 19:44:13.904069] I
[glusterd-rpc-ops.c:742:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received
ACC from uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
[2011-09-01 19:44:13.904167] I
[glusterd-rpc-ops.c:742:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received
ACC from uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
[2011-09-01 19:44:13.904188] I
[glusterd-rpc-ops.c:742:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received
ACC from uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
[2011-09-01 19:44:13.904214] I
[glusterd-utils.c:776:glusterd_volume_brickinfo_get_by_brick] 0-: brick:
node05:/gluster2
[2011-09-01 19:44:13.904231] I
[glusterd-utils.c:776:glusterd_volume_brickinfo_get_by_brick] 0-: brick:
node06:/gluster2
[2011-09-01 19:44:13.904305] I
[glusterd-op-sm.c:6453:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req
to 5 peers
[2011-09-01 19:44:13.904670] I
[glusterd-rpc-ops.c:1040:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC
from uuid: dd114546-5b94-4a62-9301-260703bf5707
[2011-09-01 19:44:13.904703] I
[glusterd-rpc-ops.c:1040:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC
from uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
[2011-09-01 19:44:13.904725] I
[glusterd-rpc-ops.c:1040:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC
from uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
[2011-09-01 19:44:13.905542] I
[glusterd-rpc-ops.c:1040:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC
from uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
[2011-09-01 19:44:13.905655] I
[glusterd-rpc-ops.c:1040:glusterd3_1_stage_op_cbk] 0-glusterd: Received RJT
from uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
[2011-09-01 19:44:13.905929] I
[glusterd-rpc-ops.c:801:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received
ACC from uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
[2011-09-01 19:44:13.905988] I
[glusterd-rpc-ops.c:801:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received
ACC from uuid: dd114546-5b94-4a62-9301-260703bf5707
[2011-09-01 19:44:13.906019] I
[glusterd-rpc-ops.c:801:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received
ACC from uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
[2011-09-01 19:44:13.906039] I
[glusterd-rpc-ops.c:801:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received
ACC from uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
[2011-09-01 19:44:13.906058] I
[glusterd-rpc-ops.c:801:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received
ACC from uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
[2011-09-01 19:44:13.906070] I
[glusterd-op-sm.c:6987:glusterd_op_txn_complete] 0-glusterd: Cleared local
lock
[2011-09-01 19:44:13.906677] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (127.0.0.1:1016)
node05:
[2011-09-01 20:57:03.419202] I
[glusterd-handler.c:448:glusterd_handle_cluster_lock] 0-glusterd: Received
LOCK from uuid: a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 20:57:03.419292] I [glusterd-utils.c:243:glusterd_lock]
0-glusterd: Cluster lock held by a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 20:57:03.419320] I
[glusterd-handler.c:2641:glusterd_op_lock_send_resp] 0-glusterd: Responded,
ret: 0
[2011-09-01 20:57:03.419818] I
[glusterd-handler.c:488:glusterd_req_ctx_create] 0-glusterd: Received op
from uuid: a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 20:57:03.419896] I
[glusterd-utils.c:776:glusterd_volume_brickinfo_get_by_brick] 0-: brick:
node05:/gluster2
[2011-09-01 20:57:03.420141] E
[glusterd-op-sm.c:715:glusterd_op_stage_add_brick] 0-glusterd: resolve brick
failed
[2011-09-01 20:57:03.420157] E
[glusterd-op-sm.c:7107:glusterd_op_ac_stage_op] 0-: Validate failed: 1
[2011-09-01 20:57:03.420183] I
[glusterd-handler.c:2733:glusterd_op_stage_send_resp] 0-glusterd: Responded
to stage, ret: 0
[2011-09-01 20:57:03.420508] I
[glusterd-handler.c:2683:glusterd_handle_cluster_unlock] 0-glusterd:
Received UNLOCK from uuid: a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 20:57:03.420553] I
[glusterd-handler.c:2661:glusterd_op_unlock_send_resp] 0-glusterd: Responded
to unlock, ret: 0
[2011-09-01 21:43:43.584850] I
[glusterd-handler.c:448:glusterd_handle_cluster_lock] 0-glusterd: Received
LOCK from uuid: a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 21:43:43.585045] I [glusterd-utils.c:243:glusterd_lock]
0-glusterd: Cluster lock held by a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 21:43:43.585077] I
[glusterd-handler.c:2641:glusterd_op_lock_send_resp] 0-glusterd: Responded,
ret: 0
[2011-09-01 21:43:43.585341] I
[glusterd-handler.c:488:glusterd_req_ctx_create] 0-glusterd: Received op
from uuid: a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 21:43:43.585424] I
[glusterd-utils.c:776:glusterd_volume_brickinfo_get_by_brick] 0-: brick:
node05:/gluster2
[2011-09-01 21:43:43.586524] E
[glusterd-op-sm.c:715:glusterd_op_stage_add_brick] 0-glusterd: resolve brick
failed
[2011-09-01 21:43:43.586540] E
[glusterd-op-sm.c:7107:glusterd_op_ac_stage_op] 0-: Validate failed: 1
[2011-09-01 21:43:43.586567] I
[glusterd-handler.c:2733:glusterd_op_stage_send_resp] 0-glusterd: Responded
to stage, ret: 0
[2011-09-01 21:43:43.586873] I
[glusterd-handler.c:2683:glusterd_handle_cluster_unlock] 0-glusterd:
Received UNLOCK from uuid: a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 21:43:43.586910] I
[glusterd-handler.c:2661:glusterd_op_unlock_send_resp] 0-glusterd: Responded
to unlock, ret: 0
I have to fix the time issues.
I fix the time issues but steel the same message
[root@node07 ~]# gluster volume add-brick cluster node05:/gluster2
node06:/gluster2
Operation failed on node05
Last thing you could try is gluster peer detach node 5 and 6 and then
add them back and try again.
I've done the detach, restart glusterd, probe, check status, restart
glusterd, check status. everything ok ... but:
[root@node07 ~]# gluster volume add-brick cluster node05:/gluster2
node06:/gluster2
Operation failed on node05
On Thu, Sep 1, 2011 at 9:58 AM, Laurent DOUCHY<[email protected]>
wrote:
ping is ok
restart of gluster done :
[root@node00 ~]# for i in `seq -w 1 10` ; do echo ; echo node$i ; echo ;
ssh
node$i "service glusterd restart";done
node01
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node02
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node03
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node04
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node05
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node06
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node07
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node08
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node09
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node10
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
but same error message ...
[root@node07 ~]# gluster volume add-brick cluster node05:/gluster2
node06:/gluster2
Operation failed on node05
Cheers,
Laurent DOUCHY.
On 9/1/11 6:54 PM, Mohit Anchlia wrote:
Can you ping node05 from node07 from where you are trying to do the
add? Also, try restarting gluster process on every node and try again.
On Thu, Sep 1, 2011 at 9:39 AM, Laurent DOUCHY<[email protected]>
wrote:
see below
Cheers,
Laurent DOUCHY.
On 9/1/11 6:01 PM, Mohit Anchlia wrote:
You can check few things on 5 and 6:
1) gluster processes are running on node5 and 6
yes:
node05
root 4902 1 0 Aug31 ? 00:00:00
/opt/glusterfs/3.2.2/sbin/glusterd
root 9626 1 0 19:55 ? 00:00:00
/opt/glusterfs/3.2.2/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol
-p
/etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
root 9690 9686 0 20:04 ? 00:00:00 bash -c ps -edf | grep
gluster
root 9704 9690 0 20:04 ? 00:00:00 grep gluster
node06
root 4441 1 0 Aug31 ? 00:00:00
/opt/glusterfs/3.2.2/sbin/glusterd
root 9178 1 0 19:55 ? 00:00:00
/opt/glusterfs/3.2.2/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol
-p
/etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
root 9242 9238 0 20:04 ? 00:00:00 bash -c ps -edf | grep
gluster
root 9256 9242 0 20:04 ? 00:00:00 grep gluster
2) both nodes are able to see each other
yes:
[root@node05 ~]# ping node06
PING node06.isdc.unige.ch (129.194.168.70) 56(84) bytes of data.
64 bytes from node06.isdc.unige.ch (129.194.168.70): icmp_seq=1 ttl=64
time=0.376 ms
[root@node06 ~]# ping node05
PING node05.isdc.unige.ch (129.194.168.69) 56(84) bytes of data.
64 bytes from node05.isdc.unige.ch (129.194.168.69): icmp_seq=1 ttl=64
time=0.337 ms
3) do gluster peer status on both the nodes and see what you see
node 5 trust node 6 and node 6 trust node 5
[root@node05 ~]# gluster peer status
Number of Peers: 5
Hostname: node08
Uuid: dd114546-5b94-4a62-9301-260703bf5707
State: Peer in Cluster (Connected)
Hostname: node06
Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
State: Peer in Cluster (Connected)
Hostname: node10
Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
State: Peer in Cluster (Connected)
Hostname: 129.194.168.71
Uuid: a35fb0a1-af35-4a04-b38a-434f68369508
State: Peer in Cluster (Connected)
Hostname: node09
Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
State: Peer in Cluster (Connected)
[root@node06 ~]# gluster peer status
Number of Peers: 5
Hostname: node08
Uuid: dd114546-5b94-4a62-9301-260703bf5707
State: Peer in Cluster (Connected)
Hostname: node09
Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
State: Peer in Cluster (Connected)
Hostname: node05
Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
State: Peer in Cluster (Connected)
Hostname: node10
Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
State: Peer in Cluster (Connected)
Hostname: 129.194.168.71
Uuid: a35fb0a1-af35-4a04-b38a-434f68369508
State: Peer in Cluster (Connected)
4) check iptables
same file on each node (the installation is manage by puppet)
On Thu, Sep 1, 2011 at 8:57 AM, Laurent
DOUCHY<[email protected]>
wrote:
It works ...
[root@node07 ~]# gluster volume add-brick cluster node09:/gluster3
node10:/gluster3
Add Brick successful
On 9/1/11 5:39 PM, Mohit Anchlia wrote:
Can you try with node09:/gluster3 and node10:gluster3 instead?
On Thu, Sep 1, 2011 at 2:49 AM, Laurent
DOUCHY<[email protected]>
wrote:
Hi,
I work on the node7 so it doesn't appear on the list.
I create a folder /gluster3 on node5 and node6 and try to add them
to
my
volume but it failed with the same message :(
[root@node07 ~]# gluster volume add-brick cluster node05:/gluster3
node06:/gluster3
Operation failed on node05
next step is to reinstall from scratch the node I hope I can avoid
this.
On 8/31/11 9:08 PM, Mohit Anchlia wrote:
I don't see node07 in above output of gluster peer status.
Can you try to add bricks on the hosts that gluster1, gluster2? So
add
gluster3 and see if that works.
On Wed, Aug 31, 2011 at 11:56 AM, Laurent DOUCHY
<[email protected]> wrote:
Hi,
I try to add 2 bricks or 4 bricks for the same effect.
I try to reinstall gluster without success.
Cheers,
Laurent DOUCHY.
On 8/31/11 8:07 PM, Burnash, James wrote:
Hi Laurent.
Since your configuration specifies replication, you must add
bricks
in
the
same number as your number of replicas.
For instance - if you have 2 replicas (most normal case), you
would
need
to do something like this:
gluster volume add-brick cluster node05:/gluster1
node06:/gluster1
James Burnash
Unix Engineer
Knight Capital Group
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Laurent
DOUCHY
Sent: Wednesday, August 31, 2011 12:49 PM
To: [email protected]
Subject: [Gluster-users] add bricks on distributed replicated
volume
failed
Hi,
I'm using gluster 3.2.2 on 10 nodes. Each node have 2x2 TB disk
for
gluster.
I manage to configure a distributed and replicated volume on 4
nodes
:
[root@node07 ~]# gluster volume info cluster
Volume Name: cluster
Type: Distributed-Replicate
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: node09:/gluster1
Brick2: node10:/gluster1
Brick3: node09:/gluster2
Brick4: node10:/gluster2
Brick5: node07:/gluster1
Brick6: node08:/gluster1
Brick7: node07:/gluster2
Brick8: node08:/gluster2
But I can't add new nodes to this volume
[root@node07 ~]# gluster peer status
Number of Peers: 5
Hostname: node10
Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
State: Peer in Cluster (Connected)
Hostname: node08
Uuid: dd114546-5b94-4a62-9301-260703bf5707
State: Peer in Cluster (Connected)
Hostname: node09
Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
State: Peer in Cluster (Connected)
Hostname: node06
Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
State: Peer in Cluster (Connected)
Hostname: node05
Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
State: Peer in Cluster (Connected)
[root@node07 ~]# gluster volume add-brick cluster
node05:/gluster1
node06:/gluster1 node05:/gluster2 node06:/gluster2 Operation
failed
on
node05
I try to detach nodes 5 and 6, restart glusterd do the probe and
the
add-brick but still nothing ...
Did some one have any idea to fix this ?
Thanks in advance,
Laurent.
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
DISCLAIMER:
This e-mail, and any attachments thereto, is intended only for
use
by
the
addressee(s) named herein and may contain legally privileged
and/or
confidential information. If you are not the intended recipient
of
this
e-mail, you are hereby notified that any dissemination,
distribution
or
copying of this e-mail, and any attachments thereto, is strictly
prohibited.
If you have received this in error, please immediately notify me
and
permanently delete the original and any copy of any e-mail and
any
printout
thereof. E-mail transmission cannot be guaranteed to be secure
or
error-free. The sender therefore does not accept liability for
any
errors or
omissions in the contents of this message which arise as a
result
of
e-mail
transmission.
NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital
Group
may,
at
its discretion, monitor and review the content of all e-mail
communications.
http://www.knight.com
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
