Hi, Finally, this was a network problem. A switch was mangling jumbo frames.
Regards. 2015-05-13 9:07 GMT+01:00 Adrián Santos Marrero <[email protected]>: > > Hi Kaushal, > > Volume info: > > >> gluster02:~# gluster volume info all >> >> Volume Name: ovirt1 >> Type: Replicate >> Volume ID: 05171231-a4a5-48ae-bcfe-63408bc0fc0c >> Status: Started >> Number of Bricks: 1 x 2 = 2 >> Transport-type: tcp >> Bricks: >> Brick1: gluster01.stic.ull.es:/gluster/ovirt1_brick_01/brick >> Brick2: gluster02.stic.ull.es:/gluster/ovirt1_brick_01/brick >> Options Reconfigured: >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> cluster.server-quorum-type: none >> network.remote-dio: enable >> cluster.eager-lock: enable >> performance.stat-prefetch: off >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> >> Volume Name: ovirt2 >> Type: Replicate >> Volume ID: 0c0be558-9d98-4403-82d4-faafa2a3630a >> Status: Started >> Number of Bricks: 1 x 2 = 2 >> Transport-type: tcp >> Bricks: >> Brick1: gluster01.stic.ull.es:/gluster/ovirt2_brick_01/brick >> Brick2: gluster02.stic.ull.es:/gluster/ovirt2_brick_01/brick >> Options Reconfigured: >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> cluster.server-quorum-type: none >> network.remote-dio: enable >> cluster.eager-lock: enable >> performance.stat-prefetch: off >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> > > > And attached you can find the debug log for gluster03. > > Gluster01 is still re-healing so I am not sure if it is safe to restart > glusterd in DEBUG mode to obtain logs.... > > Regards. > > > 2015-05-13 8:41 GMT+01:00 Kaushal M <[email protected]>: > >> Hey Adrian, >> Can you list your volume infos? Most of the times the reason for peer >> state being stuck is because of some volume information being incorrect and >> the volume infos can help debug the problem. Could you also start glusterd >> in debug mode (`glusterd -LDEBUG`) and give logs? >> >> ~kaushal >> On May 13, 2015 9:13 AM, "Adrián Santos Marrero" <[email protected]> >> wrote: >> >>> Hi again, >>> >>> I upgraded to 3.6.3 all my gluster servers but peer probes still get >>> stuck in "Accepted peer request". >>> >>> Any hint? >>> >>> Regards. >>> >>> 2015-05-11 13:55 GMT+01:00 Adrián Santos Marrero <[email protected]>: >>> >>>> Hi, I've the same error with gluster03 (first installed with 3.6.2 >>>> version, but upgraded to 3.6.3 before any peer probe) and gluster04 (fresh >>>> install with Gluster 3.6.3). I attach the whole content of >>>> /var/log/glusterfs/ directory from gluster03. >>>> >>>> Regards. >>>> >>>> 2015-05-11 12:06 GMT+01:00 Gaurav Garg <[email protected]>: >>>> >>>>> Hi Adrian, >>>>> >>>>> Did you try to upgrade/downgrade gluster03 before performing any >>>>> operation. If you attach fresh node (3.6.3 version) to (3.6.2) glusterfs >>>>> version then this problems is not coming. we need to know what operation >>>>> you performed on gluster03 node before peer probe. if possible could you >>>>> attach cmd_history.log and glusterd log. >>>>> >>>>> ~ Gaurav >>>>> >>>>> ----- Original Message ----- >>>>> From: "Adrián Santos Marrero" <[email protected]> >>>>> To: "Gluster-users" <[email protected]> >>>>> Sent: Monday, May 11, 2015 12:39:33 PM >>>>> Subject: [Gluster-users] Adding a 3.6.3 peer to a 3.6.2 trusted pool >>>>> fails >>>>> >>>>> Hi, >>>>> >>>>> I've been trying to add a third node to my two-node trusted pool with >>>>> gluster peer probe: >>>>> >>>>> >>>>> >>>>> gluster02:~# gluster peer probe gluster03.stic.ull.es >>>>> peer probe: success. >>>>> >>>>> The problem is that this node remains in "Accepted peer request" state: >>>>> >>>>> >>>>> >>>>> Hostname: gluster03.stic.ull.es >>>>> Uuid: 9ca23fb2-cd76-4d1c-8a5f-f93306119539 >>>>> State: Accepted peer request (Connected) >>>>> >>>>> If I restart glusterd in gluster02, the state changes to "Sent and >>>>> Received peer request": >>>>> >>>>> >>>>> >>>>> Hostname: gluster03.stic.ull.es >>>>> Uuid: 9ca23fb2-cd76-4d1c-8a5f-f93306119539 >>>>> State: Sent and Received peer request (Connected) >>>>> >>>>> And if I restart glusterd in gluster03 it doesn't start because is >>>>> "Unable to find friend: gluster01.stic.ull.es ": >>>>> >>>>> >>>>> >>>>> [2015-05-11 06:59:39.035184] D >>>>> [glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname] >>>>> 0-management: Unable to find friend: gluster01.stic.ull.es >>>>> [2015-05-11 06:59:39.037942] D [common-utils.c:2897:gf_is_local_addr] >>>>> 0-management: 10.107.34.17 >>>>> [2015-05-11 06:59:39.039392] D [common-utils.c:2897:gf_is_local_addr] >>>>> 0-management: 10.107.34.17 >>>>> [2015-05-11 06:59:39.040584] D [common-utils.c:2897:gf_is_local_addr] >>>>> 0-management: 10.107.34.17 >>>>> [2015-05-11 06:59:39.041790] D [common-utils.c:2913:gf_is_local_addr] >>>>> 0-management: gluster01.stic.ull.es is not local >>>>> >>>>> In fact, the only peer that gluster03 knows is gluster02: >>>>> >>>>> >>>>> >>>>> gluster03:~# ls -l /var/lib/glusterd/peers/ >>>>> total 4 >>>>> -rw-------. 1 root root 73 may 11 07:48 >>>>> 739d7c3f-99d5-4c8b-ac28-f85163e26322 >>>>> gluster03:~# cat >>>>> /var/lib/glusterd/peers/739d7c3f-99d5-4c8b-ac28-f85163e26322 >>>>> uuid=739d7c3f-99d5-4c8b-ac28-f85163e26322 >>>>> state=5 >>>>> hostname1=10.107.34.18 >>>>> >>>>> 10.107.34.18 == gluster02 >>>>> >>>>> >>>>> >>>>> gluster03:~# gluster --version >>>>> glusterfs 3.6.3 built on Apr 23 2015 16:11:45 >>>>> Repository revision: git:// git.gluster.com/glusterfs.git >>>>> Copyright (c) 2006-2011 Gluster Inc. < http://www.gluster.com > >>>>> GlusterFS comes with ABSOLUTELY NO WARRANTY. >>>>> You may redistribute copies of GlusterFS under the terms of the GNU >>>>> General Public License. >>>>> >>>>> All servers are running Centos 7, gluster01 and gluster02 have Gluster >>>>> 3.6.2 version and gluster03 3.6.3. >>>>> >>>>> Am I forced to upgrade gluster01 and gluster02 to 3.6.3? >>>>> >>>>> Regards. >>>>> >>>>> -- >>>>> ---------------------------------------------------------------- >>>>> Adrián Santos Marrero >>>>> Técnico de Sistemas - Área de Infraestructuras TIC >>>>> Servicios de Tecnologías de la Información y Comunicación (STIC) >>>>> Universidad de La Laguna (ULL) >>>>> Teléfono/Phone: +34 922 845089 >>>>> ---------------------------------------------------------------- >>>>> >>>>> Este mensaje puede contener información confidencial y/o privilegiada. >>>>> Si usted no es el destinatario o lo ha recibido por error debe borrarlo >>>>> inmediatamente. >>>>> Está estrictamente prohibido por la legislación vigente realizar >>>>> cualquier copia, revelación o distribución del contenido de este >>>>> mensaje >>>>> sin autorización expresa. >>>>> >>>>> This e-mail may contain confidential and/or privileged information. >>>>> If you are not the intended recipient or have received this e-mail in >>>>> error you must destroy it. >>>>> Any unauthorised copying, disclosure or distribution of the material in >>>>> this e-mail is strictly forbidden by current legislation. >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> [email protected] >>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>> >>>> >>>> >>>> >>>> -- >>>> ---------------------------------------------------------------- >>>> Adrián Santos Marrero >>>> Técnico de Sistemas - Área de Infraestructuras TIC >>>> Servicios de Tecnologías de la Información y Comunicación (STIC) >>>> Universidad de La Laguna (ULL) >>>> Teléfono/Phone: +34 922 845089 >>>> ---------------------------------------------------------------- >>>> >>>> Este mensaje puede contener información confidencial y/o privilegiada. >>>> Si usted no es el destinatario o lo ha recibido por error debe borrarlo >>>> inmediatamente. >>>> Está estrictamente prohibido por la legislación vigente realizar >>>> cualquier copia, revelación o distribución del contenido de este mensaje >>>> sin autorización expresa. >>>> >>>> This e-mail may contain confidential and/or privileged information. >>>> If you are not the intended recipient or have received this e-mail in >>>> error you must destroy it. >>>> Any unauthorised copying, disclosure or distribution of the material in >>>> this e-mail is strictly forbidden by current legislation. >>>> >>> >>> >>> >>> -- >>> ---------------------------------------------------------------- >>> Adrián Santos Marrero >>> Técnico de Sistemas - Área de Infraestructuras TIC >>> Servicios de Tecnologías de la Información y Comunicación (STIC) >>> Universidad de La Laguna (ULL) >>> Teléfono/Phone: +34 922 845089 >>> ---------------------------------------------------------------- >>> >>> Este mensaje puede contener información confidencial y/o privilegiada. >>> Si usted no es el destinatario o lo ha recibido por error debe borrarlo >>> inmediatamente. >>> Está estrictamente prohibido por la legislación vigente realizar >>> cualquier copia, revelación o distribución del contenido de este mensaje >>> sin autorización expresa. >>> >>> This e-mail may contain confidential and/or privileged information. >>> If you are not the intended recipient or have received this e-mail in >>> error you must destroy it. >>> Any unauthorised copying, disclosure or distribution of the material in >>> this e-mail is strictly forbidden by current legislation. >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> [email protected] >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >> > > > -- > ---------------------------------------------------------------- > Adrián Santos Marrero > Técnico de Sistemas - Área de Infraestructuras TIC > Servicios de Tecnologías de la Información y Comunicación (STIC) > Universidad de La Laguna (ULL) > Teléfono/Phone: +34 922 845089 > ---------------------------------------------------------------- > > Este mensaje puede contener información confidencial y/o privilegiada. > Si usted no es el destinatario o lo ha recibido por error debe borrarlo > inmediatamente. > Está estrictamente prohibido por la legislación vigente realizar > cualquier copia, revelación o distribución del contenido de este mensaje > sin autorización expresa. > > This e-mail may contain confidential and/or privileged information. > If you are not the intended recipient or have received this e-mail in > error you must destroy it. > Any unauthorised copying, disclosure or distribution of the material in > this e-mail is strictly forbidden by current legislation. > -- ---------------------------------------------------------------- Adrián Santos Marrero Técnico de Sistemas - Área de Infraestructuras TIC Servicios de Tecnologías de la Información y Comunicación (STIC) Universidad de La Laguna (ULL) Teléfono/Phone: +34 922 845089 ---------------------------------------------------------------- Este mensaje puede contener información confidencial y/o privilegiada. Si usted no es el destinatario o lo ha recibido por error debe borrarlo inmediatamente. Está estrictamente prohibido por la legislación vigente realizar cualquier copia, revelación o distribución del contenido de este mensaje sin autorización expresa. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error you must destroy it. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden by current legislation.
_______________________________________________ Gluster-users mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-users
