Hi Micha, Sorry for the late reply. I was busy with some other things.
If you have still the setup available Can you enable TRACE log level [1],[2] and see if you could find any log entries when the network start disconnecting. Basically I'm trying to find out any disconnection had occurred other than ping timer expire issue. [1] : gluster volume <volname> diagnostics.brick-log-level TRACE [2] : gluster volume <volname> diagnostics.client-log-level TRACE Regards Rafi KC On 12/08/2016 07:59 PM, Atin Mukherjee wrote: > > > On Thu, Dec 8, 2016 at 4:37 PM, Micha Ober <[email protected] > <mailto:[email protected]>> wrote: > > Hi Rafi, > > thank you for your support. It is greatly appreciated. > > Just some more thoughts from my side: > > There have been no reports from other users in *this* thread > until now, but I have found at least one user with a very simiar > problem in an older thread: > > https://www.gluster.org/pipermail/gluster-users/2014-November/019637.html > > <https://www.gluster.org/pipermail/gluster-users/2014-November/019637.html> > > He is also reporting disconnects with no apparent reasons, > althogh his setup is a bit more complicated, also involving a > firewall. In our setup, all servers/clients are connected via 1 > GbE with no firewall or anything that might block/throttle > traffic. Also, we are using exactly the same software versions on > all nodes. > > > I can also find some reports in the bugtracker when searching for > "rpc_client_ping_timer_expired" and "rpc_clnt_ping_timer_expired" > (looks like spelling changed during versions). > > https://bugzilla.redhat.com/show_bug.cgi?id=1096729 > <https://bugzilla.redhat.com/show_bug.cgi?id=1096729> > > > Just FYI, this is a different issue, here GlusterD fails to handle the > volume of incoming requests on time since MT-epoll is not enabled here. > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1370683 > <https://bugzilla.redhat.com/show_bug.cgi?id=1370683> > > But both reports involve large traffic/load on the bricks/disks, > which is not the case for out setup. > To give a ballpark figure: Over three days, 30 GiB were written. > And the data was not written at once, but continuously over the > whole time. > > > Just to be sure, I have checked the logfiles of one of the other > clusters right now, which are sitting in the same building, in the > same rack, even on the same switch, running the same jobs, but > with glusterfs 3.4.2 and I can see no disconnects in the logfiles. > So I can definitely rule out our infrastructure as problem. > > Regards, > Micha > > > > Am 07.12.2016 um 18:08 schrieb Mohammed Rafi K C: >> >> Hi Micha, >> >> This is great. I will provide you one debug build which has two >> fixes which I possible suspect for a frequent disconnect issue, >> though I don't have much data to validate my theory. So I will >> take one more day to dig in to that. >> >> Thanks for your support, and opensource++ >> >> Regards >> >> Rafi KC >> >> On 12/07/2016 05:02 AM, Micha Ober wrote: >>> Hi, >>> >>> thank you for your answer and even more for the question! >>> Until now, I was using FUSE. Today I changed all mounts to NFS >>> using the same 3.7.17 version. >>> >>> But: The problem is still the same. Now, the NFS logfile >>> contains lines like these: >>> >>> [2016-12-06 15:12:29.006325] C >>> [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] >>> 0-gv0-client-7: server X.X.18.62:49153 has not responded in the >>> last 42 seconds, disconnecting. >>> >>> Interestingly enough, the IP address X.X.18.62 is the same >>> machine! As I wrote earlier, each node serves both as a server >>> and a client, as each node contributes bricks to the volume. >>> Every server is connecting to itself via its hostname. For >>> example, the fstab on the node "giant2" looks like: >>> >>> #giant2:/gv0 /shared_data glusterfs defaults,noauto >>> 0 0 >>> #giant2:/gv2 /shared_slurm glusterfs defaults,noauto >>> 0 0 >>> >>> giant2:/gv0 /shared_data nfs >>> defaults,_netdev,vers=3 0 0 >>> giant2:/gv2 /shared_slurm nfs >>> defaults,_netdev,vers=3 0 0 >>> >>> So I understand the disconnects even less. >>> >>> I don't know if it's possible to create a dummy cluster which >>> exposes the same behaviour, because the disconnects only happen >>> when there are compute jobs running on those nodes - and they >>> are GPU compute jobs, so that's something which cannot be easily >>> emulated in a VM. >>> >>> As we have more clusters (which are running fine with an ancient >>> 3.4 version :-)) and we are currently not dependent on this >>> particular cluster (which may stay like this for this month, I >>> think) I should be able to deploy the debug build on the "real" >>> cluster, if you can provide a debug build. >>> >>> Regards and thanks, >>> Micha >>> >>> >>> >>> Am 06.12.2016 um 08:15 schrieb Mohammed Rafi K C: >>>> >>>> >>>> >>>> On 12/03/2016 12:56 AM, Micha Ober wrote: >>>>> ** Update: ** I have downgraded from 3.8.6 to 3.7.17 now, but >>>>> the problem still exists. >>>>> >>>>> Client log: http://paste.ubuntu.com/23569065/ >>>>> <http://paste.ubuntu.com/23569065/> >>>>> Brick log: http://paste.ubuntu.com/23569067/ >>>>> <http://paste.ubuntu.com/23569067/> >>>>> >>>>> Please note that each server has two bricks. >>>>> Whereas, according to the logs, one brick loses the connection >>>>> to all other hosts: >>>>> [2016-12-02 18:38:53.703301] W [socket.c:596:__socket_rwv] >>>>> 0-tcp.gv0-server: writev on X.X.X.219:49121 failed (Broken pipe) >>>>> [2016-12-02 18:38:53.703381] W [socket.c:596:__socket_rwv] >>>>> 0-tcp.gv0-server: writev on X.X.X.62:49118 failed (Broken pipe) >>>>> [2016-12-02 18:38:53.703380] W [socket.c:596:__socket_rwv] >>>>> 0-tcp.gv0-server: writev on X.X.X.107:49121 failed (Broken pipe) >>>>> [2016-12-02 18:38:53.703424] W [socket.c:596:__socket_rwv] >>>>> 0-tcp.gv0-server: writev on X.X.X.206:49120 failed (Broken pipe) >>>>> [2016-12-02 18:38:53.703359] W [socket.c:596:__socket_rwv] >>>>> 0-tcp.gv0-server: writev on X.X.X.58:49121 failed (Broken pipe) >>>>> >>>>> The SECOND brick on the SAME host is NOT affected, i.e. no >>>>> disconnects! >>>>> As I said, the network connection is fine and the disks are idle. >>>>> The CPU always has 2 free cores. >>>>> >>>>> It looks like I have to downgrade to 3.4 now in order for the >>>>> disconnects to stop. >>>> >>>> Hi Micha, >>>> >>>> Thanks for the update and sorry for what happened with gluster >>>> higher versions. I can understand the need for downgrade as it >>>> is a production setup. >>>> >>>> Can you tell me the clients used here ? whether it is a >>>> fuse,nfs,nfs-ganesha, smb or libgfapi ? >>>> >>>> Since I'm not able to reproduce the issue (I have been trying >>>> from last 3days) and the logs are not much helpful here (we >>>> don't have much logs in socket layer), Could you please create >>>> a dummy cluster and try to reproduce the issue? If then we can >>>> play with that volume and I could provide some debug build >>>> which we can use for further debugging? >>>> >>>> If you don't have bandwidth for this, please leave it ;). >>>> >>>> Regards >>>> Rafi KC >>>> >>>>> - Micha >>>>> >>>>> Am 30.11.2016 um 06:57 schrieb Mohammed Rafi K C: >>>>>> >>>>>> Hi Micha, >>>>>> >>>>>> I have changed the thread and subject so that your original >>>>>> thread remain same for your query. Let's try to fix the >>>>>> problem what you observed with 3.8.4, So I have started a new >>>>>> thread to discuss the frequent disconnect problem. >>>>>> >>>>>> *If any one else has experienced the same problem, please >>>>>> respond to the mail.* >>>>>> >>>>>> It would be very helpful if you could give us some more logs >>>>>> from clients and bricks. Also any reproducible steps will >>>>>> surely help to chase the problem further. >>>>>> >>>>>> Regards >>>>>> >>>>>> Rafi KC >>>>>> >>>>>> On 11/30/2016 04:44 AM, Micha Ober wrote: >>>>>>> I had opened another thread on this mailing list (Subject: >>>>>>> "After upgrade from 3.4.2 to 3.8.5 - High CPU usage >>>>>>> resulting in disconnects and split-brain"). >>>>>>> >>>>>>> The title may be a bit misleading now, as I am no longer >>>>>>> observing high CPU usage after upgrading to 3.8.6, but the >>>>>>> disconnects are still happening and the number of files in >>>>>>> split-brain is growing. >>>>>>> >>>>>>> Setup: 6 compute nodes, each serving as a glusterfs server >>>>>>> and client, Ubuntu 14.04, two bricks per node, >>>>>>> distribute-replicate >>>>>>> >>>>>>> I have two gluster volumes set up (one for scratch data, one >>>>>>> for the slurm scheduler). Only the scratch data volume shows >>>>>>> critical errors "[...] has not responded in the last 42 >>>>>>> seconds, disconnecting.". So I can rule out network >>>>>>> problems, the gigabit link between the nodes is not >>>>>>> saturated at all. The disks are almost idle (<10%). >>>>>>> >>>>>>> I have glusterfs 3.4.2 on Ubuntu 12.04 on a another compute >>>>>>> cluster, running fine since it was deployed. >>>>>>> I had glusterfs 3.4.2 on Ubuntu 14.04 on this cluster, >>>>>>> running fine for almost a year. >>>>>>> >>>>>>> After upgrading to 3.8.5, the problems (as described) >>>>>>> started. I would like to use some of the new features of the >>>>>>> newer versions (like bitrot), but the users can't run their >>>>>>> compute jobs right now because the result files are garbled. >>>>>>> >>>>>>> There also seems to be a bug report with a smiliar problem: >>>>>>> (but no progress) >>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1370683 >>>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1370683> >>>>>>> >>>>>>> For me, ALL servers are affected (not isolated to one or two >>>>>>> servers) >>>>>>> >>>>>>> I also see messages like "INFO: task gpu_graphene_bv:4476 >>>>>>> blocked for more than 120 seconds." in the syslog. >>>>>>> >>>>>>> For completeness (gv0 is the scratch volume, gv2 the slurm >>>>>>> volume): >>>>>>> >>>>>>> [root@giant2: ~]# gluster v info >>>>>>> >>>>>>> Volume Name: gv0 >>>>>>> Type: Distributed-Replicate >>>>>>> Volume ID: 993ec7c9-e4bc-44d0-b7c4-2d977e622e86 >>>>>>> Status: Started >>>>>>> Snapshot Count: 0 >>>>>>> Number of Bricks: 6 x 2 = 12 >>>>>>> Transport-type: tcp >>>>>>> Bricks: >>>>>>> Brick1: giant1:/gluster/sdc/gv0 >>>>>>> Brick2: giant2:/gluster/sdc/gv0 >>>>>>> Brick3: giant3:/gluster/sdc/gv0 >>>>>>> Brick4: giant4:/gluster/sdc/gv0 >>>>>>> Brick5: giant5:/gluster/sdc/gv0 >>>>>>> Brick6: giant6:/gluster/sdc/gv0 >>>>>>> Brick7: giant1:/gluster/sdd/gv0 >>>>>>> Brick8: giant2:/gluster/sdd/gv0 >>>>>>> Brick9: giant3:/gluster/sdd/gv0 >>>>>>> Brick10: giant4:/gluster/sdd/gv0 >>>>>>> Brick11: giant5:/gluster/sdd/gv0 >>>>>>> Brick12: giant6:/gluster/sdd/gv0 >>>>>>> Options Reconfigured: >>>>>>> auth.allow: X.X.X.*,127.0.0.1 >>>>>>> nfs.disable: on >>>>>>> >>>>>>> Volume Name: gv2 >>>>>>> Type: Replicate >>>>>>> Volume ID: 30c78928-5f2c-4671-becc-8deaee1a7a8d >>>>>>> Status: Started >>>>>>> Snapshot Count: 0 >>>>>>> Number of Bricks: 1 x 2 = 2 >>>>>>> Transport-type: tcp >>>>>>> Bricks: >>>>>>> Brick1: giant1:/gluster/sdd/gv2 >>>>>>> Brick2: giant2:/gluster/sdd/gv2 >>>>>>> Options Reconfigured: >>>>>>> auth.allow: X.X.X.*,127.0.0.1 >>>>>>> cluster.granular-entry-heal: on >>>>>>> cluster.locking-scheme: granular >>>>>>> nfs.disable: on >>>>>>> >>>>>>> >>>>>>> 2016-11-30 0:10 GMT+01:00 Micha Ober <[email protected] >>>>>>> <mailto:[email protected]>>: >>>>>>> >>>>>>> There also seems to be a bug report with a smiliar >>>>>>> problem: (but no progress) >>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1370683 >>>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1370683> >>>>>>> >>>>>>> For me, ALL servers are affected (not isolated to one or >>>>>>> two servers) >>>>>>> >>>>>>> I also see messages like "INFO: task >>>>>>> gpu_graphene_bv:4476 blocked for more than 120 seconds." >>>>>>> in the syslog. >>>>>>> >>>>>>> For completeness (gv0 is the scratch volume, gv2 the >>>>>>> slurm volume): >>>>>>> >>>>>>> [root@giant2: ~]# gluster v info >>>>>>> >>>>>>> Volume Name: gv0 >>>>>>> Type: Distributed-Replicate >>>>>>> Volume ID: 993ec7c9-e4bc-44d0-b7c4-2d977e622e86 >>>>>>> Status: Started >>>>>>> Snapshot Count: 0 >>>>>>> Number of Bricks: 6 x 2 = 12 >>>>>>> Transport-type: tcp >>>>>>> Bricks: >>>>>>> Brick1: giant1:/gluster/sdc/gv0 >>>>>>> Brick2: giant2:/gluster/sdc/gv0 >>>>>>> Brick3: giant3:/gluster/sdc/gv0 >>>>>>> Brick4: giant4:/gluster/sdc/gv0 >>>>>>> Brick5: giant5:/gluster/sdc/gv0 >>>>>>> Brick6: giant6:/gluster/sdc/gv0 >>>>>>> Brick7: giant1:/gluster/sdd/gv0 >>>>>>> Brick8: giant2:/gluster/sdd/gv0 >>>>>>> Brick9: giant3:/gluster/sdd/gv0 >>>>>>> Brick10: giant4:/gluster/sdd/gv0 >>>>>>> Brick11: giant5:/gluster/sdd/gv0 >>>>>>> Brick12: giant6:/gluster/sdd/gv0 >>>>>>> Options Reconfigured: >>>>>>> auth.allow: X.X.X.*,127.0.0.1 >>>>>>> nfs.disable: on >>>>>>> >>>>>>> Volume Name: gv2 >>>>>>> Type: Replicate >>>>>>> Volume ID: 30c78928-5f2c-4671-becc-8deaee1a7a8d >>>>>>> Status: Started >>>>>>> Snapshot Count: 0 >>>>>>> Number of Bricks: 1 x 2 = 2 >>>>>>> Transport-type: tcp >>>>>>> Bricks: >>>>>>> Brick1: giant1:/gluster/sdd/gv2 >>>>>>> Brick2: giant2:/gluster/sdd/gv2 >>>>>>> Options Reconfigured: >>>>>>> auth.allow: X.X.X.*,127.0.0.1 >>>>>>> cluster.granular-entry-heal: on >>>>>>> cluster.locking-scheme: granular >>>>>>> nfs.disable: on >>>>>>> >>>>>>> >>>>>>> 2016-11-29 19:21 GMT+01:00 Micha Ober <[email protected] >>>>>>> <mailto:[email protected]>>: >>>>>>> >>>>>>> I had opened another thread on this mailing list >>>>>>> (Subject: "After upgrade from 3.4.2 to 3.8.5 - High >>>>>>> CPU usage resulting in disconnects and split-brain"). >>>>>>> >>>>>>> The title may be a bit misleading now, as I am no >>>>>>> longer observing high CPU usage after upgrading to >>>>>>> 3.8.6, but the disconnects are still happening and >>>>>>> the number of files in split-brain is growing. >>>>>>> >>>>>>> Setup: 6 compute nodes, each serving as a glusterfs >>>>>>> server and client, Ubuntu 14.04, two bricks per >>>>>>> node, distribute-replicate >>>>>>> >>>>>>> I have two gluster volumes set up (one for scratch >>>>>>> data, one for the slurm scheduler). Only the scratch >>>>>>> data volume shows critical errors "[...] has not >>>>>>> responded in the last 42 seconds, disconnecting.". >>>>>>> So I can rule out network problems, the gigabit link >>>>>>> between the nodes is not saturated at all. The disks >>>>>>> are almost idle (<10%). >>>>>>> >>>>>>> I have glusterfs 3.4.2 on Ubuntu 12.04 on a another >>>>>>> compute cluster, running fine since it was deployed. >>>>>>> I had glusterfs 3.4.2 on Ubuntu 14.04 on this >>>>>>> cluster, running fine for almost a year. >>>>>>> >>>>>>> After upgrading to 3.8.5, the problems (as >>>>>>> described) started. I would like to use some of the >>>>>>> new features of the newer versions (like bitrot), >>>>>>> but the users can't run their compute jobs right now >>>>>>> because the result files are garbled. >>>>>>> >>>>>>> 2016-11-29 18:53 GMT+01:00 Atin Mukherjee >>>>>>> <[email protected] <mailto:[email protected]>>: >>>>>>> >>>>>>> Would you be able to share what is not working >>>>>>> for you in 3.8.x (mention the exact version). >>>>>>> 3.4 is quite old and falling back to an >>>>>>> unsupported version doesn't look a feasible option. >>>>>>> >>>>>>> On Tue, 29 Nov 2016 at 17:01, Micha Ober >>>>>>> <[email protected] <mailto:[email protected]>> >>>>>>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I was using gluster 3.4 and upgraded to 3.8, >>>>>>> but that version showed to be unusable for >>>>>>> me. I now need to downgrade. >>>>>>> >>>>>>> I'm running Ubuntu 14.04. As upgrades of the >>>>>>> op version are irreversible, I guess I have >>>>>>> to delete all gluster volumes and re-create >>>>>>> them with the downgraded version. >>>>>>> >>>>>>> 0. Backup data >>>>>>> 1. Unmount all gluster volumes >>>>>>> 2. apt-get purge glusterfs-server >>>>>>> glusterfs-client >>>>>>> 3. Remove PPA for 3.8 >>>>>>> 4. Add PPA for older version >>>>>>> 5. apt-get install glusterfs-server >>>>>>> glusterfs-client >>>>>>> 6. Create volumes >>>>>>> >>>>>>> Is "purge" enough to delete all >>>>>>> configuration files of the currently >>>>>>> installed version or do I need to manually >>>>>>> clear some residues before installing an >>>>>>> older version? >>>>>>> >>>>>>> Thanks. >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> [email protected] >>>>>>> <mailto:[email protected]> >>>>>>> >>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>>> <http://www.gluster.org/mailman/listinfo/gluster-users> >>>>>>> >>>>>>> -- >>>>>>> - Atin (atinm) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> [email protected] <mailto:[email protected]> >>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>>>> <http://www.gluster.org/mailman/listinfo/gluster-users> >>>>> > -- > ~ Atin (atinm)
_______________________________________________ Gluster-users mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-users
