Thank you for the acknowledgement. On Thu, Aug 31, 2017 at 8:30 PM, mohammad kashif <[email protected]> wrote:
> Hi Atin > > Thanks, I was not running any script or gluster command. But now gluster > status command started working. CPU usage also came down and looking at the > ganglia graph, cpu usage is strongly correlated with network activity. > > It may be due to high load that status command failed. I will keep an eye > and see whether it happens again. > > Cheers > > Kashif > > On Thu, Aug 31, 2017 at 2:40 AM, Atin Mukherjee <[email protected]> > wrote: > >> >> On Wed, 30 Aug 2017 at 20:55, mohammad kashif <[email protected]> >> wrote: >> >>> Hi >>> >>> I am running a 400TB five node purely distributed gluster setup. I am >>> troubleshooting an issue where some times files creation fails. I found >>> that volume status is not working >>> >>> gluster volume status >>> Another transaction is in progress for atlasglust. Please try again >>> after sometime. >>> >>> When I tried from other node then it seems two nodes have Locking issue >>> >>> gluster volume status >>> Locking failed on pplxgluster01... Please check log file for details. >>> Locking failed on pplxgluster04... Please check log file for details. >>> >> >> This suggests that there are concurrent gluster cli operations been >> performed on the same volume. Are you monitoring the cluster through nagios >> or you have a script on all the nodes which checks for volume's health in a >> period of interval? Please note glusterd will process one cli operation on >> a volume at one time, the rest all transactions on the same volume will be >> failed. >> >> >> >>> >>> Also noticed that glusterfsd process is using around 1000% cpu usage. >>> It is a decent server with 16 core and 64GB RAM. >>> >>> Gluster version is 3.11.2-1 >>> >>> Can you please suggest that how to troubleshoot further? >>> >>> Thanks >>> >>> Kashif >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> [email protected] >>> http://lists.gluster.org/mailman/listinfo/gluster-users >> >> -- >> - Atin (atinm) >> > >
_______________________________________________ Gluster-users mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-users
