Hi,
Today we suddenly experienced a performance drop in gluster: e.g. doing an "ls" of a directory with about 20 files takes about 5 minutes. This is way beyond (and seem separate from) some previous concerns we had.
Our gluster filesystem is two bricks hosted on a single server. Logging onto that server and doing "top" shows a load average of ~30. In general, no process is showing significant CPU usage except an occasional flash a 3300% from glusterfsd. The rest of our system is not doing any exceptional data demands on the file system (i.e. we aren't suddenly running more jobs than we were yesterday).
Any thoughts on how we can proceed with debugging this will be greatly appreciated.
Some additional information: glusterfs 3.7.11 built on Apr 27 2016 14:09:22 CentOS release 6.8 (Final) [root@mseas-data2 ~]# gluster volume status data-volume Status of volume: data-volume Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick mseas-data2:/mnt/brick1 49154 0 Y 5021 Brick mseas-data2:/mnt/brick2 49155 0 Y 5026 Task Status of Volume data-volume ------------------------------------------------------------------------------ Task : Rebalance ID : 892d9e3a-b38c-4971-b96a-8e4a496685ba Status : completed [root@mseas-data2 ~]# gluster volume info data-volume Volume Name: data-volume Type: Distribute Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: mseas-data2:/mnt/brick1 Brick2: mseas-data2:/mnt/brick2 Options Reconfigured: diagnostics.brick-sys-log-level: WARNING performance.readdir-ahead: on nfs.disable: on nfs.export-volumes: off -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: [email protected] Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 _______________________________________________ Gluster-users mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-users
