https://bugzilla.redhat.com/show_bug.cgi?id=1491126
Nigel Babu <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2017-09-14 11:16:35 --- Comment #4 from Nigel Babu <[email protected]> --- This was a lovely issue. The entire RAM was filled with rouge gluster processes that looked like this on ps ax: 30301 ? Ssl 2:05 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy194 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy194.pid -S /var/run/gluster/a4855a3707133667e741a8b25f777ba6.socket --brick-name /d/backends/patchy194 -l /var/log/glusterfs/bricks/d-backends-patchy194.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49345 --xlator-option patchy-server.listen-port=49345 30322 ? Ssl 2:04 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy195 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy195.pid -S /var/run/gluster/06c756cd2a900bafabc3a2cdfec9f252.socket --brick-name /d/backends/patchy195 -l /var/log/glusterfs/bricks/d-backends-patchy195.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49346 --xlator-option patchy-server.listen-port=49346 30343 ? Ssl 2:06 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy196 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy196.pid -S /var/run/gluster/1f718d220e02fef7d8c75178aaba87db.socket --brick-name /d/backends/patchy196 -l /var/log/glusterfs/bricks/d-backends-patchy196.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49347 --xlator-option patchy-server.listen-port=49347 30364 ? Ssl 2:05 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy197 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy197.pid -S /var/run/gluster/9c477977bd939896de48eb8256bcea6d.socket --brick-name /d/backends/patchy197 -l /var/log/glusterfs/bricks/d-backends-patchy197.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49348 --xlator-option patchy-server.listen-port=49348 30385 ? Ssl 2:06 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy198 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy198.pid -S /var/run/gluster/2f1edccbaa1bd898a747124c9ae89916.socket --brick-name /d/backends/patchy198 -l /var/log/glusterfs/bricks/d-backends-patchy198.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49349 --xlator-option patchy-server.listen-port=49349 30406 ? Ssl 2:05 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy199 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy199.pid -S /var/run/gluster/ded1419c8068570d94d7a2473dbbef38.socket --brick-name /d/backends/patchy199 -l /var/log/glusterfs/bricks/d-backends-patchy199.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49350 --xlator-option patchy-server.listen-port=49350 30427 ? Ssl 2:06 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy200 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy200.pid -S /var/run/gluster/4cd50c30298df50845c505b386973409.socket --brick-name /d/backends/patchy200 -l /var/log/glusterfs/bricks/d-backends-patchy200.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49351 --xlator-option patchy-server.listen-port=49351 30448 ? Ssl 2:05 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy201 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy201.pid -S /var/run/gluster/ff3242ef1538df4f31d0ba570496a62f.socket --brick-name /d/backends/patchy201 -l /var/log/glusterfs/bricks/d-backends-patchy201.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49352 --xlator-option patchy-server.listen-port=49352 30469 ? Ssl 2:05 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy202 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy202.pid -S /var/run/gluster/b2bcbc8daa903c74174e1c7b126c6b9b.socket --brick-name /d/backends/patchy202 -l /var/log/glusterfs/bricks/d-backends-patchy202.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49353 --xlator-option patchy-server.listen-port=49353 30490 ? Ssl 2:05 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy203 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy203.pid -S /var/run/gluster/37c7b919cf70a79744a339b5576762ac.socket --brick-name /d/backends/patchy203 -l /var/log/glusterfs/bricks/d-backends-patchy203.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49354 --xlator-option patchy-server.listen-port=49354 30511 ? Ssl 2:06 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy204 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy204.pid -S /var/run/gluster/7e939e1e271855d5657aaaf10471117a.socket --brick-name /d/backends/patchy204 -l /var/log/glusterfs/bricks/d-backends-patchy204.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49355 --xlator-option patchy-server.listen-port=49355 30532 ? Ssl 2:05 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy205 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy205.pid -S /var/run/gluster/d83845a6167892ec46adc9c4849a754d.socket --brick-name /d/backends/patchy205 -l /var/log/glusterfs/bricks/d-backends-patchy205.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49356 --xlator-option patchy-server.listen-port=49356 30553 ? Ssl 2:05 /build/install/sbin/glusterfsd -s slave1.cloud.gluster.org --volfile-id patchy.slave1.cloud.gluster.org.d-backends-patchy206 -p /var/run/gluster/vols/patchy/slave1.cloud.gluster.org-d- backends-patchy206.pid -S /var/run/gluster/060cb1f6d4c375a6b6e42b48c8cb7eab.socket --brick-name /d/backends/patchy206 -l /var/log/glusterfs/bricks/d-backends-patchy206.log --xlator-option *-posix.glusterd-uuid=a ece481c-aa10-493f-94c9-1b5e7306e1b3 --process-name brick --brick-port 49357 --xlator-option patchy-server.listen-port=49357 There was so little memory that java kept getting hit with OOMKiller. I did a pkill on gluster processes and restarted the machine to be completely sure things were normal. -- You are receiving this mail because: You are on the CC list for the bug. Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=nY7PoxQyKE&a=cc_unsubscribe _______________________________________________ Gluster-infra mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-infra
