Re: [ceph-users] syslog broke my cluster

2016-07-28 Thread Sergio A. de Carvalho Jr.
We tracked the problem down to the following rsyslog configuration in our test cluster: *.* @@: $ActionExecOnlyWhenPreviousIsSuspended on & /var/log/failover.log $ActionExecOnlyWhenPreviousIsSuspended off It seems that the $ActionExecOnlyWhenPreviousIsSuspended directive doesn't work well with

Re: [ceph-users] syslog broke my cluster

2016-07-27 Thread Karsten Heymann
Hi, The syslog socket will block if it can't deliver it's logs. This happens for example if logs are forwarded to a remote loghost via tcp and the remote server becomes unavailable. Best Karsten ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] syslog broke my cluster

2016-07-27 Thread Sergio A. de Carvalho Jr.
I guess the point I was trying to make is that, ideally, Ceph would isolate its logging system in a way that a problem with writing the logs wouldn't affect the operation of the core Ceph services. In my case, all other services running on the machine (ssh, ntp, cron, etc.) are operating

Re: [ceph-users] syslog broke my cluster

2016-07-26 Thread Brad Hubbard
On Tue, Jul 26, 2016 at 03:48:33PM +0100, Sergio A. de Carvalho Jr. wrote: > As per my previous messages on the list, I was having a strange problem in > my test cluster (Hammer 0.94.6, CentOS 6.5) where my monitors were > literally crawling to a halt, preventing them to ever reach quorum and >

[ceph-users] syslog broke my cluster

2016-07-26 Thread Sergio A. de Carvalho Jr.
As per my previous messages on the list, I was having a strange problem in my test cluster (Hammer 0.94.6, CentOS 6.5) where my monitors were literally crawling to a halt, preventing them to ever reach quorum and causing all sort of problems. As it turned out, to my surprise everything went back