Re: [ceph-users] HEALTH WARN: clock skew detected

2013-05-07 Thread Joao Eduardo Luis
On 05/06/2013 01:07 PM, Michael Lowe wrote: Um, start it? You must have synchronized clocks in a fault tolerant system (google Byzantine generals clock) and the way to do that is ntp, therefore ntp is required. On May 6, 2013, at 1:34 AM, Varun Chandramouli varun@gmail.com wrote: Hi

Re: [ceph-users] HEALTH WARN: clock skew detected

2013-05-07 Thread Varun Chandramouli
Hi All, Thanks for the replies. I started the ntp daemon and the warnings as well as the crashes seem to have gone. This is the first time I set up a cluster (of physical machines), and was unaware of the need to synchronize the clocks. Probably should have googled it more :). Pardon my

Re: [ceph-users] HEALTH WARN: clock skew detected

2013-05-07 Thread Mike Lowe
You've learned on of the three computer science facts you need to know about distributed systems, and I'm glad I could pass something on: 1. Consistent, Available, Distributed - pick any two 2. To completely guard against k failures where you don't know which one failed just by looking you need

[ceph-users] HEALTH WARN: clock skew detected

2013-05-05 Thread Varun Chandramouli
Hi All, I have a cluster of 4 nodes with 1 mds, 3 mons and 4 osds. Whenever I do ceph health or ceph -s, it shows a health warning saying clock skew detected in 2 of the 3 mons. When I run a mapreduce application on the cluster, one of the monitors crashes (the one in which the skew is not

Re: [ceph-users] HEALTH WARN: clock skew detected

2013-05-05 Thread Michael Lowe
Are you running ntpd? If so you may need to stop, run ntpdate, and restart ntpd. Sometimes if the clock is too far out of sync ntp won't update the time. On May 5, 2013, at 8:52 AM, Varun Chandramouli varun@gmail.com wrote: Hi All, I have a cluster of 4 nodes with 1 mds, 3 mons and 4

Re: [ceph-users] HEALTH WARN: clock skew detected

2013-05-05 Thread Varun Chandramouli
Hi Michael, Thanks for your response. No, the ntp daemon is not running. Any other suggestions? Regards Varun ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com