Re: [ceph-users] How can we repair OSD leveldb?

2016-08-18 Thread Sean Sullivan
We have a hammer cluster that experienced a similar power failure and ended up corrupting our monitors leveldb stores. I am still trying to repair ours but I can give you a few tips that seem to help. 1.) I would copy the database off to somewhere safe right away. Just opening it seems to change

Re: [ceph-users] How can we repair OSD leveldb?

2016-08-18 Thread Wido den Hollander
> Op 17 augustus 2016 om 23:54 schreef Dan Jakubiec : > > > Hi Wido, > > Thank you for the response: > > > On Aug 17, 2016, at 16:25, Wido den Hollander wrote: > > > > > >> Op 17 augustus 2016 om 17:44 schreef Dan Jakubiec : >

Re: [ceph-users] How can we repair OSD leveldb?

2016-08-17 Thread Christian Balzer
Hello, On Wed, 17 Aug 2016 16:54:41 -0500 Dan Jakubiec wrote: > Hi Wido, > > Thank you for the response: > > > On Aug 17, 2016, at 16:25, Wido den Hollander wrote: > > > > > >> Op 17 augustus 2016 om 17:44 schreef Dan Jakubiec : > >> > >> > >>

Re: [ceph-users] How can we repair OSD leveldb?

2016-08-17 Thread Dan Jakubiec
Hi Wido, Thank you for the response: > On Aug 17, 2016, at 16:25, Wido den Hollander wrote: > > >> Op 17 augustus 2016 om 17:44 schreef Dan Jakubiec : >> >> >> Hello, we have a Ceph cluster with 8 OSD that recently lost power to all 8 >> machines.

Re: [ceph-users] How can we repair OSD leveldb?

2016-08-17 Thread Wido den Hollander
> Op 17 augustus 2016 om 17:44 schreef Dan Jakubiec : > > > Hello, we have a Ceph cluster with 8 OSD that recently lost power to all 8 > machines. We've managed to recover the XFS filesystems on 7 of the machines, > but the OSD service is only starting on 1 of them. >

[ceph-users] How can we repair OSD leveldb?

2016-08-17 Thread Dan Jakubiec
Hello, we have a Ceph cluster with 8 OSD that recently lost power to all 8 machines. We've managed to recover the XFS filesystems on 7 of the machines, but the OSD service is only starting on 1 of them. The other 5 machines all have complaints similar to the following: 2016-08-17