Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-29 Thread Dan van der Ster
On Thu, Jun 7, 2018 at 8:40 PM Dan van der Ster wrote: > > On Thu, Jun 7, 2018 at 6:33 PM Sage Weil wrote: > > > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > > > Wait, we found something!!! > > > > > > > > > > In the 1st 4k on the block we found the block.db pointing at the wrong > > > >

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Alfredo Deza
On Thu, Jun 7, 2018 at 3:04 PM, Dan van der Ster wrote: > On Thu, Jun 7, 2018 at 8:58 PM Alfredo Deza wrote: >> >> On Thu, Jun 7, 2018 at 2:45 PM, Dan van der Ster wrote: >> > On Thu, Jun 7, 2018 at 6:58 PM Alfredo Deza wrote: >> >> >> >> On Thu, Jun 7, 2018 at 12:09 PM, Sage Weil wrote: >>

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Dan van der Ster
On Thu, Jun 7, 2018 at 8:58 PM Alfredo Deza wrote: > > On Thu, Jun 7, 2018 at 2:45 PM, Dan van der Ster wrote: > > On Thu, Jun 7, 2018 at 6:58 PM Alfredo Deza wrote: > >> > >> On Thu, Jun 7, 2018 at 12:09 PM, Sage Weil wrote: > >> > On Thu, 7 Jun 2018, Dan van der Ster wrote: > >> >> On Thu,

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Alfredo Deza
On Thu, Jun 7, 2018 at 2:45 PM, Dan van der Ster wrote: > On Thu, Jun 7, 2018 at 6:58 PM Alfredo Deza wrote: >> >> On Thu, Jun 7, 2018 at 12:09 PM, Sage Weil wrote: >> > On Thu, 7 Jun 2018, Dan van der Ster wrote: >> >> On Thu, Jun 7, 2018 at 5:36 PM Dan van der Ster >> >> wrote: >> >> > >>

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Dan van der Ster
On Thu, Jun 7, 2018 at 6:58 PM Alfredo Deza wrote: > > On Thu, Jun 7, 2018 at 12:09 PM, Sage Weil wrote: > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > >> On Thu, Jun 7, 2018 at 5:36 PM Dan van der Ster > >> wrote: > >> > > >> > On Thu, Jun 7, 2018 at 5:34 PM Sage Weil wrote: > >> > > > >>

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Dan van der Ster
On Thu, Jun 7, 2018 at 6:33 PM Sage Weil wrote: > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > > Wait, we found something!!! > > > > > > > > In the 1st 4k on the block we found the block.db pointing at the wrong > > > > device (/dev/sdc1 instead of /dev/sdaa1) > > > > > > > > 0130 6b

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Sage Weil
On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > Wait, we found something!!! > > > > > > In the 1st 4k on the block we found the block.db pointing at the wrong > > > device (/dev/sdc1 instead of /dev/sdaa1) > > > > > > 0130 6b 35 79 2b 67 3d 3d 0d 00 00 00 70 61 74 68 5f > > >

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Sage Weil
On Thu, 7 Jun 2018, Dan van der Ster wrote: > On Thu, Jun 7, 2018 at 6:09 PM Sage Weil wrote: > > > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > On Thu, Jun 7, 2018 at 5:36 PM Dan van der Ster > > > wrote: > > > > > > > > On Thu, Jun 7, 2018 at 5:34 PM Sage Weil wrote: > > > > > > > > >

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Dan van der Ster
On Thu, Jun 7, 2018 at 6:09 PM Sage Weil wrote: > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > On Thu, Jun 7, 2018 at 5:36 PM Dan van der Ster wrote: > > > > > > On Thu, Jun 7, 2018 at 5:34 PM Sage Weil wrote: > > > > > > > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > > > On Thu,

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Dan van der Ster
On Thu, Jun 7, 2018 at 6:01 PM Dan van der Ster wrote: > > On Thu, Jun 7, 2018 at 5:36 PM Dan van der Ster wrote: > > > > On Thu, Jun 7, 2018 at 5:34 PM Sage Weil wrote: > > > > > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > > On Thu, Jun 7, 2018 at 4:41 PM Sage Weil wrote: > > > > > >

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Sage Weil
On Thu, 7 Jun 2018, Dan van der Ster wrote: > On Thu, Jun 7, 2018 at 5:36 PM Dan van der Ster wrote: > > > > On Thu, Jun 7, 2018 at 5:34 PM Sage Weil wrote: > > > > > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > > On Thu, Jun 7, 2018 at 4:41 PM Sage Weil wrote: > > > > > > > > > > On

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Dan van der Ster
On Thu, Jun 7, 2018 at 5:36 PM Dan van der Ster wrote: > > On Thu, Jun 7, 2018 at 5:34 PM Sage Weil wrote: > > > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > On Thu, Jun 7, 2018 at 4:41 PM Sage Weil wrote: > > > > > > > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > > > On Thu, Jun

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Dan van der Ster
On Thu, Jun 7, 2018 at 5:34 PM Sage Weil wrote: > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > On Thu, Jun 7, 2018 at 4:41 PM Sage Weil wrote: > > > > > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > > On Thu, Jun 7, 2018 at 4:33 PM Sage Weil wrote: > > > > > > > > > > On Thu, 7 Jun

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Sage Weil
On Thu, 7 Jun 2018, Dan van der Ster wrote: > On Thu, Jun 7, 2018 at 4:41 PM Sage Weil wrote: > > > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > On Thu, Jun 7, 2018 at 4:33 PM Sage Weil wrote: > > > > > > > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > > > Hi all, > > > > > > > > >

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Dan van der Ster
On Thu, Jun 7, 2018 at 5:16 PM Alfredo Deza wrote: > > On Thu, Jun 7, 2018 at 10:54 AM, Dan van der Ster wrote: > > On Thu, Jun 7, 2018 at 4:41 PM Sage Weil wrote: > >> > >> On Thu, 7 Jun 2018, Dan van der Ster wrote: > >> > On Thu, Jun 7, 2018 at 4:33 PM Sage Weil wrote: > >> > > > >> > > On

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Alfredo Deza
On Thu, Jun 7, 2018 at 10:54 AM, Dan van der Ster wrote: > On Thu, Jun 7, 2018 at 4:41 PM Sage Weil wrote: >> >> On Thu, 7 Jun 2018, Dan van der Ster wrote: >> > On Thu, Jun 7, 2018 at 4:33 PM Sage Weil wrote: >> > > >> > > On Thu, 7 Jun 2018, Dan van der Ster wrote: >> > > > Hi all, >> > > >

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Dan van der Ster
On Thu, Jun 7, 2018 at 4:41 PM Sage Weil wrote: > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > On Thu, Jun 7, 2018 at 4:33 PM Sage Weil wrote: > > > > > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > > Hi all, > > > > > > > > We have an intermittent issue where bluestore osds sometimes

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Alfredo Deza
On Thu, Jun 7, 2018 at 10:40 AM, Dan van der Ster wrote: > On Thu, Jun 7, 2018 at 4:31 PM Alfredo Deza wrote: >> >> On Thu, Jun 7, 2018 at 10:23 AM, Dan van der Ster >> wrote: >> > Hi all, >> > >> > We have an intermittent issue where bluestore osds sometimes fail to >> > start after a reboot.

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Sage Weil
On Thu, 7 Jun 2018, Dan van der Ster wrote: > On Thu, Jun 7, 2018 at 4:33 PM Sage Weil wrote: > > > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > > Hi all, > > > > > > We have an intermittent issue where bluestore osds sometimes fail to > > > start after a reboot. > > > The osds all fail the

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Dan van der Ster
On Thu, Jun 7, 2018 at 4:31 PM Alfredo Deza wrote: > > On Thu, Jun 7, 2018 at 10:23 AM, Dan van der Ster wrote: > > Hi all, > > > > We have an intermittent issue where bluestore osds sometimes fail to > > start after a reboot. > > The osds all fail the same way [see 2], failing to open the

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Dan van der Ster
On Thu, Jun 7, 2018 at 4:33 PM Sage Weil wrote: > > On Thu, 7 Jun 2018, Dan van der Ster wrote: > > Hi all, > > > > We have an intermittent issue where bluestore osds sometimes fail to > > start after a reboot. > > The osds all fail the same way [see 2], failing to open the superblock. > > One

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Sage Weil
On Thu, 7 Jun 2018, Dan van der Ster wrote: > Hi all, > > We have an intermittent issue where bluestore osds sometimes fail to > start after a reboot. > The osds all fail the same way [see 2], failing to open the superblock. > One one particular host, there are 24 osds and 4 SSDs partitioned for

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Alfredo Deza
On Thu, Jun 7, 2018 at 10:23 AM, Dan van der Ster wrote: > Hi all, > > We have an intermittent issue where bluestore osds sometimes fail to > start after a reboot. > The osds all fail the same way [see 2], failing to open the superblock. > One one particular host, there are 24 osds and 4 SSDs

[ceph-users] ceph-volume: failed to activate some bluestore osds

2018-06-07 Thread Dan van der Ster
Hi all, We have an intermittent issue where bluestore osds sometimes fail to start after a reboot. The osds all fail the same way [see 2], failing to open the superblock. One one particular host, there are 24 osds and 4 SSDs partitioned for the block.db's. The affected non-starting OSDs all have