Re: [ceph-users] ceph-volume activation

[Dan van der Ster]

Hi Oliver,

No ticket yet... we were distracted.

I have the same observations as what you show below...

-- dan



On Tue, Feb 27, 2018 at 2:33 PM, Oliver Freyermuth
 wrote:
> Am 22.02.2018 um 09:44 schrieb Dan van der Ster:
>> On Wed, Feb 21, 2018 at 11:56 PM, Oliver Freyermuth
>>  wrote:
>>> Am 21.02.2018 um 15:58 schrieb Alfredo Deza:
 On Wed, Feb 21, 2018 at 9:40 AM, Dan van der Ster  
 wrote:
> On Wed, Feb 21, 2018 at 2:24 PM, Alfredo Deza  wrote:
>> On Tue, Feb 20, 2018 at 9:05 PM, Oliver Freyermuth
>>  wrote:
>>> Many thanks for your replies!
>>>
>>> Are there plans to have something like
>>> "ceph-volume discover-and-activate"
>>> which would effectively do something like:
>>> ceph-volume list and activate all OSDs which are re-discovered from LVM 
>>> metadata?
>>
>> This is a good idea, I think ceph-disk had an 'activate all', and it
>> would make it easier for the situation you explain with ceph-volume
>>
>> I've created http://tracker.ceph.com/issues/23067 to follow up on this
>> an implement it.
>
> +1 thanks a lot for this thread and clear answers!
> We were literally stuck today not knowing how to restart a ceph-volume
> lvm created OSD.
>
> (It seems that once you systemctl stop ceph-osd@* on a machine, the
> only way to get them back is ceph-volume lvm activate ... )
>
> BTW, ceph-osd.target now has less obvious functionality. For example,
> this works:
>
>   systemctl restart ceph-osd.target
>
> But if you stop ceph-osd.target, then you can no longer start 
> ceph-osd.target.
>
> Is this a regression or something we'll have to live with?

 This sounds surprising. Stopping a ceph-osd target should not do
 anything with the devices. All that 'activate' does when called in
 ceph-volume is to ensure that
 the devices are available and mounted in the right places so that the
 OSD can start.

 If you are experiencing a problem stopping an OSD that can't be
 started again, then something is going on. I would urge you to create
 a ticket with as many details as you can
 at http://tracker.ceph.com/projects/ceph-volume/issues/new
>>>
>>> I also see this - but it's not really that "the osd can not be started 
>>> again".
>>> The problem is that once the osd is stopped, e.g. via
>>> systemctl stop ceph.target
>>> doing a
>>> systemctl start ceph.target
>>> will not bring it up again.
>>>
>>> Doing a manual
>>> systemctl start ceph-osd@36.service
>>> will work, though.
>>
>> In our case even that does not work reliably.
>> We're gathering info to create a tracker ticket.
>>
>> Cheers, Dan
>
> Dear Dan,
>
> did you manage to come around to report a ticket? If so, could you share the 
> ticket number?
> Then I'd happily subscribe to it (with the flood of tickets it's hard to 
> find...).
>
> On related news, I observe this:
>   # systemctl list-dependencies ceph-osd.target
>   ceph-osd.target
>   ● ├─ceph-osd@2.service
>   ● └─ceph-osd@3.service
> on a host installed with ceph-deploy < 2.0 (i.e. using ceph-disk),
> while I observe this:
>   # systemctl list-dependencies ceph-osd.target
>   ceph-osd.target
> for a host installed with ceph-deploy 2.0, i.e. using ceph-volume.
>
> I think this is caused by the ceph-volume systemd-files triggering the 
> ceph-osd services,
> so they are not enabled at all, and hence not considered as dependencies of 
> the target.
>
> Unsure how to solve this cleanly without refactoring the concept, but again, 
> I'm no systemd expert ;-).
>
> Cheers,
> Oliver
>
>>
>>>
>>> The ceph-osd@36.service, in fact, is not enabled on my machine,
>>> which is likely why ceph.target will not cause it to come up.
>>>
>>> I am not a systemd expert, but I think the issue is that the 
>>> ceph-volume@-services which
>>> (I think) take care to activate the OSD services are not re-triggered.
>>>
>>> Cheers,
>>> Oliver
>>>

>
> Cheers, Dan
>>>
>>>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-volume activation

[Oliver Freyermuth]

Am 22.02.2018 um 09:44 schrieb Dan van der Ster:
> On Wed, Feb 21, 2018 at 11:56 PM, Oliver Freyermuth
>  wrote:
>> Am 21.02.2018 um 15:58 schrieb Alfredo Deza:
>>> On Wed, Feb 21, 2018 at 9:40 AM, Dan van der Ster  
>>> wrote:
 On Wed, Feb 21, 2018 at 2:24 PM, Alfredo Deza  wrote:
> On Tue, Feb 20, 2018 at 9:05 PM, Oliver Freyermuth
>  wrote:
>> Many thanks for your replies!
>>
>> Are there plans to have something like
>> "ceph-volume discover-and-activate"
>> which would effectively do something like:
>> ceph-volume list and activate all OSDs which are re-discovered from LVM 
>> metadata?
>
> This is a good idea, I think ceph-disk had an 'activate all', and it
> would make it easier for the situation you explain with ceph-volume
>
> I've created http://tracker.ceph.com/issues/23067 to follow up on this
> an implement it.

 +1 thanks a lot for this thread and clear answers!
 We were literally stuck today not knowing how to restart a ceph-volume
 lvm created OSD.

 (It seems that once you systemctl stop ceph-osd@* on a machine, the
 only way to get them back is ceph-volume lvm activate ... )

 BTW, ceph-osd.target now has less obvious functionality. For example,
 this works:

   systemctl restart ceph-osd.target

 But if you stop ceph-osd.target, then you can no longer start 
 ceph-osd.target.

 Is this a regression or something we'll have to live with?
>>>
>>> This sounds surprising. Stopping a ceph-osd target should not do
>>> anything with the devices. All that 'activate' does when called in
>>> ceph-volume is to ensure that
>>> the devices are available and mounted in the right places so that the
>>> OSD can start.
>>>
>>> If you are experiencing a problem stopping an OSD that can't be
>>> started again, then something is going on. I would urge you to create
>>> a ticket with as many details as you can
>>> at http://tracker.ceph.com/projects/ceph-volume/issues/new
>>
>> I also see this - but it's not really that "the osd can not be started 
>> again".
>> The problem is that once the osd is stopped, e.g. via
>> systemctl stop ceph.target
>> doing a
>> systemctl start ceph.target
>> will not bring it up again.
>>
>> Doing a manual
>> systemctl start ceph-osd@36.service
>> will work, though.
> 
> In our case even that does not work reliably.
> We're gathering info to create a tracker ticket.
> 
> Cheers, Dan

Dear Dan,

did you manage to come around to report a ticket? If so, could you share the 
ticket number?
Then I'd happily subscribe to it (with the flood of tickets it's hard to 
find...). 

On related news, I observe this:
  # systemctl list-dependencies ceph-osd.target
  ceph-osd.target   

  ● ├─ceph-osd@2.service

  ● └─ceph-osd@3.service
on a host installed with ceph-deploy < 2.0 (i.e. using ceph-disk),
while I observe this:
  # systemctl list-dependencies ceph-osd.target
  ceph-osd.target
for a host installed with ceph-deploy 2.0, i.e. using ceph-volume. 

I think this is caused by the ceph-volume systemd-files triggering the ceph-osd 
services,
so they are not enabled at all, and hence not considered as dependencies of the 
target. 

Unsure how to solve this cleanly without refactoring the concept, but again, 
I'm no systemd expert ;-). 

Cheers,
Oliver

> 
>>
>> The ceph-osd@36.service, in fact, is not enabled on my machine,
>> which is likely why ceph.target will not cause it to come up.
>>
>> I am not a systemd expert, but I think the issue is that the 
>> ceph-volume@-services which
>> (I think) take care to activate the OSD services are not re-triggered.
>>
>> Cheers,
>> Oliver
>>
>>>

 Cheers, Dan
>>
>>




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-volume activation

[Dan van der Ster]

On Wed, Feb 21, 2018 at 11:56 PM, Oliver Freyermuth
 wrote:
> Am 21.02.2018 um 15:58 schrieb Alfredo Deza:
>> On Wed, Feb 21, 2018 at 9:40 AM, Dan van der Ster  
>> wrote:
>>> On Wed, Feb 21, 2018 at 2:24 PM, Alfredo Deza  wrote:
 On Tue, Feb 20, 2018 at 9:05 PM, Oliver Freyermuth
  wrote:
> Many thanks for your replies!
>
> Are there plans to have something like
> "ceph-volume discover-and-activate"
> which would effectively do something like:
> ceph-volume list and activate all OSDs which are re-discovered from LVM 
> metadata?

 This is a good idea, I think ceph-disk had an 'activate all', and it
 would make it easier for the situation you explain with ceph-volume

 I've created http://tracker.ceph.com/issues/23067 to follow up on this
 an implement it.
>>>
>>> +1 thanks a lot for this thread and clear answers!
>>> We were literally stuck today not knowing how to restart a ceph-volume
>>> lvm created OSD.
>>>
>>> (It seems that once you systemctl stop ceph-osd@* on a machine, the
>>> only way to get them back is ceph-volume lvm activate ... )
>>>
>>> BTW, ceph-osd.target now has less obvious functionality. For example,
>>> this works:
>>>
>>>   systemctl restart ceph-osd.target
>>>
>>> But if you stop ceph-osd.target, then you can no longer start 
>>> ceph-osd.target.
>>>
>>> Is this a regression or something we'll have to live with?
>>
>> This sounds surprising. Stopping a ceph-osd target should not do
>> anything with the devices. All that 'activate' does when called in
>> ceph-volume is to ensure that
>> the devices are available and mounted in the right places so that the
>> OSD can start.
>>
>> If you are experiencing a problem stopping an OSD that can't be
>> started again, then something is going on. I would urge you to create
>> a ticket with as many details as you can
>> at http://tracker.ceph.com/projects/ceph-volume/issues/new
>
> I also see this - but it's not really that "the osd can not be started again".
> The problem is that once the osd is stopped, e.g. via
> systemctl stop ceph.target
> doing a
> systemctl start ceph.target
> will not bring it up again.
>
> Doing a manual
> systemctl start ceph-osd@36.service
> will work, though.

In our case even that does not work reliably.
We're gathering info to create a tracker ticket.

Cheers, Dan

>
> The ceph-osd@36.service, in fact, is not enabled on my machine,
> which is likely why ceph.target will not cause it to come up.
>
> I am not a systemd expert, but I think the issue is that the 
> ceph-volume@-services which
> (I think) take care to activate the OSD services are not re-triggered.
>
> Cheers,
> Oliver
>
>>
>>>
>>> Cheers, Dan
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-volume activation

[Oliver Freyermuth]

Am 21.02.2018 um 14:24 schrieb Alfredo Deza:
[snip]
>> Are there plans to have something like
>> "ceph-volume discover-and-activate"
>> which would effectively do something like:
>> ceph-volume list and activate all OSDs which are re-discovered from LVM 
>> metadata?
> 
> This is a good idea, I think ceph-disk had an 'activate all', and it
> would make it easier for the situation you explain with ceph-volume
> 
> I've created http://tracker.ceph.com/issues/23067 to follow up on this
> an implement it.

Many thanks for creating the issue, and also thanks for the extensive and clear 
reply! 

In case somebody finds it useful, I am right now using the following 
incantation:

lvs -o lv_tags | awk -vFS=, /ceph.osd_fsid/'{ 
OSD_ID=gensub(".*ceph.osd_id=([0-9]+),.*", "\\1", ""); 
OSD_FSID=gensub(".*ceph.osd_fsid=([a-z0-9-]+),.*", "\\1", ""); print 
OSD_ID,OSD_FSID }' | sort -n | uniq

which runs very fast and directly returns the parameters to be consumed by 
"ceph-volume lvm activate" (i.e. OSD-ID and OSD-FSID). 
Or course, use that at your own risk until a good implementation in ceph-volume 
is available. 

Cheers,
Oliver



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-volume activation

[Oliver Freyermuth]

Am 21.02.2018 um 15:58 schrieb Alfredo Deza:
> On Wed, Feb 21, 2018 at 9:40 AM, Dan van der Ster  wrote:
>> On Wed, Feb 21, 2018 at 2:24 PM, Alfredo Deza  wrote:
>>> On Tue, Feb 20, 2018 at 9:05 PM, Oliver Freyermuth
>>>  wrote:
 Many thanks for your replies!

 Are there plans to have something like
 "ceph-volume discover-and-activate"
 which would effectively do something like:
 ceph-volume list and activate all OSDs which are re-discovered from LVM 
 metadata?
>>>
>>> This is a good idea, I think ceph-disk had an 'activate all', and it
>>> would make it easier for the situation you explain with ceph-volume
>>>
>>> I've created http://tracker.ceph.com/issues/23067 to follow up on this
>>> an implement it.
>>
>> +1 thanks a lot for this thread and clear answers!
>> We were literally stuck today not knowing how to restart a ceph-volume
>> lvm created OSD.
>>
>> (It seems that once you systemctl stop ceph-osd@* on a machine, the
>> only way to get them back is ceph-volume lvm activate ... )
>>
>> BTW, ceph-osd.target now has less obvious functionality. For example,
>> this works:
>>
>>   systemctl restart ceph-osd.target
>>
>> But if you stop ceph-osd.target, then you can no longer start 
>> ceph-osd.target.
>>
>> Is this a regression or something we'll have to live with?
> 
> This sounds surprising. Stopping a ceph-osd target should not do
> anything with the devices. All that 'activate' does when called in
> ceph-volume is to ensure that
> the devices are available and mounted in the right places so that the
> OSD can start.
> 
> If you are experiencing a problem stopping an OSD that can't be
> started again, then something is going on. I would urge you to create
> a ticket with as many details as you can
> at http://tracker.ceph.com/projects/ceph-volume/issues/new

I also see this - but it's not really that "the osd can not be started again". 
The problem is that once the osd is stopped, e.g. via
systemctl stop ceph.target
doing a
systemctl start ceph.target
will not bring it up again. 

Doing a manual
systemctl start ceph-osd@36.service
will work, though. 

The ceph-osd@36.service, in fact, is not enabled on my machine,
which is likely why ceph.target will not cause it to come up. 

I am not a systemd expert, but I think the issue is that the 
ceph-volume@-services which
(I think) take care to activate the OSD services are not re-triggered. 

Cheers,
Oliver

> 
>>
>> Cheers, Dan




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-volume activation

[Alfredo Deza]

On Wed, Feb 21, 2018 at 9:40 AM, Dan van der Ster  wrote:
> On Wed, Feb 21, 2018 at 2:24 PM, Alfredo Deza  wrote:
>> On Tue, Feb 20, 2018 at 9:05 PM, Oliver Freyermuth
>>  wrote:
>>> Many thanks for your replies!
>>>
>>> Are there plans to have something like
>>> "ceph-volume discover-and-activate"
>>> which would effectively do something like:
>>> ceph-volume list and activate all OSDs which are re-discovered from LVM 
>>> metadata?
>>
>> This is a good idea, I think ceph-disk had an 'activate all', and it
>> would make it easier for the situation you explain with ceph-volume
>>
>> I've created http://tracker.ceph.com/issues/23067 to follow up on this
>> an implement it.
>
> +1 thanks a lot for this thread and clear answers!
> We were literally stuck today not knowing how to restart a ceph-volume
> lvm created OSD.
>
> (It seems that once you systemctl stop ceph-osd@* on a machine, the
> only way to get them back is ceph-volume lvm activate ... )
>
> BTW, ceph-osd.target now has less obvious functionality. For example,
> this works:
>
>   systemctl restart ceph-osd.target
>
> But if you stop ceph-osd.target, then you can no longer start ceph-osd.target.
>
> Is this a regression or something we'll have to live with?

This sounds surprising. Stopping a ceph-osd target should not do
anything with the devices. All that 'activate' does when called in
ceph-volume is to ensure that
the devices are available and mounted in the right places so that the
OSD can start.

If you are experiencing a problem stopping an OSD that can't be
started again, then something is going on. I would urge you to create
a ticket with as many details as you can
at http://tracker.ceph.com/projects/ceph-volume/issues/new

>
> Cheers, Dan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-volume activation

[Dan van der Ster]

On Wed, Feb 21, 2018 at 2:24 PM, Alfredo Deza  wrote:
> On Tue, Feb 20, 2018 at 9:05 PM, Oliver Freyermuth
>  wrote:
>> Many thanks for your replies!
>>
>> Are there plans to have something like
>> "ceph-volume discover-and-activate"
>> which would effectively do something like:
>> ceph-volume list and activate all OSDs which are re-discovered from LVM 
>> metadata?
>
> This is a good idea, I think ceph-disk had an 'activate all', and it
> would make it easier for the situation you explain with ceph-volume
>
> I've created http://tracker.ceph.com/issues/23067 to follow up on this
> an implement it.

+1 thanks a lot for this thread and clear answers!
We were literally stuck today not knowing how to restart a ceph-volume
lvm created OSD.

(It seems that once you systemctl stop ceph-osd@* on a machine, the
only way to get them back is ceph-volume lvm activate ... )

BTW, ceph-osd.target now has less obvious functionality. For example,
this works:

  systemctl restart ceph-osd.target

But if you stop ceph-osd.target, then you can no longer start ceph-osd.target.

Is this a regression or something we'll have to live with?

Cheers, Dan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-volume activation

[Alfredo Deza]

On Tue, Feb 20, 2018 at 9:05 PM, Oliver Freyermuth
 wrote:
> Many thanks for your replies!
>
> Am 21.02.2018 um 02:20 schrieb Alfredo Deza:
>> On Tue, Feb 20, 2018 at 5:56 PM, Oliver Freyermuth
>>  wrote:
>>> Dear Cephalopodians,
>>>
>>> with the release of ceph-deploy we are thinking about migrating our 
>>> Bluestore-OSDs (currently created with ceph-disk via old ceph-deploy)
>>> to be created via ceph-volume (with LVM).
>>
>> When you say migrating, do you mean creating them again from scratch
>> or making ceph-volume take over the previously created OSDs
>> (ceph-volume can do both)
>
> I would recreate from scratch to switch to LVM, we have a k=4 m=2 EC-pool 
> with 6 hosts, so I can just take down a full host and recreate.
> But good to know both would work!
>
>>
>>>
>>> I note two major changes:
>>> 1. It seems the block.db partitions have to be created beforehand, manually.
>>>With ceph-disk, one should not do that - or manually set the correct 
>>> PARTTYPE ID.
>>>Will ceph-volume take care of setting the PARTTYPE on existing 
>>> partitions for block.db now?
>>>Is it not necessary anymore?
>>>Is the config option bluestore_block_db_size now also obsoleted?
>>
>> Right, ceph-volume will not create any partitions for you, so no, it
>> will not take care of setting PARTTYPE either. If your setup requires
>> a block.db, then this must be
>> created beforehand and then passed onto ceph-volume. The one
>> requirement if it is a partition is to have a PARTUUID. For logical
>> volumes it can just work as-is. This is
>> explained in detail at
>> http://docs.ceph.com/docs/master/ceph-volume/lvm/prepare/#bluestore
>>
>> PARTUUID information for ceph-volume at:
>> http://docs.ceph.com/docs/master/ceph-volume/lvm/prepare/#partitioning
>
> Ok.
> So do I understand correctly that the PARTTYPE setting (i.e. those magic 
> numbers as found e.g. in ceph-disk sources in PTYPE:
> https://github.com/ceph/ceph/blob/master/src/ceph-disk/ceph_disk/main.py#L62 )
> is not needed anymore for the block.db partitions, since it was effectively 
> only there
> to have udev work?

Right, the PARTTYPE was only for udev. We need the PARTUUID to ensure
that we can find the right device/partition always (in the case of
partitions only)

>
> I remember from ceph-disk that if I created the block.db partition beforehand 
> and without setting the magic PARTTYPE,
> it would become unhappy.
> ceph-volume and the systemd activation path should not care at all if I 
> understand this correctly.

Right again, this was part of the complex set of things that a
partition had to have in order for ceph-disk to work. A lot of users
thought that the partition approach was simple enough,
but without being aware that a lot of extra things were needed for
those partitions to be recognized by ceph-disk

>
> So in short, to create a new OSD, steps for me would be:
> - Create block.db partition (and don't care about PARTTYPE).
>   I do only have to make sure it has a PARTUUID.
> - ceph-volume lvm create --bluestore --block.db /dev/sdag1 --data /dev/sda
>   (or the same via ceph-deploy)

That would work, yes. When you pass a whole device to --data in that
example, ceph-volume will create a whole volume group and logical
volume out of that device and use it for bluestore. That
may or may not be what you want though. With LVM you can chop that
device in many pieces and use what you want. That "shim" in
ceph-volume is there to allow users that don't care about this to just
move forward with a whole device.

Similarly, if you want to use a logical volume for --block.db, you can.

To recap: yes, your example would work, but you have a lot of other
options if you need more flexibility.

>
>
>>>
>>> 2. Activation does not work via udev anymore, which solves some racy things.
>>>
>>> This second major change makes me curious: How does activation work now?
>>> In the past, I could reinstall the full OS, install ceph packages, trigger 
>>> udev / reboot and all OSDs would come back,
>>> without storing any state / activating any services in the OS.
>>
>> Activation works via systemd. This is explained in detail here
>> http://docs.ceph.com/docs/master/ceph-volume/lvm/activate
>>
>> Nothing with `ceph-volume lvm` requires udev for discovery. If you
>> need to re-install the OS and recover your OSDs all you need to do is
>> to
>> re-activate them. You would need to know what the ID and UUID of the OSDs is.
>>
>> If you don't have that information handy, you can run:
>>
>> ceph-volume lvm list
>>
>> And all the information will be available. This will persist even on
>> system re-installs
>
> Understood - so indeed the manual step would be to run "list" and then 
> activate the OSDs one-by-one
> to re-create the service files.
> More cumbersome than letting udev do it's thing, but it certainly gives more 
> control,
> so it seems preferrable.
>
> Are there plans to have something like

Re: [ceph-users] ceph-volume activation

[Oliver Freyermuth]

Many thanks for your replies! 

Am 21.02.2018 um 02:20 schrieb Alfredo Deza:
> On Tue, Feb 20, 2018 at 5:56 PM, Oliver Freyermuth
>  wrote:
>> Dear Cephalopodians,
>>
>> with the release of ceph-deploy we are thinking about migrating our 
>> Bluestore-OSDs (currently created with ceph-disk via old ceph-deploy)
>> to be created via ceph-volume (with LVM).
> 
> When you say migrating, do you mean creating them again from scratch
> or making ceph-volume take over the previously created OSDs
> (ceph-volume can do both)

I would recreate from scratch to switch to LVM, we have a k=4 m=2 EC-pool with 
6 hosts, so I can just take down a full host and recreate. 
But good to know both would work! 

> 
>>
>> I note two major changes:
>> 1. It seems the block.db partitions have to be created beforehand, manually.
>>With ceph-disk, one should not do that - or manually set the correct 
>> PARTTYPE ID.
>>Will ceph-volume take care of setting the PARTTYPE on existing partitions 
>> for block.db now?
>>Is it not necessary anymore?
>>Is the config option bluestore_block_db_size now also obsoleted?
> 
> Right, ceph-volume will not create any partitions for you, so no, it
> will not take care of setting PARTTYPE either. If your setup requires
> a block.db, then this must be
> created beforehand and then passed onto ceph-volume. The one
> requirement if it is a partition is to have a PARTUUID. For logical
> volumes it can just work as-is. This is
> explained in detail at
> http://docs.ceph.com/docs/master/ceph-volume/lvm/prepare/#bluestore
> 
> PARTUUID information for ceph-volume at:
> http://docs.ceph.com/docs/master/ceph-volume/lvm/prepare/#partitioning

Ok. 
So do I understand correctly that the PARTTYPE setting (i.e. those magic 
numbers as found e.g. in ceph-disk sources in PTYPE:
https://github.com/ceph/ceph/blob/master/src/ceph-disk/ceph_disk/main.py#L62 )
is not needed anymore for the block.db partitions, since it was effectively 
only there
to have udev work?

I remember from ceph-disk that if I created the block.db partition beforehand 
and without setting the magic PARTTYPE,
it would become unhappy. 
ceph-volume and the systemd activation path should not care at all if I 
understand this correctly. 

So in short, to create a new OSD, steps for me would be:
- Create block.db partition (and don't care about PARTTYPE). 
  I do only have to make sure it has a PARTUUID. 
- ceph-volume lvm create --bluestore --block.db /dev/sdag1 --data /dev/sda
  (or the same via ceph-deploy)


>>
>> 2. Activation does not work via udev anymore, which solves some racy things.
>>
>> This second major change makes me curious: How does activation work now?
>> In the past, I could reinstall the full OS, install ceph packages, trigger 
>> udev / reboot and all OSDs would come back,
>> without storing any state / activating any services in the OS.
> 
> Activation works via systemd. This is explained in detail here
> http://docs.ceph.com/docs/master/ceph-volume/lvm/activate
> 
> Nothing with `ceph-volume lvm` requires udev for discovery. If you
> need to re-install the OS and recover your OSDs all you need to do is
> to
> re-activate them. You would need to know what the ID and UUID of the OSDs is.
> 
> If you don't have that information handy, you can run:
> 
> ceph-volume lvm list
> 
> And all the information will be available. This will persist even on
> system re-installs

Understood - so indeed the manual step would be to run "list" and then activate 
the OSDs one-by-one
to re-create the service files. 
More cumbersome than letting udev do it's thing, but it certainly gives more 
control,
so it seems preferrable. 

Are there plans to have something like
"ceph-volume discover-and-activate" 
which would effectively do something like:
ceph-volume list and activate all OSDs which are re-discovered from LVM 
metadata? 

This would largely simplify OS reinstalls (otherwise I'll likely write a small 
shell script to do exactly that),
and as far as I understand, activating an already activated OSD should be 
harmless (it should only re-enable
an already enabled service file). 

> 
>>
>> Does this still work?
>> Or is there a manual step needed to restore the ceph-osd@ID-UUID services 
>> which at first glance appear to store state (namely ID and UUID)?
> 
> The manual step would be to call activate as described here
> http://docs.ceph.com/docs/master/ceph-volume/lvm/activate/#new-osds
>>
>> If that's the case:
>> - What is this magic manual step?
> 
> Linked above
> 
>> - Is it still possible to flip two disks within the same OSD host without 
>> issues?
> 
> What do you mean by "flip" ?

Sorry, I was unclear on this. I meant exchanging two harddrives with each other 
within a single OSD host,
e.g. /dev/sda => /dev/sdc and /dev/sdc => /dev/sda (for controller weirdness or 
whatever reason). 
If I understand correctly, this should not be a problem at all, since OSD-ID 
and PARTUUID are unaffected 

Re: [ceph-users] ceph-volume activation

[Alfredo Deza]

On Tue, Feb 20, 2018 at 5:56 PM, Oliver Freyermuth
 wrote:
> Dear Cephalopodians,
>
> with the release of ceph-deploy we are thinking about migrating our 
> Bluestore-OSDs (currently created with ceph-disk via old ceph-deploy)
> to be created via ceph-volume (with LVM).

When you say migrating, do you mean creating them again from scratch
or making ceph-volume take over the previously created OSDs
(ceph-volume can do both)

>
> I note two major changes:
> 1. It seems the block.db partitions have to be created beforehand, manually.
>With ceph-disk, one should not do that - or manually set the correct 
> PARTTYPE ID.
>Will ceph-volume take care of setting the PARTTYPE on existing partitions 
> for block.db now?
>Is it not necessary anymore?
>Is the config option bluestore_block_db_size now also obsoleted?

Right, ceph-volume will not create any partitions for you, so no, it
will not take care of setting PARTTYPE either. If your setup requires
a block.db, then this must be
created beforehand and then passed onto ceph-volume. The one
requirement if it is a partition is to have a PARTUUID. For logical
volumes it can just work as-is. This is
explained in detail at
http://docs.ceph.com/docs/master/ceph-volume/lvm/prepare/#bluestore

PARTUUID information for ceph-volume at:
http://docs.ceph.com/docs/master/ceph-volume/lvm/prepare/#partitioning
>
> 2. Activation does not work via udev anymore, which solves some racy things.
>
> This second major change makes me curious: How does activation work now?
> In the past, I could reinstall the full OS, install ceph packages, trigger 
> udev / reboot and all OSDs would come back,
> without storing any state / activating any services in the OS.

Activation works via systemd. This is explained in detail here
http://docs.ceph.com/docs/master/ceph-volume/lvm/activate

Nothing with `ceph-volume lvm` requires udev for discovery. If you
need to re-install the OS and recover your OSDs all you need to do is
to
re-activate them. You would need to know what the ID and UUID of the OSDs is.

If you don't have that information handy, you can run:

ceph-volume lvm list

And all the information will be available. This will persist even on
system re-installs

>
> Does this still work?
> Or is there a manual step needed to restore the ceph-osd@ID-UUID services 
> which at first glance appear to store state (namely ID and UUID)?

The manual step would be to call activate as described here
http://docs.ceph.com/docs/master/ceph-volume/lvm/activate/#new-osds
>
> If that's the case:
> - What is this magic manual step?

Linked above

> - Is it still possible to flip two disks within the same OSD host without 
> issues?

What do you mean by "flip" ?

>   I would guess so, since the services would detect the disk in the 
> ceph-volume trigger phase.
> - Is it still possible to take a disk from one OSD host, and put it in 
> another one, or does this now need a manual interaction?
>   With ceph-disk / udev, it did not, since udev triggered disk activation and 
> then the service was created at runtime.

It is technically possible, the lvm part of it was built with this in
mind. The LVM metadata will persist with the device, so this is not a
problem. Just manual activation would be needed.

>
> Many thanks for your help and cheers,
> Oliver
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com