Re: [ceph-users] Questions about rbd-mirror and clones

[Jason Dillaman]

On Fri, Mar 1, 2019 at 1:09 PM Anthony D'Atri  wrote:
>
> Thanks!  I'll do some experimentation.  Our orchestration service might need 
> an overhaul to manage parents / children together.
>
> Related question, in this thread:
>
> https://www.spinics.net/lists/ceph-users/msg35257.html
>
> there was discussion of rate limiting.  I have a 4TB migration going right 
> now that's writing at around 900MB/s, so this is of keen interest.

There isn't any rate-limiting available for rbd-mirror as of yet, but
it's on the backlog. The linked email thread talked about the per-pool
configuration overrides, and that will be available in Nautilus via
the new "rbd config global/pool/image ..." commands.

>
> > On Feb 26, 2019, at 5:27 PM, Jason Dillaman  wrote:
> >
> > On Tue, Feb 26, 2019 at 7:49 PM Anthony D'Atri  
> > wrote:
> >>
> >> Hello again.
> >>
> >> I have a couple of questions about rbd-mirror that I'm hoping you can help 
> >> me with.
> >>
> >>
> >> 1) http://docs.ceph.com/docs/mimic/rbd/rbd-snapshot/ indicates that 
> >> protecting is required for cloning.  We somehow had the notion that this 
> >> had been / will be done away with, but don't remember where we saw that.  
> >> Thoughts?
> >
> > By default, if the cluster is configured to to require mimic or later
> > clients, you no longer need to protect/unprotect snapshots prior to
> > cloning [1]. The documentation still talks about
> > protecting/unprotecting snapshots since the new clone v2 format isn't
> > currently enabled by default in order to preserve backwards
> > compatibility to older librbd/krbd clients. Once we no longer support
> > upgrading from pre-Mimic releases, we can enable clone v2 by default
> > and start deprecating snapshot protect/unprotect features.
> >
> >> 2) We're currently running 12.2.2 on our cluster nodes, with rbd-mirror 
> >> running in a container built against 12.2.8.  Should we expect images with 
> >> clones / parents to successfully migrate with rbd-mirror? I've had a few 
> >> rude awakenings here where I've flattened to remove the dependency, but in 
> >> the general case would rather not have to sacrifice the underlying 
> >> capacity.
> >
> > Yes, thinly provisioned cloned images have always been supported with
> > RBD mirroring (Jewel release). You do, however, need to ensure that
> > the parent image has mirroring enabled.
> >
> >>
> >>
> >>
> >> Context:  We aren't using rbd-mirror for DR, we're using it to move 
> >> volumes between clusters for capacity management.
> >>
> >> Hope to see you at Cephalocon.
> >>
> >>
> >>
> >>
> >> Anthony D'Atri
> >> Storage Engineer
> >> 425-343-5133
> >> ada...@digitalocean.com
> >> 
> >> We're Hiring! | @digitalocean | linkedin
> >>
> >
> > [1] https://ceph.com/community/new-mimic-simplified-rbd-image-cloning/
> >
> > --
> > Jason
>


-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Questions about rbd-mirror and clones

[Jason Dillaman]

On Tue, Feb 26, 2019 at 7:49 PM Anthony D'Atri  wrote:
>
> Hello again.
>
> I have a couple of questions about rbd-mirror that I'm hoping you can help me 
> with.
>
>
> 1) http://docs.ceph.com/docs/mimic/rbd/rbd-snapshot/ indicates that 
> protecting is required for cloning.  We somehow had the notion that this had 
> been / will be done away with, but don't remember where we saw that.  
> Thoughts?

By default, if the cluster is configured to to require mimic or later
clients, you no longer need to protect/unprotect snapshots prior to
cloning [1]. The documentation still talks about
protecting/unprotecting snapshots since the new clone v2 format isn't
currently enabled by default in order to preserve backwards
compatibility to older librbd/krbd clients. Once we no longer support
upgrading from pre-Mimic releases, we can enable clone v2 by default
and start deprecating snapshot protect/unprotect features.

> 2) We're currently running 12.2.2 on our cluster nodes, with rbd-mirror 
> running in a container built against 12.2.8.  Should we expect images with 
> clones / parents to successfully migrate with rbd-mirror? I've had a few rude 
> awakenings here where I've flattened to remove the dependency, but in the 
> general case would rather not have to sacrifice the underlying capacity.

Yes, thinly provisioned cloned images have always been supported with
RBD mirroring (Jewel release). You do, however, need to ensure that
the parent image has mirroring enabled.

>
>
>
> Context:  We aren't using rbd-mirror for DR, we're using it to move volumes 
> between clusters for capacity management.
>
> Hope to see you at Cephalocon.
>
>
>
>
> Anthony D'Atri
> Storage Engineer
> 425-343-5133
> ada...@digitalocean.com
> 
> We're Hiring! | @digitalocean | linkedin
>

[1] https://ceph.com/community/new-mimic-simplified-rbd-image-cloning/

-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Questions on rbd-mirror

[Jason Dillaman]

This is something we have talked about in the past -- and I think it
would be a good addition. The caveat is that if we would now have
config-level, per-pool level, per-image level (via rbd image-meta),
and command-line/environment variable configuration overrides.

I think there needs to be clear tooling in the rbd CLI to dump RBD
configuration overrides and the source of those overrides -- and it
needs to be well documented.

On Mon, Mar 27, 2017 at 9:31 AM, Dongsheng Yang
 wrote:
> Jason,
> do you think it's good idea to introduce a rbd_config object to
> record some configurations of per-pool, such as default_features.
>
> That means, we can set some configurations differently in different
> pool. In this way, we can also handle the per-pool setting in rbd-mirror.
>
> Thanx
> Yang
>
>
> On 27/03/2017, 21:20, Jason Dillaman wrote:
>>
>> On Mon, Mar 27, 2017 at 4:00 AM, Dongsheng Yang
>>  wrote:
>>>
>>> Hi Fulvio,
>>>
>>> On 03/24/2017 07:19 PM, Fulvio Galeazzi wrote:
>>>
>>> Hallo, apologies for my (silly) questions, I did try to find some doc on
>>> rbd-mirror but was unable to, apart from a number of pages explaining how
>>> to
>>> install it.
>>>
>>> My environment is CenOS7 and Ceph 10.2.5.
>>>
>>> Can anyone help me understand a few minor things:
>>>
>>>   - is there a cleaner way to configure the user which will be used for
>>> rbd-mirror, other than editing the ExecStart in file
>>> /usr/lib/systemd/system/ceph-rbd-mirror@.service ?
>>> For example some line in ceph.conf... looks like the username
>>> defaults to the cluster name, am I right?
>>>
>>>
>>> It should just be "ceph", no matter what the cluster name is, if I read
>>> the
>>> code correctly.
>>
>> The user id is passed in via the systemd instance name. For example,
>> if you wanted to use the "mirror" user id to connect to the local
>> cluster, you would run "systemctl enable ceph-rbd-mirror@mirror".
>>
>>>   - is it possible to throttle mirroring? Sure, it's a crazy thing to do
>>> for "cinder" pools, but may make sense for slowly changing ones, like
>>> a "glance" pool.
>>>
>>>
>>> The rbd core team is working on this. Jason, right?
>>
>> This is in our backlog of desired items for the rbd-mirror daemon.
>> Having different settings for different pools was not in our original
>> plan, but this is something that also came up during the Vault
>> conference last week. I've added an additional backlog item to cover
>> per-pool settings.
>>
>>>   - is it possible to set per-pool default features? I read about
>>>  "rbd default features = ###"
>>> but this is a global setting. (Ok, I can still restrict pools to be
>>> mirrored with "ceph auth" for the user doing mirroring)
>>>
>>>
>>> "per-pool default features" sounds like a reasonable feature request.
>>>
>>> About the "ceph auth" for mirroring, I am working on a rbd acl design,
>>> will consider pool-level, namespace-level and image-level. Then I think
>>> we can do a permission check on this.
>>
>> Right now, the best way to achieve that is by using different configs
>> / user ids for different services. For example, if OpenStack glance
>> used "glance" and cinder user "cinder", the ceph.conf's
>> "[client.glance]" section could have different default features as
>> compared to a "[client.cinder]" section.
>>
>>> Thanx
>>> Yang
>>>
>>>
>>>
>>>Thanks!
>>>
>>>  Fulvio
>>>
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>
>>
>
>



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Questions on rbd-mirror

[Dongsheng Yang]

Jason,
do you think it's good idea to introduce a rbd_config object to
record some configurations of per-pool, such as default_features.

That means, we can set some configurations differently in different
pool. In this way, we can also handle the per-pool setting in rbd-mirror.

Thanx
Yang

On 27/03/2017, 21:20, Jason Dillaman wrote:

On Mon, Mar 27, 2017 at 4:00 AM, Dongsheng Yang
 wrote:

Hi Fulvio,

On 03/24/2017 07:19 PM, Fulvio Galeazzi wrote:

Hallo, apologies for my (silly) questions, I did try to find some doc on
rbd-mirror but was unable to, apart from a number of pages explaining how to
install it.

My environment is CenOS7 and Ceph 10.2.5.

Can anyone help me understand a few minor things:

  - is there a cleaner way to configure the user which will be used for
rbd-mirror, other than editing the ExecStart in file
/usr/lib/systemd/system/ceph-rbd-mirror@.service ?
For example some line in ceph.conf... looks like the username
defaults to the cluster name, am I right?


It should just be "ceph", no matter what the cluster name is, if I read the
code correctly.

The user id is passed in via the systemd instance name. For example,
if you wanted to use the "mirror" user id to connect to the local
cluster, you would run "systemctl enable ceph-rbd-mirror@mirror".


  - is it possible to throttle mirroring? Sure, it's a crazy thing to do
for "cinder" pools, but may make sense for slowly changing ones, like
a "glance" pool.


The rbd core team is working on this. Jason, right?

This is in our backlog of desired items for the rbd-mirror daemon.
Having different settings for different pools was not in our original
plan, but this is something that also came up during the Vault
conference last week. I've added an additional backlog item to cover
per-pool settings.


  - is it possible to set per-pool default features? I read about
 "rbd default features = ###"
but this is a global setting. (Ok, I can still restrict pools to be
mirrored with "ceph auth" for the user doing mirroring)


"per-pool default features" sounds like a reasonable feature request.

About the "ceph auth" for mirroring, I am working on a rbd acl design,
will consider pool-level, namespace-level and image-level. Then I think
we can do a permission check on this.

Right now, the best way to achieve that is by using different configs
/ user ids for different services. For example, if OpenStack glance
used "glance" and cinder user "cinder", the ceph.conf's
"[client.glance]" section could have different default features as
compared to a "[client.cinder]" section.


Thanx
Yang



   Thanks!

 Fulvio



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com








___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Questions on rbd-mirror

[Jason Dillaman]

On Mon, Mar 27, 2017 at 4:00 AM, Dongsheng Yang
 wrote:
> Hi Fulvio,
>
> On 03/24/2017 07:19 PM, Fulvio Galeazzi wrote:
>
> Hallo, apologies for my (silly) questions, I did try to find some doc on
> rbd-mirror but was unable to, apart from a number of pages explaining how to
> install it.
>
> My environment is CenOS7 and Ceph 10.2.5.
>
> Can anyone help me understand a few minor things:
>
>  - is there a cleaner way to configure the user which will be used for
>rbd-mirror, other than editing the ExecStart in file
> /usr/lib/systemd/system/ceph-rbd-mirror@.service ?
>For example some line in ceph.conf... looks like the username
>defaults to the cluster name, am I right?
>
>
> It should just be "ceph", no matter what the cluster name is, if I read the
> code correctly.

The user id is passed in via the systemd instance name. For example,
if you wanted to use the "mirror" user id to connect to the local
cluster, you would run "systemctl enable ceph-rbd-mirror@mirror".

>  - is it possible to throttle mirroring? Sure, it's a crazy thing to do
>for "cinder" pools, but may make sense for slowly changing ones, like
>a "glance" pool.
>
>
> The rbd core team is working on this. Jason, right?

This is in our backlog of desired items for the rbd-mirror daemon.
Having different settings for different pools was not in our original
plan, but this is something that also came up during the Vault
conference last week. I've added an additional backlog item to cover
per-pool settings.

>  - is it possible to set per-pool default features? I read about
> "rbd default features = ###"
>but this is a global setting. (Ok, I can still restrict pools to be
>mirrored with "ceph auth" for the user doing mirroring)
>
>
> "per-pool default features" sounds like a reasonable feature request.
>
> About the "ceph auth" for mirroring, I am working on a rbd acl design,
> will consider pool-level, namespace-level and image-level. Then I think
> we can do a permission check on this.

Right now, the best way to achieve that is by using different configs
/ user ids for different services. For example, if OpenStack glance
used "glance" and cinder user "cinder", the ceph.conf's
"[client.glance]" section could have different default features as
compared to a "[client.cinder]" section.

> Thanx
> Yang
>
>
>
>   Thanks!
>
> Fulvio
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Questions on rbd-mirror

[Dongsheng Yang]

Hi Fulvio,

On 03/24/2017 07:19 PM, Fulvio Galeazzi wrote:
Hallo, apologies for my (silly) questions, I did try to find some doc 
on rbd-mirror but was unable to, apart from a number of pages 
explaining how to install it.


My environment is CenOS7 and Ceph 10.2.5.

Can anyone help me understand a few minor things:

 - is there a cleaner way to configure the user which will be used for
   rbd-mirror, other than editing the ExecStart in file 
/usr/lib/systemd/system/ceph-rbd-mirror@.service ?

   For example some line in ceph.conf... looks like the username
   defaults to the cluster name, am I right?


It should just be "ceph", no matter what the cluster name is, if I read 
the code correctly.


 - is it possible to throttle mirroring? Sure, it's a crazy thing to do
   for "cinder" pools, but may make sense for slowly changing ones, like
   a "glance" pool.


The rbd core team is working on this. Jason, right?


 - is it possible to set per-pool default features? I read about
"rbd default features = ###"
   but this is a global setting. (Ok, I can still restrict pools to be
   mirrored with "ceph auth" for the user doing mirroring)


"per-pool default features" sounds like a reasonable feature request.

About the "ceph auth" for mirroring, I am working on a rbd acl design,
will consider pool-level, namespace-level and image-level. Then I think
we can do a permission check on this.

Thanx
Yang



  Thanks!

Fulvio



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Questions on rbd-mirror

[Fulvio Galeazzi]

Hallo, apologies for my (silly) questions, I did try to find some doc on 
rbd-mirror but was unable to, apart from a number of pages explaining 
how to install it.


My environment is CenOS7 and Ceph 10.2.5.

Can anyone help me understand a few minor things:

 - is there a cleaner way to configure the user which will be used for
   rbd-mirror, other than editing the ExecStart in file 
/usr/lib/systemd/system/ceph-rbd-mirror@.service ?

   For example some line in ceph.conf... looks like the username
   defaults to the cluster name, am I right?

 - is it possible to throttle mirroring? Sure, it's a crazy thing to do
   for "cinder" pools, but may make sense for slowly changing ones, like
   a "glance" pool.

 - is it possible to set per-pool default features? I read about
"rbd default features = ###"
   but this is a global setting. (Ok, I can still restrict pools to be
   mirrored with "ceph auth" for the user doing mirroring)


  Thanks!

Fulvio



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com