Re: [ceph-users] howto: multiple ceph filesystems

[Webert de Souza Lima]

Basically what we're trying to figure out looks like what is being done
here:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-September/020958.html

But instead of using LIBRADOS to store EMAILs directly into RADOS we're
still using CEPHFS for it, just figuring out if it makes sense to separate
them in different workloads.


Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
*IRC NICK - WebertRLZ*

On Fri, May 11, 2018 at 2:07 AM, Marc Roos <m.r...@f1-outsourcing.eu> wrote:

>
>
> If I would like to use an erasurecode pool for a cephfs directory how
> would I create these placement rules?
>
>
>
>
> -Original Message-
> From: David Turner [mailto:drakonst...@gmail.com]
> Sent: vrijdag 11 mei 2018 1:54
> To: João Paulo Sacchetto Ribeiro Bastos
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] howto: multiple ceph filesystems
>
> Another option you could do is to use a placement rule. You could create
> a general pool for most data to go to and a special pool for specific
> folders on the filesystem. Particularly I think of a pool for replica vs
> EC vs flash for specific folders in the filesystem.
>
> If the pool and OSDs wasn't the main concern for multiple filesystems
> and the mds servers are then you could have multiple active mds servers
> and pin the metadata for the indexes to one of them while the rest is
> served by the other active mds servers.
>
> I really haven't come across a need for multiple filesystems in ceph
> with the type of granularity you can achieve with mds pinning, folder
> placement rules, and cephx authentication to limit a user to a specific
> subfolder.
>
>
> On Thu, May 10, 2018, 5:10 PM João Paulo Sacchetto Ribeiro Bastos
> <joaopaulos...@gmail.com> wrote:
>
>
> Hey John, thanks for you answer. For sure the hardware robustness
> will be nice enough. My true concern was actually the two FS ecosystem
> coexistence. In fact I realized that we may not use this as well because
> it may be represent a high overhead, despite the fact that it's a
> experiental feature yet.
>
> On Thu, 10 May 2018 at 15:48 John Spray <jsp...@redhat.com> wrote:
>
>
> On Thu, May 10, 2018 at 7:38 PM, João Paulo Sacchetto
> Ribeiro
> Bastos
> <joaopaulos...@gmail.com> wrote:
> > Hello guys,
> >
> > My company is about to rebuild its whole infrastructure,
> so
> I was called in
> > order to help on the planning. We are essentially an
> corporate mail
> > provider, so we handle daily lots of clients using
> dovecot
> and roundcube and
> > in order to do so we want to design a better plant of
> our
> cluster. Today,
> > using Jewel, we have a single cephFS for both index and
> mail
> from dovecot,
> > but we want to split it into an index_FS and a mail_FS
> to
> handle the
> > workload a little better, is it profitable nowadays?
> From my
> research I
> > realized that we will need data and metadata individual
> pools for each FS
> > such as a group of MDS for each of then, also.
> >
> > The one thing that really scares me about all of this
> is: we
> are planning to
> > have four machines at full disposal to handle our MDS
> instances. We started
> > to think if an idea like the one below is valid, can
> anybody
> give a hint on
> > this? We basically want to handle two MDS instances on
> each
> machine (one for
> > each FS) and wonder if we'll be able to have them
> swapping
> between active
> > and standby simultaneously without any trouble.
> >
> > index_FS: (active={machines 1 and 3}, standby={machines
> 2
> and 4})
> > mail_FS: (active={machines 2 and 4}, standby={machines 1
> and
> 3})
>
> Nothing wrong with that setup, but remember that those
> servers
> are
> going to have to be well-resourced enough to run all four
> at
> once
> (when a failure occurs), so it might not matter very much
> exactly
> which servers are running which daemons.
>
> With a filesystem's MDS daemons (i.e. daemons with the same
> standby_for_fscid setting), Ceph will activate whichever
> daemon comes
>  

Re: [ceph-users] howto: multiple ceph filesystems

[Marc Roos]

 

If I would like to use an erasurecode pool for a cephfs directory how 
would I create these placement rules?




-Original Message-
From: David Turner [mailto:drakonst...@gmail.com] 
Sent: vrijdag 11 mei 2018 1:54
To: João Paulo Sacchetto Ribeiro Bastos
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] howto: multiple ceph filesystems

Another option you could do is to use a placement rule. You could create 
a general pool for most data to go to and a special pool for specific 
folders on the filesystem. Particularly I think of a pool for replica vs 
EC vs flash for specific folders in the filesystem.

If the pool and OSDs wasn't the main concern for multiple filesystems 
and the mds servers are then you could have multiple active mds servers 
and pin the metadata for the indexes to one of them while the rest is 
served by the other active mds servers.

I really haven't come across a need for multiple filesystems in ceph 
with the type of granularity you can achieve with mds pinning, folder 
placement rules, and cephx authentication to limit a user to a specific 
subfolder.


On Thu, May 10, 2018, 5:10 PM João Paulo Sacchetto Ribeiro Bastos 
<joaopaulos...@gmail.com> wrote:


Hey John, thanks for you answer. For sure the hardware robustness 
will be nice enough. My true concern was actually the two FS ecosystem 
coexistence. In fact I realized that we may not use this as well because 
it may be represent a high overhead, despite the fact that it's a 
experiental feature yet.

On Thu, 10 May 2018 at 15:48 John Spray <jsp...@redhat.com> wrote:


On Thu, May 10, 2018 at 7:38 PM, João Paulo Sacchetto Ribeiro 
Bastos
<joaopaulos...@gmail.com> wrote:
> Hello guys,
>
> My company is about to rebuild its whole infrastructure, so 
I was called in
> order to help on the planning. We are essentially an 
corporate mail
> provider, so we handle daily lots of clients using dovecot 
and roundcube and
> in order to do so we want to design a better plant of our 
cluster. Today,
> using Jewel, we have a single cephFS for both index and mail 
from dovecot,
> but we want to split it into an index_FS and a mail_FS to 
handle the
> workload a little better, is it profitable nowadays? From my 
research I
> realized that we will need data and metadata individual 
pools for each FS
> such as a group of MDS for each of then, also.
>
> The one thing that really scares me about all of this is: we 
are planning to
> have four machines at full disposal to handle our MDS 
instances. We started
> to think if an idea like the one below is valid, can anybody 
give a hint on
> this? We basically want to handle two MDS instances on each 
machine (one for
> each FS) and wonder if we'll be able to have them swapping 
between active
> and standby simultaneously without any trouble.
>
> index_FS: (active={machines 1 and 3}, standby={machines 2 
and 4})
> mail_FS: (active={machines 2 and 4}, standby={machines 1 and 
3})

Nothing wrong with that setup, but remember that those servers 
are
going to have to be well-resourced enough to run all four at 
once
(when a failure occurs), so it might not matter very much 
exactly
which servers are running which daemons.

With a filesystem's MDS daemons (i.e. daemons with the same
standby_for_fscid setting), Ceph will activate whichever 
daemon comes
up first, so if it's important to you to have particular 
daemons
active then you would need to take care of that at the point 
you're
starting them up.

John

>
> Regards,
> --
>
> João Paulo Sacchetto Ribeiro Bastos
> +55 31 99279-7092
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>


-- 


João Paulo Sacchetto Ribeiro Bastos
+55 31 99279-7092

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] howto: multiple ceph filesystems

[David Turner]

Another option you could do is to use a placement rule. You could create a
general pool for most data to go to and a special pool for specific folders
on the filesystem. Particularly I think of a pool for replica vs EC vs
flash for specific folders in the filesystem.

If the pool and OSDs wasn't the main concern for multiple filesystems and
the mds servers are then you could have multiple active mds servers and pin
the metadata for the indexes to one of them while the rest is served by the
other active mds servers.

I really haven't come across a need for multiple filesystems in ceph with
the type of granularity you can achieve with mds pinning, folder placement
rules, and cephx authentication to limit a user to a specific subfolder.

On Thu, May 10, 2018, 5:10 PM João Paulo Sacchetto Ribeiro Bastos <
joaopaulos...@gmail.com> wrote:

> Hey John, thanks for you answer. For sure the hardware robustness will be
> nice enough. My true concern was actually the two FS ecosystem coexistence.
> In fact I realized that we may not use this as well because it may be
> represent a high overhead, despite the fact that it's a experiental feature
> yet.
> On Thu, 10 May 2018 at 15:48 John Spray  wrote:
>
>> On Thu, May 10, 2018 at 7:38 PM, João Paulo Sacchetto Ribeiro Bastos
>>  wrote:
>> > Hello guys,
>> >
>> > My company is about to rebuild its whole infrastructure, so I was
>> called in
>> > order to help on the planning. We are essentially an corporate mail
>> > provider, so we handle daily lots of clients using dovecot and
>> roundcube and
>> > in order to do so we want to design a better plant of our cluster.
>> Today,
>> > using Jewel, we have a single cephFS for both index and mail from
>> dovecot,
>> > but we want to split it into an index_FS and a mail_FS to handle the
>> > workload a little better, is it profitable nowadays? From my research I
>> > realized that we will need data and metadata individual pools for each
>> FS
>> > such as a group of MDS for each of then, also.
>> >
>> > The one thing that really scares me about all of this is: we are
>> planning to
>> > have four machines at full disposal to handle our MDS instances. We
>> started
>> > to think if an idea like the one below is valid, can anybody give a
>> hint on
>> > this? We basically want to handle two MDS instances on each machine
>> (one for
>> > each FS) and wonder if we'll be able to have them swapping between
>> active
>> > and standby simultaneously without any trouble.
>> >
>> > index_FS: (active={machines 1 and 3}, standby={machines 2 and 4})
>> > mail_FS: (active={machines 2 and 4}, standby={machines 1 and 3})
>>
>> Nothing wrong with that setup, but remember that those servers are
>> going to have to be well-resourced enough to run all four at once
>> (when a failure occurs), so it might not matter very much exactly
>> which servers are running which daemons.
>>
>> With a filesystem's MDS daemons (i.e. daemons with the same
>> standby_for_fscid setting), Ceph will activate whichever daemon comes
>> up first, so if it's important to you to have particular daemons
>> active then you would need to take care of that at the point you're
>> starting them up.
>>
>> John
>>
>> >
>> > Regards,
>> > --
>> >
>> > João Paulo Sacchetto Ribeiro Bastos
>> > +55 31 99279-7092
>> >
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>>
> --
>
> João Paulo Sacchetto Ribeiro Bastos
> +55 31 99279-7092
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] howto: multiple ceph filesystems

[João Paulo Sacchetto Ribeiro Bastos]

Hey John, thanks for you answer. For sure the hardware robustness will be
nice enough. My true concern was actually the two FS ecosystem coexistence.
In fact I realized that we may not use this as well because it may be
represent a high overhead, despite the fact that it's a experiental feature
yet.
On Thu, 10 May 2018 at 15:48 John Spray  wrote:

> On Thu, May 10, 2018 at 7:38 PM, João Paulo Sacchetto Ribeiro Bastos
>  wrote:
> > Hello guys,
> >
> > My company is about to rebuild its whole infrastructure, so I was called
> in
> > order to help on the planning. We are essentially an corporate mail
> > provider, so we handle daily lots of clients using dovecot and roundcube
> and
> > in order to do so we want to design a better plant of our cluster. Today,
> > using Jewel, we have a single cephFS for both index and mail from
> dovecot,
> > but we want to split it into an index_FS and a mail_FS to handle the
> > workload a little better, is it profitable nowadays? From my research I
> > realized that we will need data and metadata individual pools for each FS
> > such as a group of MDS for each of then, also.
> >
> > The one thing that really scares me about all of this is: we are
> planning to
> > have four machines at full disposal to handle our MDS instances. We
> started
> > to think if an idea like the one below is valid, can anybody give a hint
> on
> > this? We basically want to handle two MDS instances on each machine (one
> for
> > each FS) and wonder if we'll be able to have them swapping between active
> > and standby simultaneously without any trouble.
> >
> > index_FS: (active={machines 1 and 3}, standby={machines 2 and 4})
> > mail_FS: (active={machines 2 and 4}, standby={machines 1 and 3})
>
> Nothing wrong with that setup, but remember that those servers are
> going to have to be well-resourced enough to run all four at once
> (when a failure occurs), so it might not matter very much exactly
> which servers are running which daemons.
>
> With a filesystem's MDS daemons (i.e. daemons with the same
> standby_for_fscid setting), Ceph will activate whichever daemon comes
> up first, so if it's important to you to have particular daemons
> active then you would need to take care of that at the point you're
> starting them up.
>
> John
>
> >
> > Regards,
> > --
> >
> > João Paulo Sacchetto Ribeiro Bastos
> > +55 31 99279-7092
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
-- 

João Paulo Sacchetto Ribeiro Bastos
+55 31 99279-7092
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] howto: multiple ceph filesystems

[John Spray]

On Thu, May 10, 2018 at 7:38 PM, João Paulo Sacchetto Ribeiro Bastos
 wrote:
> Hello guys,
>
> My company is about to rebuild its whole infrastructure, so I was called in
> order to help on the planning. We are essentially an corporate mail
> provider, so we handle daily lots of clients using dovecot and roundcube and
> in order to do so we want to design a better plant of our cluster. Today,
> using Jewel, we have a single cephFS for both index and mail from dovecot,
> but we want to split it into an index_FS and a mail_FS to handle the
> workload a little better, is it profitable nowadays? From my research I
> realized that we will need data and metadata individual pools for each FS
> such as a group of MDS for each of then, also.
>
> The one thing that really scares me about all of this is: we are planning to
> have four machines at full disposal to handle our MDS instances. We started
> to think if an idea like the one below is valid, can anybody give a hint on
> this? We basically want to handle two MDS instances on each machine (one for
> each FS) and wonder if we'll be able to have them swapping between active
> and standby simultaneously without any trouble.
>
> index_FS: (active={machines 1 and 3}, standby={machines 2 and 4})
> mail_FS: (active={machines 2 and 4}, standby={machines 1 and 3})

Nothing wrong with that setup, but remember that those servers are
going to have to be well-resourced enough to run all four at once
(when a failure occurs), so it might not matter very much exactly
which servers are running which daemons.

With a filesystem's MDS daemons (i.e. daemons with the same
standby_for_fscid setting), Ceph will activate whichever daemon comes
up first, so if it's important to you to have particular daemons
active then you would need to take care of that at the point you're
starting them up.

John

>
> Regards,
> --
>
> João Paulo Sacchetto Ribeiro Bastos
> +55 31 99279-7092
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] howto: multiple ceph filesystems

[João Paulo Sacchetto Ribeiro Bastos]

Hello guys,

My company is about to rebuild its whole infrastructure, so I was called in
order to help on the planning. We are essentially an corporate mail
provider, so we handle daily lots of clients using dovecot and roundcube
and in order to do so we want to design a better plant of our cluster.
Today, using Jewel, we have a single cephFS for both index and mail from
dovecot, but we want to split it into an index_FS and a mail_FS to handle
the workload a little better, is it profitable nowadays? From my research I
realized that we will need data and metadata individual pools for each FS
such as a group of MDS for each of then, also.

The one thing that really scares me about all of this is: we are planning
to have four machines at full disposal to handle our MDS instances. We
started to think if an idea like the one below is valid, can anybody give a
hint on this? We basically want to handle two MDS instances on each machine
(one for each FS) and wonder if we'll be able to have them swapping between
active and standby simultaneously without any trouble.

index_FS: (active={machines 1 and 3}, standby={machines 2 and 4})
mail_FS: (active={machines 2 and 4}, standby={machines 1 and 3})

Regards,
-- 

João Paulo Sacchetto Ribeiro Bastos
+55 31 99279-7092
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com