Re: [ceph-users] bcache vs flashcache vs cache tiering
Hi all, I'd also not like to see cache tiering in the current form go away. We've explored using it in situations where we have a data pool with replicas spread across WAN sites which we then overlay with a fast cache tier local to the site where most clients will be using the pool. This significantly speeds up operations for that set of clients as long as they don't involve waiting for the cache to flush data. In theory we could move the cache tier around (flush, delete, recreate) as needed. The drawback of course is that clients not near the cache tier pool would still be forced into using it by Ceph. What would be really useful is multiple cache pools with rulesets that allow us to direct clients by IP, GeoIP proximity lookups or something like that. As I see it, the alternative configuration for this case is to simply map replicas at the appropriate site. Which makes more sense for us might depend on different factors for different project users. thanks, Ben --- OSiRIS htttp://www.osris.org On Thu, Feb 16, 2017 at 3:40 AM, Dongsheng Yang wrote: > BTW, is there any body using EnhanceIO? > > On 02/15/2017 05:51 PM, Dongsheng Yang wrote: >> >> thanx Nick, Gregory and Wido, >> So at least, we can say the cache tiering in Jewel is stable enough I >> think. >> I like cache tiering more than the others, but yes, there is a problem >> about cache tiering in >> flushing data between different nodes, which are not a problem in local >> caching solution. >> >> guys: >> Is there any plan to enhance cache tiering to solve such problem? Or >> as Nick asked, is >> that cache tiering fading away? >> >> Yang >> >> >> On 15/02/2017, 06:42, Nick Fisk wrote: >>>> >>>> -Original Message- >>>> From: Gregory Farnum [mailto:gfar...@redhat.com] >>>> Sent: 14 February 2017 21:05 >>>> To: Wido den Hollander >>>> Cc: Dongsheng Yang ; Nick Fisk >>>> ; Ceph Users >>>> Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering >>>> >>>> On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander >>>> wrote: >>>>>> >>>>>> Op 14 februari 2017 om 11:14 schreef Nick Fisk : >>>>>> >>>>>> >>>>>>> -Original Message- >>>>>>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On >>>>>>> Behalf Of Dongsheng Yang >>>>>>> Sent: 14 February 2017 09:01 >>>>>>> To: Sage Weil >>>>>>> Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com >>>>>>> Subject: [ceph-users] bcache vs flashcache vs cache tiering >>>>>>> >>>>>>> Hi Sage and all, >>>>>>> We are going to use SSDs for cache in ceph. But I am not sure >>>>>>> which one is the best solution, bcache? flashcache? or cache >>>>>> >>>>>> tier? >>>>>> >>>>>> I would vote for cache tier. Being able to manage it from within >>>>>> Ceph, instead of having to manage X number of bcache/flashcache >>>>>> instances, appeals to me more. Also last time I looked Flashcache >>>>>> seems unmaintained and bcache might be going that way with talk of >>>>>> this new bcachefs. Another point to consider is that Ceph has had a >>>>>> lot of >>>> >>>> work done on it to ensure data consistency; I don't ever want to be in a >>>> position where I'm trying to diagnose problems that might be being >>>> caused >>>> by another layer sitting in-between Ceph and the Disk. >>>>>> >>>>>> However, I know several people on here are using bcache and >>>>>> potentially getting better performance than with cache tiering, so >>>> >>>> hopefully someone will give their views. >>>>> >>>>> I am using Bcache on various systems and it performs really well. The >>>> >>>> caching layer in Ceph is slow. Promoting Objects is slow and it also >>>> involves >>>> additional RADOS lookups. >>>> >>>> Yeah. Cache tiers have gotten a lot more usable in Ceph, but the use >>>> cases >>>> where they're effective are still pretty limited and I think in-node >>>> caching has >>>> a brighter future. We just don't like to maintain the global state that >>>> makes >>
Re: [ceph-users] bcache vs flashcache vs cache tiering
BTW, is there any body using EnhanceIO? On 02/15/2017 05:51 PM, Dongsheng Yang wrote: thanx Nick, Gregory and Wido, So at least, we can say the cache tiering in Jewel is stable enough I think. I like cache tiering more than the others, but yes, there is a problem about cache tiering in flushing data between different nodes, which are not a problem in local caching solution. guys: Is there any plan to enhance cache tiering to solve such problem? Or as Nick asked, is that cache tiering fading away? Yang On 15/02/2017, 06:42, Nick Fisk wrote: -Original Message- From: Gregory Farnum [mailto:gfar...@redhat.com] Sent: 14 February 2017 21:05 To: Wido den Hollander Cc: Dongsheng Yang ; Nick Fisk ; Ceph Users Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander wrote: Op 14 februari 2017 om 11:14 schreef Nick Fisk : -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dongsheng Yang Sent: 14 February 2017 09:01 To: Sage Weil Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com Subject: [ceph-users] bcache vs flashcache vs cache tiering Hi Sage and all, We are going to use SSDs for cache in ceph. But I am not sure which one is the best solution, bcache? flashcache? or cache tier? I would vote for cache tier. Being able to manage it from within Ceph, instead of having to manage X number of bcache/flashcache instances, appeals to me more. Also last time I looked Flashcache seems unmaintained and bcache might be going that way with talk of this new bcachefs. Another point to consider is that Ceph has had a lot of work done on it to ensure data consistency; I don't ever want to be in a position where I'm trying to diagnose problems that might be being caused by another layer sitting in-between Ceph and the Disk. However, I know several people on here are using bcache and potentially getting better performance than with cache tiering, so hopefully someone will give their views. I am using Bcache on various systems and it performs really well. The caching layer in Ceph is slow. Promoting Objects is slow and it also involves additional RADOS lookups. Yeah. Cache tiers have gotten a lot more usable in Ceph, but the use cases where they're effective are still pretty limited and I think in-node caching has a brighter future. We just don't like to maintain the global state that makes separate caching locations viable and unless you're doing something analogous to the supercomputing "burst buffers" (which some people are!), it's going to be hard to beat something that doesn't have to pay the cost of extra network hops/bandwidth. Cache tiers are also not a feature that all the vendors support in their downstream products, so it will probably see less ongoing investment than you'd expect from such a system. Should that be taken as an unofficial sign that the tiering support is likely to fade away? I think both approaches have different strengths and probably the difference between a tiering system and a caching one is what causes some of the problems. If something like bcache is going to be the preferred approach, then I think more work needs to be done around certifying it for use with Ceph and allowing its behavior to be more controlled by Ceph as well. I assume there are issues around backfilling and scrubbing polluting the cache? Maybe you would want to be able to pass hints down from Ceph, which could also allow per pool cache behavior?? -Greg -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bcache vs flashcache vs cache tiering
Hello Sage, On Wed, 15 Feb 2017 22:10:26 + (UTC) Sage Weil wrote: > On Wed, 15 Feb 2017, Nick Fisk wrote: > > Just an update. I spoke to Sage today and the general consensus is that > > something like bcache or dmcache is probably the long term goal, but work > > needs to be done before its ready for prime time. The current tiering > > functionality won't be going away in the short term and not until there is a > > solid replacement with bcache/dmcache/whatever. But from the sounds of it, > > there won't be any core dev time allocated to it. > > Quick clarification: I meant the cache tiering isn't going anywhere until > there is another solid *rados* tiering replacement. The rados tiering > plans will keep metadata in the base pool but link to data in one or more > other (presumably colder) tiers (vs a sparse cache pool in front of the > base pool). > I see where you're coming from (EC pools basically as you mention below), but for "burst buffers" users this looks rather glum. In my use case (mentioned a dozen times by now I'm sure ^o^), we have hundreds of VMs where the main application pretty much constantly writes to (small, cyclic) logs, locks and state files. Creating a nice stream of (currently) about 8-10MB/s and 1000 IOPS (as Ceph sees them). This results in a comperatively small but quite hot working set, on quiet days (no VM reboots, upgrades etc) we have nearly no tier promotions going on. While fast metadata is nice in the scenario you describe above I would basically have to go all SSD or at least massively bcache to get the performance needed w/o the current cache-tier design. My 2 yen, Christian > That is, you should consider rados tiering as totally orthogonal to > tiering devices beneath a single OSD with dm-cache/bcache/flashcache. > > > I'm not really too bothered what the solution ends up being, but as we have > > discussed the flexibility to shrink/grow the cache without having to > > rebuild all your nodes/OSD's is a major, almost essential, benefit to me. > > Exactly. The new rados tiering approach would still provide this. > > > I've still got some ideas which I think can improve performance of the > > tiering functionality, but unsure as to whether I have the coding skills to > > pull it off. This might motivate me though to try and improve it in its > > current form. > > FWIW the effectiveness of the existing rados cache tiering will also > improve significantly with the EC overwrite support. Whether it is > removed as part of a new/different rados tiering function in rados is > really a function of how the code refactor works out and how difficult it > is to support vs the use cases it covers that the new tiering does not. > > sage > -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bcache vs flashcache vs cache tiering
> On Wed, 15 Feb 2017, Nick Fisk wrote: > > Just an update. I spoke to Sage today and the general consensus is > > that something like bcache or dmcache is probably the long term goal, > > but work needs to be done before its ready for prime time. The current > > tiering functionality won't be going away in the short term and not > > until there is a solid replacement with bcache/dmcache/whatever. But > > from the sounds of it, there won't be any core dev time allocated to it. > > Quick clarification: I meant the cache tiering isn't going anywhere until > there > is another solid *rados* tiering replacement. The rados tiering plans will > keep metadata in the base pool but link to data in one or more other > (presumably colder) tiers (vs a sparse cache pool in front of the base pool). > > That is, you should consider rados tiering as totally orthogonal to tiering > devices beneath a single OSD with dm-cache/bcache/flashcache. > > > I'm not really too bothered what the solution ends up being, but as we > > have discussed the flexibility to shrink/grow the cache without > > having to rebuild all your nodes/OSD's is a major, almost essential, benefit > to me. > > Exactly. The new rados tiering approach would still provide this. > > > I've still got some ideas which I think can improve performance of the > > tiering functionality, but unsure as to whether I have the coding > > skills to pull it off. This might motivate me though to try and > > improve it in its current form. > > FWIW the effectiveness of the existing rados cache tiering will also improve > significantly with the EC overwrite support. Whether it is removed as part of > a new/different rados tiering function in rados is really a function of how > the > code refactor works out and how difficult it is to support vs the use cases it > covers that the new tiering does not. Oh ok, awesome then. Keep me in the loop, I will be there with my trusty old testing rig :-) > > sage ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bcache vs flashcache vs cache tiering
On Wed, 15 Feb 2017, Nick Fisk wrote: > Just an update. I spoke to Sage today and the general consensus is that > something like bcache or dmcache is probably the long term goal, but work > needs to be done before its ready for prime time. The current tiering > functionality won't be going away in the short term and not until there is a > solid replacement with bcache/dmcache/whatever. But from the sounds of it, > there won't be any core dev time allocated to it. Quick clarification: I meant the cache tiering isn't going anywhere until there is another solid *rados* tiering replacement. The rados tiering plans will keep metadata in the base pool but link to data in one or more other (presumably colder) tiers (vs a sparse cache pool in front of the base pool). That is, you should consider rados tiering as totally orthogonal to tiering devices beneath a single OSD with dm-cache/bcache/flashcache. > I'm not really too bothered what the solution ends up being, but as we have > discussed the flexibility to shrink/grow the cache without having to > rebuild all your nodes/OSD's is a major, almost essential, benefit to me. Exactly. The new rados tiering approach would still provide this. > I've still got some ideas which I think can improve performance of the > tiering functionality, but unsure as to whether I have the coding skills to > pull it off. This might motivate me though to try and improve it in its > current form. FWIW the effectiveness of the existing rados cache tiering will also improve significantly with the EC overwrite support. Whether it is removed as part of a new/different rados tiering function in rados is really a function of how the code refactor works out and how difficult it is to support vs the use cases it covers that the new tiering does not. sage ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bcache vs flashcache vs cache tiering
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Nick Fisk > Sent: 15 February 2017 09:53 > To: 'Christian Balzer' ; 'Ceph Users' us...@lists.ceph.com> > Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering > > > -Original Message- > > From: Christian Balzer [mailto:ch...@gol.com] > > Sent: 15 February 2017 01:42 > > To: 'Ceph Users' > > Cc: Nick Fisk ; 'Gregory Farnum' > > Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering > > > > On Tue, 14 Feb 2017 22:42:21 - Nick Fisk wrote: > > > > > > -Original Message- > > > > From: Gregory Farnum [mailto:gfar...@redhat.com] > > > > Sent: 14 February 2017 21:05 > > > > To: Wido den Hollander > > > > Cc: Dongsheng Yang ; Nick Fisk > > > > ; Ceph Users > > > > Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering > > > > > > > > On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander > > > > > > > > wrote: > > > > > > > > > >> Op 14 februari 2017 om 11:14 schreef Nick Fisk : > > > > >> > > > > >> > > > > >> > -Original Message- > > > > >> > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] > > > > >> > On Behalf Of Dongsheng Yang > > > > >> > Sent: 14 February 2017 09:01 > > > > >> > To: Sage Weil > > > > >> > Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com > > > > >> > Subject: [ceph-users] bcache vs flashcache vs cache tiering > > > > >> > > > > > >> > Hi Sage and all, > > > > >> > We are going to use SSDs for cache in ceph. But I am not > > > > >> > sure which one is the best solution, bcache? flashcache? or > > > > >> > cache > > > > >> tier? > > > > >> > > > > >> I would vote for cache tier. Being able to manage it from > > > > >> within Ceph, instead of having to manage X number of > > > > >> bcache/flashcache instances, appeals to me more. Also last time > > > > >> I looked Flashcache seems unmaintained and bcache might be > > > > >> going that way with talk of this new bcachefs. Another point to > > > > >> consider is that Ceph has had a lot of > > > > work done on it to ensure data consistency; I don't ever want to > > > > be in a position where I'm trying to diagnose problems that might > > > > be being caused by another layer sitting in-between Ceph and the Disk. > > > > >> > > > > >> However, I know several people on here are using bcache and > > > > >> potentially getting better performance than with cache tiering, > > > > >> so > > > > hopefully someone will give their views. > > > > > > > > > > I am using Bcache on various systems and it performs really well. > > > > > The > > > > caching layer in Ceph is slow. Promoting Objects is slow and it > > > > also involves additional RADOS lookups. > > > > > > > > Yeah. Cache tiers have gotten a lot more usable in Ceph, but the > > > > use cases where they're effective are still pretty limited and I > > > > think in-node caching has a brighter future. We just don't like to > > > > maintain the global state that makes separate caching locations > > > > viable and unless you're doing something analogous to the > > > > supercomputing "burst buffers" (which some people are!), it's > > > > going to be hard to beat something that doesn't have to pay the cost of > extra network hops/bandwidth. > > > > Cache tiers are also not a feature that all the vendors support in > > > > their downstream products, so it will probably see less ongoing > > > > investment than you'd expect from such a system. > > > > > > Should that be taken as an unofficial sign that the tiering support is likely > to fade away? > > > > > Nick, you also posted back in October in the "cache tiering deprecated > > in RHCS 2.0" thread and should remember the deafening silence when I > asked that question. > > > > I'm actually surprised that Greg said as much as he d
Re: [ceph-users] bcache vs flashcache vs cache tiering
> -Original Message- > From: Christian Balzer [mailto:ch...@gol.com] > Sent: 15 February 2017 01:42 > To: 'Ceph Users' > Cc: Nick Fisk ; 'Gregory Farnum' > Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering > > On Tue, 14 Feb 2017 22:42:21 - Nick Fisk wrote: > > > > -Original Message- > > > From: Gregory Farnum [mailto:gfar...@redhat.com] > > > Sent: 14 February 2017 21:05 > > > To: Wido den Hollander > > > Cc: Dongsheng Yang ; Nick Fisk > > > ; Ceph Users > > > Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering > > > > > > On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander > > > wrote: > > > > > > > >> Op 14 februari 2017 om 11:14 schreef Nick Fisk : > > > >> > > > >> > > > >> > -Original Message- > > > >> > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On > > > >> > Behalf Of Dongsheng Yang > > > >> > Sent: 14 February 2017 09:01 > > > >> > To: Sage Weil > > > >> > Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com > > > >> > Subject: [ceph-users] bcache vs flashcache vs cache tiering > > > >> > > > > >> > Hi Sage and all, > > > >> > We are going to use SSDs for cache in ceph. But I am not > > > >> > sure which one is the best solution, bcache? flashcache? or > > > >> > cache > > > >> tier? > > > >> > > > >> I would vote for cache tier. Being able to manage it from within > > > >> Ceph, instead of having to manage X number of bcache/flashcache > > > >> instances, appeals to me more. Also last time I looked Flashcache > > > >> seems unmaintained and bcache might be going that way with talk > > > >> of this new bcachefs. Another point to consider is that Ceph has > > > >> had a lot of > > > work done on it to ensure data consistency; I don't ever want to be > > > in a position where I'm trying to diagnose problems that might be > > > being caused by another layer sitting in-between Ceph and the Disk. > > > >> > > > >> However, I know several people on here are using bcache and > > > >> potentially getting better performance than with cache tiering, > > > >> so > > > hopefully someone will give their views. > > > > > > > > I am using Bcache on various systems and it performs really well. > > > > The > > > caching layer in Ceph is slow. Promoting Objects is slow and it also > > > involves additional RADOS lookups. > > > > > > Yeah. Cache tiers have gotten a lot more usable in Ceph, but the use > > > cases where they're effective are still pretty limited and I think > > > in-node caching has a brighter future. We just don't like to > > > maintain the global state that makes separate caching locations > > > viable and unless you're doing something analogous to the > > > supercomputing "burst buffers" (which some people are!), it's going > > > to be hard to beat something that doesn't have to pay the cost of extra > > > network hops/bandwidth. > > > Cache tiers are also not a feature that all the vendors support in > > > their downstream products, so it will probably see less ongoing > > > investment than you'd expect from such a system. > > > > Should that be taken as an unofficial sign that the tiering support is > > likely to fade away? > > > Nick, you also posted back in October in the "cache tiering deprecated in > RHCS 2.0" thread and should remember the deafening > silence when I asked that question. > > I'm actually surprised that Greg said as much as he did now, unfortunately > that doesn't really cover all the questions I had back then, in > particular long term support and bug fixes, not necessarily more features. > > We're literally about to order our next cluster and cache-tiering works like > a charm for us, even in Hammer. > With the (still undocumented) knobs in Jewel and read-forward it will be even > more effective. > > So given the lack of any statements that next cluster will still use the same > design as the previous one, since BlueStore isn't ready, > bcache and others haven't been tested here to my satisfaction and we know > very well what works and what not
Re: [ceph-users] bcache vs flashcache vs cache tiering
thanx Nick, Gregory and Wido, So at least, we can say the cache tiering in Jewel is stable enough I think. I like cache tiering more than the others, but yes, there is a problem about cache tiering in flushing data between different nodes, which are not a problem in local caching solution. guys: Is there any plan to enhance cache tiering to solve such problem? Or as Nick asked, is that cache tiering fading away? Yang On 15/02/2017, 06:42, Nick Fisk wrote: -Original Message- From: Gregory Farnum [mailto:gfar...@redhat.com] Sent: 14 February 2017 21:05 To: Wido den Hollander Cc: Dongsheng Yang ; Nick Fisk ; Ceph Users Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander wrote: Op 14 februari 2017 om 11:14 schreef Nick Fisk : -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dongsheng Yang Sent: 14 February 2017 09:01 To: Sage Weil Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com Subject: [ceph-users] bcache vs flashcache vs cache tiering Hi Sage and all, We are going to use SSDs for cache in ceph. But I am not sure which one is the best solution, bcache? flashcache? or cache tier? I would vote for cache tier. Being able to manage it from within Ceph, instead of having to manage X number of bcache/flashcache instances, appeals to me more. Also last time I looked Flashcache seems unmaintained and bcache might be going that way with talk of this new bcachefs. Another point to consider is that Ceph has had a lot of work done on it to ensure data consistency; I don't ever want to be in a position where I'm trying to diagnose problems that might be being caused by another layer sitting in-between Ceph and the Disk. However, I know several people on here are using bcache and potentially getting better performance than with cache tiering, so hopefully someone will give their views. I am using Bcache on various systems and it performs really well. The caching layer in Ceph is slow. Promoting Objects is slow and it also involves additional RADOS lookups. Yeah. Cache tiers have gotten a lot more usable in Ceph, but the use cases where they're effective are still pretty limited and I think in-node caching has a brighter future. We just don't like to maintain the global state that makes separate caching locations viable and unless you're doing something analogous to the supercomputing "burst buffers" (which some people are!), it's going to be hard to beat something that doesn't have to pay the cost of extra network hops/bandwidth. Cache tiers are also not a feature that all the vendors support in their downstream products, so it will probably see less ongoing investment than you'd expect from such a system. Should that be taken as an unofficial sign that the tiering support is likely to fade away? I think both approaches have different strengths and probably the difference between a tiering system and a caching one is what causes some of the problems. If something like bcache is going to be the preferred approach, then I think more work needs to be done around certifying it for use with Ceph and allowing its behavior to be more controlled by Ceph as well. I assume there are issues around backfilling and scrubbing polluting the cache? Maybe you would want to be able to pass hints down from Ceph, which could also allow per pool cache behavior?? -Greg ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bcache vs flashcache vs cache tiering
I've been testing flashcache, bcache, dm-cache and even dm-writeboost in production ceph clusters. The only one that is working fine and gives the speed we need is bcache. All others failed with slow speeds or low latencies. Stefan Excuse my typo sent from my mobile phone. > Am 15.02.2017 um 02:42 schrieb Christian Balzer : > > On Tue, 14 Feb 2017 22:42:21 - Nick Fisk wrote: > >>> -Original Message- >>> From: Gregory Farnum [mailto:gfar...@redhat.com] >>> Sent: 14 February 2017 21:05 >>> To: Wido den Hollander >>> Cc: Dongsheng Yang ; Nick Fisk >>> ; Ceph Users >>> Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering >>> >>> On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander >>> wrote: >>>> >>>>> Op 14 februari 2017 om 11:14 schreef Nick Fisk : >>>>> >>>>> >>>>>> -Original Message- >>>>>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On >>>>>> Behalf Of Dongsheng Yang >>>>>> Sent: 14 February 2017 09:01 >>>>>> To: Sage Weil >>>>>> Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com >>>>>> Subject: [ceph-users] bcache vs flashcache vs cache tiering >>>>>> >>>>>> Hi Sage and all, >>>>>> We are going to use SSDs for cache in ceph. But I am not sure >>>>>> which one is the best solution, bcache? flashcache? or cache >>>>> tier? >>>>> >>>>> I would vote for cache tier. Being able to manage it from within >>>>> Ceph, instead of having to manage X number of bcache/flashcache >>>>> instances, appeals to me more. Also last time I looked Flashcache >>>>> seems unmaintained and bcache might be going that way with talk of >>>>> this new bcachefs. Another point to consider is that Ceph has had a lot >>>>> of >>> work done on it to ensure data consistency; I don't ever want to be in a >>> position where I'm trying to diagnose problems that might be being caused >>> by another layer sitting in-between Ceph and the Disk. >>>>> >>>>> However, I know several people on here are using bcache and >>>>> potentially getting better performance than with cache tiering, so >>> hopefully someone will give their views. >>>> >>>> I am using Bcache on various systems and it performs really well. The >>> caching layer in Ceph is slow. Promoting Objects is slow and it also >>> involves >>> additional RADOS lookups. >>> >>> Yeah. Cache tiers have gotten a lot more usable in Ceph, but the use cases >>> where they're effective are still pretty limited and I think in-node >>> caching has >>> a brighter future. We just don't like to maintain the global state that >>> makes >>> separate caching locations viable and unless you're doing something >>> analogous to the supercomputing "burst buffers" (which some people are!), >>> it's going to be hard to beat something that doesn't have to pay the cost of >>> extra network hops/bandwidth. >>> Cache tiers are also not a feature that all the vendors support in their >>> downstream products, so it will probably see less ongoing investment than >>> you'd expect from such a system. >> >> Should that be taken as an unofficial sign that the tiering support is >> likely to fade away? >> > Nick, you also posted back in October in the > "cache tiering deprecated in RHCS 2.0" thread and should remember the > deafening silence when I asked that question. > > I'm actually surprised that Greg said as much as he did now, > unfortunately that doesn't really cover all the questions I had back then, > in particular long term support and bug fixes, not necessarily more > features. > > We're literally about to order our next cluster and cache-tiering works > like a charm for us, even in Hammer. > With the (still undocumented) knobs in Jewel and read-forward it will be > even more effective. > > So given the lack of any statements that next cluster will still use the > same design as the previous one, since BlueStore isn't ready, bcache and > others haven't been tested here to my satisfaction and we know very well > what works and what not. > > So 3 regular (HDD OSD, journal SSD) nodes and 3 cache-tier ones. Dedicated > cache
Re: [ceph-users] bcache vs flashcache vs cache tiering
On Tue, 14 Feb 2017 22:42:21 - Nick Fisk wrote: > > -Original Message- > > From: Gregory Farnum [mailto:gfar...@redhat.com] > > Sent: 14 February 2017 21:05 > > To: Wido den Hollander > > Cc: Dongsheng Yang ; Nick Fisk > > ; Ceph Users > > Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering > > > > On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander > > wrote: > > > > > >> Op 14 februari 2017 om 11:14 schreef Nick Fisk : > > >> > > >> > > >> > -Original Message- > > >> > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On > > >> > Behalf Of Dongsheng Yang > > >> > Sent: 14 February 2017 09:01 > > >> > To: Sage Weil > > >> > Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com > > >> > Subject: [ceph-users] bcache vs flashcache vs cache tiering > > >> > > > >> > Hi Sage and all, > > >> > We are going to use SSDs for cache in ceph. But I am not sure > > >> > which one is the best solution, bcache? flashcache? or cache > > >> tier? > > >> > > >> I would vote for cache tier. Being able to manage it from within > > >> Ceph, instead of having to manage X number of bcache/flashcache > > >> instances, appeals to me more. Also last time I looked Flashcache > > >> seems unmaintained and bcache might be going that way with talk of > > >> this new bcachefs. Another point to consider is that Ceph has had a lot > > >> of > > work done on it to ensure data consistency; I don't ever want to be in a > > position where I'm trying to diagnose problems that might be being caused > > by another layer sitting in-between Ceph and the Disk. > > >> > > >> However, I know several people on here are using bcache and > > >> potentially getting better performance than with cache tiering, so > > hopefully someone will give their views. > > > > > > I am using Bcache on various systems and it performs really well. The > > caching layer in Ceph is slow. Promoting Objects is slow and it also > > involves > > additional RADOS lookups. > > > > Yeah. Cache tiers have gotten a lot more usable in Ceph, but the use cases > > where they're effective are still pretty limited and I think in-node > > caching has > > a brighter future. We just don't like to maintain the global state that > > makes > > separate caching locations viable and unless you're doing something > > analogous to the supercomputing "burst buffers" (which some people are!), > > it's going to be hard to beat something that doesn't have to pay the cost of > > extra network hops/bandwidth. > > Cache tiers are also not a feature that all the vendors support in their > > downstream products, so it will probably see less ongoing investment than > > you'd expect from such a system. > > Should that be taken as an unofficial sign that the tiering support is likely > to fade away? > Nick, you also posted back in October in the "cache tiering deprecated in RHCS 2.0" thread and should remember the deafening silence when I asked that question. I'm actually surprised that Greg said as much as he did now, unfortunately that doesn't really cover all the questions I had back then, in particular long term support and bug fixes, not necessarily more features. We're literally about to order our next cluster and cache-tiering works like a charm for us, even in Hammer. With the (still undocumented) knobs in Jewel and read-forward it will be even more effective. So given the lack of any statements that next cluster will still use the same design as the previous one, since BlueStore isn't ready, bcache and others haven't been tested here to my satisfaction and we know very well what works and what not. So 3 regular (HDD OSD, journal SSD) nodes and 3 cache-tier ones. Dedicated cache-tier nodes allow for deployment of high end CPUs only in those nodes. Another point in favor of cache-tiering is that it can be added at a later stage, while in-node caching requires an initial design with large local SSDs/NVMes or at least the space for them. Because the journal SSDs most people will deploy initially don't tend to be large enough to be effective when used with bcache or similar. > I think both approaches have different strengths and probably the difference > between a tiering system and a caching one is what causes some of the > problems. > > If something like b
Re: [ceph-users] bcache vs flashcache vs cache tiering
> -Original Message- > From: Gregory Farnum [mailto:gfar...@redhat.com] > Sent: 14 February 2017 21:05 > To: Wido den Hollander > Cc: Dongsheng Yang ; Nick Fisk > ; Ceph Users > Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering > > On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander > wrote: > > > >> Op 14 februari 2017 om 11:14 schreef Nick Fisk : > >> > >> > >> > -Original Message- > >> > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On > >> > Behalf Of Dongsheng Yang > >> > Sent: 14 February 2017 09:01 > >> > To: Sage Weil > >> > Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com > >> > Subject: [ceph-users] bcache vs flashcache vs cache tiering > >> > > >> > Hi Sage and all, > >> > We are going to use SSDs for cache in ceph. But I am not sure > >> > which one is the best solution, bcache? flashcache? or cache > >> tier? > >> > >> I would vote for cache tier. Being able to manage it from within > >> Ceph, instead of having to manage X number of bcache/flashcache > >> instances, appeals to me more. Also last time I looked Flashcache > >> seems unmaintained and bcache might be going that way with talk of > >> this new bcachefs. Another point to consider is that Ceph has had a lot of > work done on it to ensure data consistency; I don't ever want to be in a > position where I'm trying to diagnose problems that might be being caused > by another layer sitting in-between Ceph and the Disk. > >> > >> However, I know several people on here are using bcache and > >> potentially getting better performance than with cache tiering, so > hopefully someone will give their views. > > > > I am using Bcache on various systems and it performs really well. The > caching layer in Ceph is slow. Promoting Objects is slow and it also involves > additional RADOS lookups. > > Yeah. Cache tiers have gotten a lot more usable in Ceph, but the use cases > where they're effective are still pretty limited and I think in-node caching > has > a brighter future. We just don't like to maintain the global state that makes > separate caching locations viable and unless you're doing something > analogous to the supercomputing "burst buffers" (which some people are!), > it's going to be hard to beat something that doesn't have to pay the cost of > extra network hops/bandwidth. > Cache tiers are also not a feature that all the vendors support in their > downstream products, so it will probably see less ongoing investment than > you'd expect from such a system. Should that be taken as an unofficial sign that the tiering support is likely to fade away? I think both approaches have different strengths and probably the difference between a tiering system and a caching one is what causes some of the problems. If something like bcache is going to be the preferred approach, then I think more work needs to be done around certifying it for use with Ceph and allowing its behavior to be more controlled by Ceph as well. I assume there are issues around backfilling and scrubbing polluting the cache? Maybe you would want to be able to pass hints down from Ceph, which could also allow per pool cache behavior?? > -Greg ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bcache vs flashcache vs cache tiering
On Tue, Feb 14, 2017 at 8:25 AM, Wido den Hollander wrote: > >> Op 14 februari 2017 om 11:14 schreef Nick Fisk : >> >> >> > -Original Message- >> > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >> > Dongsheng Yang >> > Sent: 14 February 2017 09:01 >> > To: Sage Weil >> > Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com >> > Subject: [ceph-users] bcache vs flashcache vs cache tiering >> > >> > Hi Sage and all, >> > We are going to use SSDs for cache in ceph. But I am not sure which >> > one is the best solution, bcache? flashcache? or cache >> tier? >> >> I would vote for cache tier. Being able to manage it from within Ceph, >> instead of having to manage X number of bcache/flashcache >> instances, appeals to me more. Also last time I looked Flashcache seems >> unmaintained and bcache might be going that way with talk of >> this new bcachefs. Another point to consider is that Ceph has had a lot of >> work done on it to ensure data consistency; I don't ever >> want to be in a position where I'm trying to diagnose problems that might be >> being caused by another layer sitting in-between Ceph >> and the Disk. >> >> However, I know several people on here are using bcache and potentially >> getting better performance than with cache tiering, so >> hopefully someone will give their views. > > I am using Bcache on various systems and it performs really well. The caching > layer in Ceph is slow. Promoting Objects is slow and it also involves > additional RADOS lookups. Yeah. Cache tiers have gotten a lot more usable in Ceph, but the use cases where they're effective are still pretty limited and I think in-node caching has a brighter future. We just don't like to maintain the global state that makes separate caching locations viable and unless you're doing something analogous to the supercomputing "burst buffers" (which some people are!), it's going to be hard to beat something that doesn't have to pay the cost of extra network hops/bandwidth. Cache tiers are also not a feature that all the vendors support in their downstream products, so it will probably see less ongoing investment than you'd expect from such a system. -Greg ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bcache vs flashcache vs cache tiering
We are running flashcache in production for RBD behind OSDs since over two years now. We had a few issues with it: • one rare kernel livelock between XFS and flashcache that took some effort to track down and fix (we could release patched flashcache if there is interest) • careful tuning of skip seq threshold so cache is not tainted with deep scrub and backfilling • flashcache does not support FUA so hdd write cache has to be disabled to prevent data loss in case of power failure (results in ~5% performance drop in our case) • XFS should be formatted with forced 4kb sector • requires patching for newer kernels because it's no longer maintained Overall it allows us to get ~40% more IOPS from hdds than if we had not used it. Whether you should use it or not really depends on the use case. If your hot working set size fits in cache then it could give you a lot of performance increase. -- Tomasz Kuzemko tomasz.kuze...@corp.ovh.com Dnia 14.02.2017 o godz. 17:25 Wido den Hollander napisał(a): > >> Op 14 februari 2017 om 11:14 schreef Nick Fisk : >> >> >>> -Original Message- >>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >>> Dongsheng Yang >>> Sent: 14 February 2017 09:01 >>> To: Sage Weil >>> Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com >>> Subject: [ceph-users] bcache vs flashcache vs cache tiering >>> >>> Hi Sage and all, >>> We are going to use SSDs for cache in ceph. But I am not sure which one >>> is the best solution, bcache? flashcache? or cache >> tier? >> >> I would vote for cache tier. Being able to manage it from within Ceph, >> instead of having to manage X number of bcache/flashcache >> instances, appeals to me more. Also last time I looked Flashcache seems >> unmaintained and bcache might be going that way with talk of >> this new bcachefs. Another point to consider is that Ceph has had a lot of >> work done on it to ensure data consistency; I don't ever >> want to be in a position where I'm trying to diagnose problems that might be >> being caused by another layer sitting in-between Ceph >> and the Disk. >> >> However, I know several people on here are using bcache and potentially >> getting better performance than with cache tiering, so >> hopefully someone will give their views. > > I am using Bcache on various systems and it performs really well. The caching > layer in Ceph is slow. Promoting Objects is slow and it also involves > additional RADOS lookups. > > The benefit with bcache is that it's handled by the OS locally, see it being > a extension of the page cache. > > A Fast NVM-e device of 1 to 2TB can vastly improve the performance of a bunch > of spinning disks. What I've seen is that overall the I/O pattern on the > disks stabilizes and has less spikes. > > Frequent reads will be cached in the page cache and less frequent by bcache. > > Running this with a few clients now for over 18 months and no issues so far. > > Starting from kernel 4.11 you can also create partitions on bcache devices > which makes it very easy to use bcache with ceph-disk and thus FileStore and > BlueStore. > > Wido > >> >>> >>> I found there are some CAUTION in ceph.com about cache tiering. Is cache >>> tiering is already production ready? especially for rbd. >> >> Several people have been using it in production and with Jewel I would say >> it's stable. There were a few gotcha's in previous >> releases, but they all appear to be fixed in Jewel. The main reasons for the >> warnings now are that unless you have a cacheable >> workload, performance can actually be degraded. If you can predict that say >> 10% of your data will be hot and provision enough SSD >> capacity for this hot data, then it should work really well. If you data >> will be uniformly random or sequential in nature, then I >> would steer clear, but this applies to most caching solutions albeit with >> maybe more graceful degradation >> >>> >>> thanx in advance. >>> Yang >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bcache vs flashcache vs cache tiering
> -Original Message- > From: Wido den Hollander [mailto:w...@42on.com] > Sent: 14 February 2017 16:25 > To: Dongsheng Yang ; n...@fisk.me.uk > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] bcache vs flashcache vs cache tiering > > > > Op 14 februari 2017 om 11:14 schreef Nick Fisk : > > > > > > > -Original Message- > > > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On > > > Behalf Of Dongsheng Yang > > > Sent: 14 February 2017 09:01 > > > To: Sage Weil > > > Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com > > > Subject: [ceph-users] bcache vs flashcache vs cache tiering > > > > > > Hi Sage and all, > > > We are going to use SSDs for cache in ceph. But I am not sure > > > which one is the best solution, bcache? flashcache? or cache > > tier? > > > > I would vote for cache tier. Being able to manage it from within Ceph, > > instead of having to manage X number of bcache/flashcache instances, > > appeals to me more. Also last time I looked Flashcache seems > > unmaintained and bcache might be going that way with talk of this new > > bcachefs. Another point to consider is that Ceph has had a lot of work done > > on it to ensure data consistency; I don't ever want to be > in a position where I'm trying to diagnose problems that might be being > caused by another layer sitting in-between Ceph and the Disk. > > > > However, I know several people on here are using bcache and > > potentially getting better performance than with cache tiering, so > > hopefully someone will give their views. > > I am using Bcache on various systems and it performs really well. The caching > layer in Ceph is slow. Promoting Objects is slow and it > also involves additional RADOS lookups. > > The benefit with bcache is that it's handled by the OS locally, see it being > a extension of the page cache. > > A Fast NVM-e device of 1 to 2TB can vastly improve the performance of a bunch > of spinning disks. What I've seen is that overall the > I/O pattern on the disks stabilizes and has less spikes. > > Frequent reads will be cached in the page cache and less frequent by bcache. > > Running this with a few clients now for over 18 months and no issues so far. > > Starting from kernel 4.11 you can also create partitions on bcache devices > which makes it very easy to use bcache with ceph-disk and > thus FileStore and BlueStore. Thanks for the input Wido. So I assume you currently run with the Journals on separate raw SSD partitions, but post 4.11 you will allow ceph-disk to partition a single bcache device for both data and journal? Have you seen any quirks with bcache over the time you have been using it? I know when I 1st looked at it for non-ceph use a few years back it had a few gremlins hidden in it. Nick > > Wido > > > > > > > > > I found there are some CAUTION in ceph.com about cache tiering. Is cache > > > tiering is already production ready? especially for rbd. > > > > Several people have been using it in production and with Jewel I would > > say it's stable. There were a few gotcha's in previous releases, but > > they all appear to be fixed in Jewel. The main reasons for the > > warnings now are that unless you have a cacheable workload, > > performance can actually be degraded. If you can predict that say 10% > > of your data will be hot and provision enough SSD capacity for this > > hot data, then it should work really well. If you data will be > > uniformly random or sequential in nature, then I would steer clear, > > but this applies to most caching solutions albeit with maybe more > > graceful degradation > > > > > > > > thanx in advance. > > > Yang > > > ___ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bcache vs flashcache vs cache tiering
> Op 14 februari 2017 om 11:14 schreef Nick Fisk : > > > > -Original Message- > > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > > Dongsheng Yang > > Sent: 14 February 2017 09:01 > > To: Sage Weil > > Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com > > Subject: [ceph-users] bcache vs flashcache vs cache tiering > > > > Hi Sage and all, > > We are going to use SSDs for cache in ceph. But I am not sure which > > one is the best solution, bcache? flashcache? or cache > tier? > > I would vote for cache tier. Being able to manage it from within Ceph, > instead of having to manage X number of bcache/flashcache > instances, appeals to me more. Also last time I looked Flashcache seems > unmaintained and bcache might be going that way with talk of > this new bcachefs. Another point to consider is that Ceph has had a lot of > work done on it to ensure data consistency; I don't ever > want to be in a position where I'm trying to diagnose problems that might be > being caused by another layer sitting in-between Ceph > and the Disk. > > However, I know several people on here are using bcache and potentially > getting better performance than with cache tiering, so > hopefully someone will give their views. I am using Bcache on various systems and it performs really well. The caching layer in Ceph is slow. Promoting Objects is slow and it also involves additional RADOS lookups. The benefit with bcache is that it's handled by the OS locally, see it being a extension of the page cache. A Fast NVM-e device of 1 to 2TB can vastly improve the performance of a bunch of spinning disks. What I've seen is that overall the I/O pattern on the disks stabilizes and has less spikes. Frequent reads will be cached in the page cache and less frequent by bcache. Running this with a few clients now for over 18 months and no issues so far. Starting from kernel 4.11 you can also create partitions on bcache devices which makes it very easy to use bcache with ceph-disk and thus FileStore and BlueStore. Wido > > > > > I found there are some CAUTION in ceph.com about cache tiering. Is cache > > tiering is already production ready? especially for rbd. > > Several people have been using it in production and with Jewel I would say > it's stable. There were a few gotcha's in previous > releases, but they all appear to be fixed in Jewel. The main reasons for the > warnings now are that unless you have a cacheable > workload, performance can actually be degraded. If you can predict that say > 10% of your data will be hot and provision enough SSD > capacity for this hot data, then it should work really well. If you data will > be uniformly random or sequential in nature, then I > would steer clear, but this applies to most caching solutions albeit with > maybe more graceful degradation > > > > > thanx in advance. > > Yang > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bcache vs flashcache vs cache tiering
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Dongsheng Yang > Sent: 14 February 2017 09:01 > To: Sage Weil > Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com > Subject: [ceph-users] bcache vs flashcache vs cache tiering > > Hi Sage and all, > We are going to use SSDs for cache in ceph. But I am not sure which one > is the best solution, bcache? flashcache? or cache tier? I would vote for cache tier. Being able to manage it from within Ceph, instead of having to manage X number of bcache/flashcache instances, appeals to me more. Also last time I looked Flashcache seems unmaintained and bcache might be going that way with talk of this new bcachefs. Another point to consider is that Ceph has had a lot of work done on it to ensure data consistency; I don't ever want to be in a position where I'm trying to diagnose problems that might be being caused by another layer sitting in-between Ceph and the Disk. However, I know several people on here are using bcache and potentially getting better performance than with cache tiering, so hopefully someone will give their views. > > I found there are some CAUTION in ceph.com about cache tiering. Is cache > tiering is already production ready? especially for rbd. Several people have been using it in production and with Jewel I would say it's stable. There were a few gotcha's in previous releases, but they all appear to be fixed in Jewel. The main reasons for the warnings now are that unless you have a cacheable workload, performance can actually be degraded. If you can predict that say 10% of your data will be hot and provision enough SSD capacity for this hot data, then it should work really well. If you data will be uniformly random or sequential in nature, then I would steer clear, but this applies to most caching solutions albeit with maybe more graceful degradation > > thanx in advance. > Yang > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
