Hey Mark, Thanks a lot for the info. You should really make a paper of it and post it :)
First of all, I am sorry if I say something wrong, I am still learning about this topic and I am speaking from totally unawareness. Second, I understood that ratios ar a way of controling priorities and they make that bloom filters and indexes don't get page out from cache, which really makes sense. Also the 512MB restriction kind of makes sense but I dont really know if It would make any sense to give more space for rocksdb block cache (like 1GB). I think only testing can resolve that question because I think it really depends on workloads. what I don't understand is why data ins't cache at all, even if there is free space for it. I undertand the importance order would be: bloom filter and index >> metadata >> data, but if there is free space left for data then why not go for it? maybe setting ratios of 0.90 k/v 0.09 metadata and 0.01 data would make more sense. El 11/10/2017 a las 15:44, Mark Nelson escribió: > Hi Jorge, > > I was sort of responsible for all of this. :) > > So basically there are different caches in different places: > > - rocksdb cache > - rocksdb block cache (which can be configured to include filters and > indexes) > - rocksdb compressed block cache > - bluestore onode cache > > The bluestore onode cache is the only one that stores > onode/extent/blob metadata before it is encoded, ie it's bigger but > has lower impact on the CPU. The next step is the regular rocksdb > block cache where we've already encoded the data, but it's not > compressed. Optionally we could also compress the data and then cache > it using rocksdb's compressed block cache. Finally, rocksdb can set > memory aside for bloom filters and indexes but we're configuring those > to go into the block cache so we can get a better accounting for how > memory is being used (otherwise it's difficult to control how much > memory index and filters get). The downside is that bloom filters and > indexes can theoretically get paged out under heavy cache pressure. > We set these to be high priority in the block cache and also pin the > L0 filters/index though to help avoid this. > > In the testing I did earlier this year, what I saw is that in low > memory scenarios it's almost always best to give all of the cache to > rocksdb's block cache. Once you hit about the 512MB mark, we start > seeing bigger gains by giving additional memory to bluestore's onode > cache. So we devised a mechanism where you can decide where to cut > over. It's quite possible that on very fast CPUs it might make sense > ot use rocksdb compressed cache, or possibly if you have a huge number > of objects these ratios might change. The values we have now were > sort of the best jack-of-all-trades values we found. > > Mark > > On 10/11/2017 08:32 AM, Jorge Pinilla López wrote: >> okay, thanks for the explanation, so from the 3GB of Cache (default >> cache for SSD) only a 0.5GB is going to K/V and 2.5 going to metadata. >> >> Is there a way of knowing how much k/v, metadata, data is storing and >> how full cache is so I can adjust my ratios?, I was thinking some ratios >> (like 0.9 k/v, 0.07 meta 0.03 data) but only speculating, I dont have >> any real data. >> >> El 11/10/2017 a las 14:32, Mohamad Gebai escribió: >>> Hi Jorge, >>> >>> On 10/10/2017 07:23 AM, Jorge Pinilla López wrote: >>>> Are .99 KV, .01 MetaData and .0 Data ratios right? they seem a little >>>> too disproporcionate. >>> Yes, this is correct. >>> >>>> Also .99 KV and Cache of 3GB for SSD means that almost the 3GB would >>>> be used for KV but there is also another attributed called >>>> bluestore_cache_kv_max which is by fault 512MB, then what is the rest >>>> of the cache used for?, nothing? shouldnt it be more kv_max value or >>>> less KV ratio? >>> Anything over the *cache_kv_max value goes to the metadata cache. You >>> can look in your logs to see the final values of kv, metadata and data >>> cache ratios. To get data cache, you need to lower the ratios of >>> metadata and kv caches. >>> >>> Mohamad >> >> -- >> ------------------------------------------------------------------------ >> *Jorge Pinilla López* >> [email protected] >> Estudiante de ingenieria informática >> Becario del area de sistemas (SICUZ) >> Universidad de Zaragoza >> PGP-KeyID: A34331932EBC715A >> <http://pgp.rediris.es:11371/pks/lookup?op=get&search=0xA34331932EBC715A> >> >> ------------------------------------------------------------------------ >> >> >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- ------------------------------------------------------------------------ *Jorge Pinilla López* [email protected] Estudiante de ingenieria informática Becario del area de sistemas (SICUZ) Universidad de Zaragoza PGP-KeyID: A34331932EBC715A <http://pgp.rediris.es:11371/pks/lookup?op=get&search=0xA34331932EBC715A> ------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
