(memory_requested / (chunk_size * chunk_used)) * 100
is roughly the storage overhead of memory used in the system. the rest is
free memory, which should be measured separately. If you're getting
evictions in class A but there's too much free memory in classes C, D, etc
then you have a balance issue. for example. An efficiency stat which just
adds up the total pages doesn't tell you what to do with it.
On Sat, 4 Jul 2020, Shweta Agrawal wrote:
> > I'll need the raw output from "stats items" and "stats slabs". I don't
> > think that efficiency column is very helpful. ohkay no worries. I can get
> > by Tuesday and will share.
>
> Efficiency for each slab is calcuated as
> (("stats slabs" -> memory_requested) / (("stats slabs" -> total_pages) *
> page_size)) * 100
>
>
> Attaching script which has calculations for the same. The script is from
> memcahe repo with additional calculation for efficiency.
> Will it be possible for you to verify if the efficiency calculation is
> correct?
>
> Thank you,
> Shweta
>
> On Saturday, July 4, 2020 at 1:08:23 PM UTC+5:30, Dormando wrote:
> ah okay.
>
> I'll need the raw output from "stats items" and "stats slabs". I don't
> think that efficiency column is very helpful.
>
> On Fri, 3 Jul 2020, Shweta Agrawal wrote:
>
> >
> >
> > On Saturday, July 4, 2020 at 9:41:49 AM UTC+5:30, Dormando wrote:
> > No attachment
> >
> > On Fri, 3 Jul 2020, Shweta Agrawal wrote:
> >
> > >
> > > Wooo...so quick. :):)
> > > > Correct, close. It actually uses more like 3 512k chunks
> and then one
> > > > smaller chunk from a different class to fit exactly 1.6MB.
> > > I see.Got it.
> > >
> > > >Can you share snapshots from "stats items" and "stats slabs"
> for one of
> > > these instances?
> > >
> > > Currently I have summary of it, sharing the same below. I can
> get snapshot by Tuesday as need to request for it.
> > >
> > > pages have value from total_pages from stats slab for each
> slab
> > > item_size have value from chunk_size from stats slab for each
> slab
> > > Used memory is calculated as pages*page size ---> This has to
> corrected now.
> > >
> > >
> > > prod_stats.png
> > >
> > >
> > > > 90%+ are perfectly doable. You probably need to look a bit
> more closely
> > > > into why you're not getting the efficiency you expect. The
> detailed stats
> > > > output should point to why. I can help with that if it's
> confusing.
> > >
> > > Great. Will surely ask for your input whenever I have
> question. It is really kind of you to offer help.
> > >
> > > > Either the slab rebalancer isn't keeping up or you actually
> do have 39GB
> > > > of data and your expecations are a bit off. This will also
> depending on
> > > > the TTL's you're setting and how often/quickly your items
> change size.
> > > > Also things like your serialization method / compression /
> key length vs
> > > > data length / etc.
> > >
> > > We have much less data than 39 GB. As after facing evictions,
> it has been always kept higher than expected data-size.
> > > TTL is two days or more.
> > > From my observation items size(data-length) is in the range
> of 300Bytes to 500K after compression.
> > > Key length is in the range of 40-80 bytes.
> > >
> > > Thank you,
> > > Shweta
> > >
> > > On Saturday, July 4, 2020 at 8:38:31 AM UTC+5:30, Dormando
> wrote:
> > > Hey,
> > >
> > > > Putting my understanding to re-confirm:
> > > > 1) Page size will always be 1MB and we cannot change
> it.Moreover, it's not required to be changed.
> > >
> > > Correct.
> > >
> > > > 2) We can store items larger than 1MB and it is done
> by combining chunks together. (example: let's say item size: ~1.6MB -->
> 4 slab
> > > chunks(512k slab) from
> > > > 2 pages will be used)
> > >
> > > Correct, close. It actually uses more like 3 512k
> chunks and then one
> > > smaller chunk from a different class to fit exactly
> 1.6MB.
> > >
> > > > We use memcache in production and in past we saw
> evictions even when free memory was present. Also currently we use cluster
> with
> > 39GB RAM in
> > > total to
> > > > cache data even when data size we expect is ~15GB to
> avoid eviction of active items.
> > >
> > > Can you share snapshots from "stats items" and "stats
> slabs" for one of
> > > these instances?
> > >
> > > > But as our data varies in size, it is possible to
> avoid evictions by tuning parameters: chunk_size, growth_factor,
> slab_automove.
> > Also I
> > > believe memcache
> > > > is efficient and we can reduce cost by reducing
> memory size for cluster.
> > > > So I am trying to find the best possible memory size
> and parameters we can have.So want to be clear with my understanding
> and
> > calculations.
> > > >
> > > > So while trying different parameters and putting all
> calculations, I observed that total_pages * item_size_max > physical
> memory for
> > a
> > > machine. And from
> > > > all blogs,and docs it didnot match my understanding.
> But it's clear now. Thanks to you.
> > > >
> > > > One last question: >From my trials I find that we can
> achieve ~90% storage efficiency with memcache. (i.e we need 10MB of
> physical
> > memory to
> > > store 9MB of
> > > > data. Do you recommend any idle memory-size interms
> of percentage of expected data-size?
> > >
> > > 90%+ are perfectly doable. You probably need to look a
> bit more closely
> > > into why you're not getting the efficiency you expect.
> The detailed stats
> > > output should point to why. I can help with that if
> it's confusing.
> > >
> > > Either the slab rebalancer isn't keeping up or you
> actually do have 39GB
> > > of data and your expecations are a bit off. This will
> also depending on
> > > the TTL's you're setting and how often/quickly your
> items change size.
> > > Also things like your serialization method /
> compression / key length vs
> > > data length / etc.
> > >
> > > -Dormando
> > >
> > > > On Saturday, July 4, 2020 at 12:23:09 AM UTC+5:30,
> Dormando wrote:
> > > > Hey,
> > > >
> > > > Looks like I never updated the manpage. In the
> past the item size max was
> > > > achieved by changing the slab page size, but
> that hasn't been true for a
> > > > long time.
> > > >
> > > > From ./memcached -h:
> > > > -m, --memory-limit=<num> item memory in
> megabytes (default: 64)
> > > >
> > > > ... -m just means the memory limit in
> megabytes, abstract from the page
> > > > size. I think that was always true.
> > > >
> > > > In any recentish version, any item larger than
> half a page size (512k) is
> > > > created by stitching page chunks together. This
> prevents waste when an
> > > > item would be more than half a page size.
> > > >
> > > > Is there a problem you're trying to track down?
> > > >
> > > > I'll update the manpage.
> > > >
> > > > On Fri, 3 Jul 2020, Shweta Agrawal wrote:
> > > >
> > > > > Hi,
> > > > > Sorry if I am repeating the question, I
> searched the list but could not find definite answer. So posting it.
> > > > >
> > > > > Memcache version: 1.5.10
> > > > > I have started memcahce with option: -I 4m
> (setting maximum item size to 4MB).Verified it is set by command stats
> settings ,
> > I can
> > > see STAT
> > > > item_size_max
> > > > > 4194304.
> > > > >
> > > > > Documentation from git repository here stats
> that:
> > > > >
> > > > > -I, --max-item-size=<size>
> > > > > Override the default size of each slab page.
> The default size is 1mb. Default
> > > > > value for this parameter is 1m, minimum is
> 1k, max is 1G (1024 * 1024 * 1024).
> > > > > Adjusting this value changes the item size
> limit.
> > > > > My understanding from documentation is this
> option will allow to save items with size till 4MB and the page size for
> each
> > slab will
> > > be 4MB
> > > > (as I set it as
> > > > > -I 4m).
> > > > >
> > > > > I am able to save items till 4MB but the
> page-size is still 1MB.
> > > > >
> > > > > -m memory size is default 64MB.
> > > > >
> > > > > Calculation:
> > > > > -> Calculated total pages used from stats
> slabs output parameter total_pages = 64 (If page size is 4MB then total
> pages
> > should not
> > > be more
> > > > than 16. Also
> > > > > when I store 8 items of ~3MB it uses 25 pages
> but if page size is 4MB, it should use 8 pages right.)
> > > > >
> > > > > Can you please help me in understanding the
> behaviour?
> > > > >
> > > > > Attached files with details for output of
> command stats settings and stats slabs.
> > > > > Below is the summarized view of the
> distribution.
> > > > > First added items with variable sizes, then
> then added items with 3MB and above.
> > > > >
> > > > > data_distribution.png
> > > > >
> > > > >
> > > > >
> > > > > Please let me know in case more details are
> required or question is not clear.
> > > > >
> > > > > Thank You,
> > > > > Shweta
> > > > >
> > > > > --
> > > > >
> > > > > ---
> > > > > You received this message because you are
> subscribed to the Google Groups "memcached" group.
> > > > > To unsubscribe from this group and stop
> receiving emails from it, send an email to memc...@googlegroups.com.
> > > > > To view this discussion on the web visit
> > > >
> https://groups.google.com/d/msgid/memcached/2b640e1f-9f59-4432-a930-d830cbe8566do%40googlegroups.com.
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > ---
> > > > You received this message because you are subscribed
> to the Google Groups "memcached" group.
> > > > To unsubscribe from this group and stop receiving
> emails from it, send an email to memc...@googlegroups.com.
> > > > To view this discussion on the web visit
> > >
> https://groups.google.com/d/msgid/memcached/586aad58-c6fb-4ed8-89ce-6b005d59ba12o%40googlegroups.com.
> > > >
> > > >
> > >
> > > prod_stats.png
> > >
> > > --
> > >
> > > ---
> > > You received this message because you are subscribed to the
> Google Groups "memcached" group.
> > > To unsubscribe from this group and stop receiving emails from
> it, send an email to memc...@googlegroups.com.
> > > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/memcached/8d011c1a-deec-463f-a17e-4e9908d97bdfo%40googlegroups.com.
> > >
> > >
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google
> Groups "memcached" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an email to memc...@googlegroups.com.
> > To view this discussion on the web visit
>
> https://groups.google.com/d/msgid/memcached/f0c2bfe1-d65d-4b62-9a87-68fc42446c3do%40googlegroups.com.
> >
> >
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to memcached+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/memcached/bcd4da5a-ae8e-470f-beb9-2705c0f0202ao%40googlegroups.com.
>
>
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to memcached+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.2007041240450.18887%40dskull.
