Re: Large slab class
Look for cases in the code for 'it_flags & ITEM_CHUNKED'. There are a few cases where the data is looped through. (append/prepend code).On Jun 6, 2023, at 9:46 PM, boaz shavit wrote:Thanks a lot
Dormando.We have made some customization to the code and we are using ITEM_data macro in order to retrieve the value of the key.This work for all slabs except the last one. Is there any way to programmatically retrieve the data ?Thanks,Bob. On Tuesday, June 6, 2023 at 11:22:44 PM UTC+3 dormando wrote:Hey,
Items larger than the slab class max are "chunked" across multiple slab
chunks. See: https://github.com/memcached/memcached/wiki/ReleaseNotes1429
Since that release a "cap" chunk mode was added, so if chunk max is set to
16k and you store a 17k item, it will split into:
1) tiny chunk for key and header
2) 16k main chunk
3) attempt to "cap" with a 1k chunk
if it cannot allocate the 1k cap memory for some reason it will allocate
an extra 16k chunk instead.
The theory is that at larger item sizes we can spend a little CPU to
improve the memory efficiency. Making the "max slab class" smaller means
we can make better use of the slab classes. At some point I will be
reducing the default setting from 512k to 256k or lower, but I need to
revisit it and add some stats counters first.
-Dormando
On Tue, 6 Jun 2023, boaz shavit wrote:
> Hello,I'm trying to understand how data is saved in memcached for items with size > 0.5M.
> When I check the slabclass structure array, I see it only has values for classes up to .5 MB and another entry in place zero which looks like this:
> (gdb) p slabclass[0]
> $87 = {size = 0, perslab = 0, slots = 0x0, sl_curr = 0, slabs = 386, slab_list = 0x7f15b4015ef0, list_size = 2048}
>
> when inserting a value which is very big (the key is 100 bytes, but the value is 800k) I see that the key goes to class 2 (which is 196 bytes) but I do
> not see where the value is stored.
>
> Can someone explain how this works for big values?
>
> Thanks in advanced,
> Bob.
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/dedecb91-b3d2-4a83-8253-0ad965cecd68n%40googlegroups.com.
>
>
--
---
You received this message because you are subscribed to the Google Groups "memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/e2254937-3369-4cc2-9de5-6982d1ef9066n%40googlegroups.com.
--
---
You received this message because you are subscribed to the Google Groups "memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/4D53899E-5268-4A88-9CED-62F18A758078%40rydia.net.
Re: Large slab class
Thanks a lot Dormando.
We have made some customization to the code and we are using ITEM_data
macro in order to retrieve the value of the key.
This work for all slabs except the last one. Is there any way to
programmatically retrieve the data ?
Thanks,
Bob.
On Tuesday, June 6, 2023 at 11:22:44 PM UTC+3 dormando wrote:
> Hey,
>
> Items larger than the slab class max are "chunked" across multiple slab
> chunks. See: https://github.com/memcached/memcached/wiki/ReleaseNotes1429
>
> Since that release a "cap" chunk mode was added, so if chunk max is set to
> 16k and you store a 17k item, it will split into:
>
> 1) tiny chunk for key and header
> 2) 16k main chunk
> 3) attempt to "cap" with a 1k chunk
>
> if it cannot allocate the 1k cap memory for some reason it will allocate
> an extra 16k chunk instead.
>
> The theory is that at larger item sizes we can spend a little CPU to
> improve the memory efficiency. Making the "max slab class" smaller means
> we can make better use of the slab classes. At some point I will be
> reducing the default setting from 512k to 256k or lower, but I need to
> revisit it and add some stats counters first.
>
> -Dormando
>
> On Tue, 6 Jun 2023, boaz shavit wrote:
>
> > Hello,I'm trying to understand how data is saved in memcached for items
> with size > 0.5M.
> > When I check the slabclass structure array, I see it only has values for
> classes up to .5 MB and another entry in place zero which looks like this:
> > (gdb) p slabclass[0]
> > $87 = {size = 0, perslab = 0, slots = 0x0, sl_curr = 0, slabs = 386,
> slab_list = 0x7f15b4015ef0, list_size = 2048}
> >
> > when inserting a value which is very big (the key is 100 bytes, but the
> value is 800k) I see that the key goes to class 2 (which is 196 bytes) but
> I do
> > not see where the value is stored.
> >
> > Can someone explain how this works for big values?
> >
> > Thanks in advanced,
> > Bob.
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google
> Groups "memcached" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to memcached+...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/memcached/dedecb91-b3d2-4a83-8253-0ad965cecd68n%40googlegroups.com
> .
> >
> >
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to memcached+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/memcached/e2254937-3369-4cc2-9de5-6982d1ef9066n%40googlegroups.com.
Re: Large slab class
Hey,
Items larger than the slab class max are "chunked" across multiple slab
chunks. See: https://github.com/memcached/memcached/wiki/ReleaseNotes1429
Since that release a "cap" chunk mode was added, so if chunk max is set to
16k and you store a 17k item, it will split into:
1) tiny chunk for key and header
2) 16k main chunk
3) attempt to "cap" with a 1k chunk
if it cannot allocate the 1k cap memory for some reason it will allocate
an extra 16k chunk instead.
The theory is that at larger item sizes we can spend a little CPU to
improve the memory efficiency. Making the "max slab class" smaller means
we can make better use of the slab classes. At some point I will be
reducing the default setting from 512k to 256k or lower, but I need to
revisit it and add some stats counters first.
-Dormando
On Tue, 6 Jun 2023, boaz shavit wrote:
> Hello,I'm trying to understand how data is saved in memcached for items with
> size > 0.5M.
> When I check the slabclass structure array, I see it only has values for
> classes up to .5 MB and another entry in place zero which looks like this:
> (gdb) p slabclass[0]
> $87 = {size = 0, perslab = 0, slots = 0x0, sl_curr = 0, slabs = 386,
> slab_list = 0x7f15b4015ef0, list_size = 2048}
>
> when inserting a value which is very big (the key is 100 bytes, but the value
> is 800k) I see that the key goes to class 2 (which is 196 bytes) but I do
> not see where the value is stored.
>
> Can someone explain how this works for big values?
>
> Thanks in advanced,
> Bob.
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to memcached+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/memcached/dedecb91-b3d2-4a83-8253-0ad965cecd68n%40googlegroups.com.
>
>
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to memcached+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/memcached/f9c9451b-9256-7cf3-fdf-423cbd51796d%40rydia.net.
Large slab class
Hello,
I'm trying to understand how data is saved in memcached for items with size
> 0.5M.
When I check the slabclass structure array, I see it only has values for
classes up to .5 MB and another entry in place zero which looks like this:
(gdb) p slabclass[0]
*$87 = {size = 0, perslab = 0, slots = 0x0, sl_curr = 0, slabs = 386,
slab_list = 0x7f15b4015ef0, list_size = 2048}*
when inserting a value which is very big (the key is 100 bytes, but the
value is 800k) I see that the key goes to class 2 (which is 196 bytes) but
I do not see where the value is stored.
Can someone explain how this works for big values?
Thanks in advanced,
Bob.
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to memcached+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/memcached/dedecb91-b3d2-4a83-8253-0ad965cecd68n%40googlegroups.com.
