Re: Question Regarding Slab Imbalance
Yeah, default is 1.25. We can certainly start with 1.25. Thanks again, Tony Wu > On Nov 12, 2020, at 4:09 PM, dormando wrote: > > what's the default? you probably don't need to change it from the default > at all, which should have a good spread. default is 1.25 I think. > > On Thu, 12 Nov 2020, Tony Wu wrote: > >> Hi, >> >> The engine version we have running right now is 1.5.16. I just saw an >> announcement that 1.6.6 is available, so we’ll probably plan to migrate. >> >> Adjusting growth factor makes sense. I did a quick google and don’t see >> any definitive formula to calculate an appropriate number. I’ll probably >> start with 2 or 3 and adjust it from there. >> >> Thanks, >> >> Tony Wu >> >>> On Nov 12, 2020, at 3:50 PM, dormando wrote: >>> >>> 1.5.what? >>> >>> also yes, chunk_size_growth_factor. don't set it to 1.02. set it to >>> something so your slab classes are more evenly distributed. the max is 63. >>> >>> On Thu, 12 Nov 2020, Tony Wu wrote: >>> Hi Dormando, Thanks for the reply, I believe the engine version is 1.5. By multiplier I assumed you meant “chunk_size_growth_factor”, which is currently set at 1.02. Best, Tony Wu > On Nov 12, 2020, at 2:26 PM, dormando wrote: > > What version are they running now? That stat output looks pretty sparse. > > Unfortunately when it comes to elasticache the answer is probably to just > beg them to upgrade to a newer version. they tend to run really old and > newer verisons do a lot better at balancing memory. > > This is also suspect: >> STAT 62:chunk_size 712 >> STAT 62:total_chunks 4416 >> STAT 62:free_chunks 631 >> STAT 62:free_chunks_end 0 >> STAT 63:chunk_size 524288 > > anything you store that's larger than 712 bytes will use half a megabyte > of memory. If you're setting the slab multiplier (-f) to something really > aggressive, you should stop that :) > > On Thu, 12 Nov 2020, Tony Wu wrote: > >> We are currently using memcached provided by AWS ElastiCache service in >> modern mode to store web session keys >> among other things. We are observing session keys being evicted before >> TTL even though the cluster seems to have >> ample free memory, which hints at slab imbalance. I've included a print >> out of slab stats below in terms of >> sizing and total / free pages. >> What would be the best way to smooth out this imbalance? We already >> enabled slab reassign / auto-move under >> modern mode. >> >> Thanks. >> >> STAT 1:chunk_size 96 >> STAT 1:total_chunks 131064 >> STAT 1:free_chunks 1664 >> STAT 1:free_chunks_end 0 >> STAT 2:chunk_size 104 >> STAT 2:total_chunks 60492 >> STAT 2:free_chunks 5327 >> STAT 2:free_chunks_end 0 >> STAT 3:chunk_size 112 >> STAT 3:total_chunks 46810 >> STAT 3:free_chunks 79 >> STAT 3:free_chunks_end 0 >> STAT 4:chunk_size 120 >> STAT 4:total_chunks 26214 >> STAT 4:free_chunks 48 >> STAT 4:free_chunks_end 0 >> STAT 5:chunk_size 128 >> STAT 5:total_chunks 32768 >> STAT 5:free_chunks 6193 >> STAT 5:free_chunks_end 0 >> STAT 6:chunk_size 136 >> STAT 6:total_chunks 23130 >> STAT 6:free_chunks 4594 >> STAT 6:free_chunks_end 0 >> STAT 7:chunk_size 144 >> STAT 7:total_chunks 7281 >> STAT 7:free_chunks 5894 >> STAT 7:free_chunks_end 0 >> STAT 8:chunk_size 152 >> STAT 8:total_chunks 13796 >> STAT 8:free_chunks 12008 >> STAT 8:free_chunks_end 0 >> STAT 9:chunk_size 160 >> STAT 9:total_chunks 6553 >> STAT 9:free_chunks 6316 >> STAT 9:free_chunks_end 0 >> STAT 10:chunk_size 168 >> STAT 10:total_chunks 6241 >> STAT 10:free_chunks 6186 >> STAT 10:free_chunks_end 0 >> STAT 11:chunk_size 176 >> STAT 11:total_chunks 5957 >> STAT 11:free_chunks 5946 >> STAT 11:free_chunks_end 0 >> STAT 12:chunk_size 184 >> STAT 12:total_chunks 5698 >> STAT 12:free_chunks 5664 >> STAT 12:free_chunks_end 0 >> STAT 13:chunk_size 192 >> STAT 13:total_chunks 5461 >> STAT 13:free_chunks 5380 >> STAT 13:free_chunks_end 0 >> STAT 14:chunk_size 200 >> STAT 14:total_chunks 5242 >> STAT 14:free_chunks 5135 >> STAT 14:free_chunks_end 0 >> STAT 15:chunk_size 208 >> STAT 15:total_chunks 5041 >> STAT 15:free_chunks 4892 >> STAT 15:free_chunks_end 0 >> STAT 16:chunk_size 216 >> STAT 16:total_chunks 4854 >> STAT 16:free_chunks 4770 >> STAT 16:free_chunks_end 0 >> STAT 17:chunk_size 224 >> STAT 17:total_chunks 4681 >> STAT 17:free_chunks 4655 >> STAT 17:free_chunks_end 0 >> STAT 18:chunk_size 232 >> STAT 18:total_chunks 4519 >> STAT 18:free_chunks 4489 >> STAT 18:free_chunks_end 0 >> STAT 19:chunk_size 240 >> STAT 19:total_chunks 4369 >> STAT 19:free_chunks
Re: Question Regarding Slab Imbalance
what's the default? you probably don't need to change it from the default at all, which should have a good spread. default is 1.25 I think. On Thu, 12 Nov 2020, Tony Wu wrote: > Hi, > > The engine version we have running right now is 1.5.16. I just saw an > announcement that 1.6.6 is available, so we’ll probably plan to migrate. > > Adjusting growth factor makes sense. I did a quick google and don’t see > any definitive formula to calculate an appropriate number. I’ll probably > start with 2 or 3 and adjust it from there. > > Thanks, > > Tony Wu > > > On Nov 12, 2020, at 3:50 PM, dormando wrote: > > > > 1.5.what? > > > > also yes, chunk_size_growth_factor. don't set it to 1.02. set it to > > something so your slab classes are more evenly distributed. the max is 63. > > > > On Thu, 12 Nov 2020, Tony Wu wrote: > > > >> Hi Dormando, > >> > >> Thanks for the reply, I believe the engine version is 1.5. By multiplier > >> I assumed you meant “chunk_size_growth_factor”, which is currently set > >> at 1.02. > >> > >> Best, > >> > >> Tony Wu > >> > >>> On Nov 12, 2020, at 2:26 PM, dormando wrote: > >>> > >>> What version are they running now? That stat output looks pretty sparse. > >>> > >>> Unfortunately when it comes to elasticache the answer is probably to just > >>> beg them to upgrade to a newer version. they tend to run really old and > >>> newer verisons do a lot better at balancing memory. > >>> > >>> This is also suspect: > STAT 62:chunk_size 712 > STAT 62:total_chunks 4416 > STAT 62:free_chunks 631 > STAT 62:free_chunks_end 0 > STAT 63:chunk_size 524288 > >>> > >>> anything you store that's larger than 712 bytes will use half a megabyte > >>> of memory. If you're setting the slab multiplier (-f) to something really > >>> aggressive, you should stop that :) > >>> > >>> On Thu, 12 Nov 2020, Tony Wu wrote: > >>> > We are currently using memcached provided by AWS ElastiCache service in > modern mode to store web session keys > among other things. We are observing session keys being evicted before > TTL even though the cluster seems to have > ample free memory, which hints at slab imbalance. I've included a print > out of slab stats below in terms of > sizing and total / free pages. > What would be the best way to smooth out this imbalance? We already > enabled slab reassign / auto-move under > modern mode. > > Thanks. > > STAT 1:chunk_size 96 > STAT 1:total_chunks 131064 > STAT 1:free_chunks 1664 > STAT 1:free_chunks_end 0 > STAT 2:chunk_size 104 > STAT 2:total_chunks 60492 > STAT 2:free_chunks 5327 > STAT 2:free_chunks_end 0 > STAT 3:chunk_size 112 > STAT 3:total_chunks 46810 > STAT 3:free_chunks 79 > STAT 3:free_chunks_end 0 > STAT 4:chunk_size 120 > STAT 4:total_chunks 26214 > STAT 4:free_chunks 48 > STAT 4:free_chunks_end 0 > STAT 5:chunk_size 128 > STAT 5:total_chunks 32768 > STAT 5:free_chunks 6193 > STAT 5:free_chunks_end 0 > STAT 6:chunk_size 136 > STAT 6:total_chunks 23130 > STAT 6:free_chunks 4594 > STAT 6:free_chunks_end 0 > STAT 7:chunk_size 144 > STAT 7:total_chunks 7281 > STAT 7:free_chunks 5894 > STAT 7:free_chunks_end 0 > STAT 8:chunk_size 152 > STAT 8:total_chunks 13796 > STAT 8:free_chunks 12008 > STAT 8:free_chunks_end 0 > STAT 9:chunk_size 160 > STAT 9:total_chunks 6553 > STAT 9:free_chunks 6316 > STAT 9:free_chunks_end 0 > STAT 10:chunk_size 168 > STAT 10:total_chunks 6241 > STAT 10:free_chunks 6186 > STAT 10:free_chunks_end 0 > STAT 11:chunk_size 176 > STAT 11:total_chunks 5957 > STAT 11:free_chunks 5946 > STAT 11:free_chunks_end 0 > STAT 12:chunk_size 184 > STAT 12:total_chunks 5698 > STAT 12:free_chunks 5664 > STAT 12:free_chunks_end 0 > STAT 13:chunk_size 192 > STAT 13:total_chunks 5461 > STAT 13:free_chunks 5380 > STAT 13:free_chunks_end 0 > STAT 14:chunk_size 200 > STAT 14:total_chunks 5242 > STAT 14:free_chunks 5135 > STAT 14:free_chunks_end 0 > STAT 15:chunk_size 208 > STAT 15:total_chunks 5041 > STAT 15:free_chunks 4892 > STAT 15:free_chunks_end 0 > STAT 16:chunk_size 216 > STAT 16:total_chunks 4854 > STAT 16:free_chunks 4770 > STAT 16:free_chunks_end 0 > STAT 17:chunk_size 224 > STAT 17:total_chunks 4681 > STAT 17:free_chunks 4655 > STAT 17:free_chunks_end 0 > STAT 18:chunk_size 232 > STAT 18:total_chunks 4519 > STAT 18:free_chunks 4489 > STAT 18:free_chunks_end 0 > STAT 19:chunk_size 240 > STAT 19:total_chunks 4369 > STAT 19:free_chunks 4345 > STAT 19:free_chunks_end 0 > STAT 20:chunk_size 248 > STAT 20:total_chunks 4228 > STAT 20:free_chunks 4217 > STAT 20:free_chunks_end 0 > STAT
Re: Question Regarding Slab Imbalance
Hi, The engine version we have running right now is 1.5.16. I just saw an announcement that 1.6.6 is available, so we’ll probably plan to migrate. Adjusting growth factor makes sense. I did a quick google and don’t see any definitive formula to calculate an appropriate number. I’ll probably start with 2 or 3 and adjust it from there. Thanks, Tony Wu > On Nov 12, 2020, at 3:50 PM, dormando wrote: > > 1.5.what? > > also yes, chunk_size_growth_factor. don't set it to 1.02. set it to > something so your slab classes are more evenly distributed. the max is 63. > > On Thu, 12 Nov 2020, Tony Wu wrote: > >> Hi Dormando, >> >> Thanks for the reply, I believe the engine version is 1.5. By multiplier >> I assumed you meant “chunk_size_growth_factor”, which is currently set >> at 1.02. >> >> Best, >> >> Tony Wu >> >>> On Nov 12, 2020, at 2:26 PM, dormando wrote: >>> >>> What version are they running now? That stat output looks pretty sparse. >>> >>> Unfortunately when it comes to elasticache the answer is probably to just >>> beg them to upgrade to a newer version. they tend to run really old and >>> newer verisons do a lot better at balancing memory. >>> >>> This is also suspect: STAT 62:chunk_size 712 STAT 62:total_chunks 4416 STAT 62:free_chunks 631 STAT 62:free_chunks_end 0 STAT 63:chunk_size 524288 >>> >>> anything you store that's larger than 712 bytes will use half a megabyte >>> of memory. If you're setting the slab multiplier (-f) to something really >>> aggressive, you should stop that :) >>> >>> On Thu, 12 Nov 2020, Tony Wu wrote: >>> We are currently using memcached provided by AWS ElastiCache service in modern mode to store web session keys among other things. We are observing session keys being evicted before TTL even though the cluster seems to have ample free memory, which hints at slab imbalance. I've included a print out of slab stats below in terms of sizing and total / free pages. What would be the best way to smooth out this imbalance? We already enabled slab reassign / auto-move under modern mode. Thanks. STAT 1:chunk_size 96 STAT 1:total_chunks 131064 STAT 1:free_chunks 1664 STAT 1:free_chunks_end 0 STAT 2:chunk_size 104 STAT 2:total_chunks 60492 STAT 2:free_chunks 5327 STAT 2:free_chunks_end 0 STAT 3:chunk_size 112 STAT 3:total_chunks 46810 STAT 3:free_chunks 79 STAT 3:free_chunks_end 0 STAT 4:chunk_size 120 STAT 4:total_chunks 26214 STAT 4:free_chunks 48 STAT 4:free_chunks_end 0 STAT 5:chunk_size 128 STAT 5:total_chunks 32768 STAT 5:free_chunks 6193 STAT 5:free_chunks_end 0 STAT 6:chunk_size 136 STAT 6:total_chunks 23130 STAT 6:free_chunks 4594 STAT 6:free_chunks_end 0 STAT 7:chunk_size 144 STAT 7:total_chunks 7281 STAT 7:free_chunks 5894 STAT 7:free_chunks_end 0 STAT 8:chunk_size 152 STAT 8:total_chunks 13796 STAT 8:free_chunks 12008 STAT 8:free_chunks_end 0 STAT 9:chunk_size 160 STAT 9:total_chunks 6553 STAT 9:free_chunks 6316 STAT 9:free_chunks_end 0 STAT 10:chunk_size 168 STAT 10:total_chunks 6241 STAT 10:free_chunks 6186 STAT 10:free_chunks_end 0 STAT 11:chunk_size 176 STAT 11:total_chunks 5957 STAT 11:free_chunks 5946 STAT 11:free_chunks_end 0 STAT 12:chunk_size 184 STAT 12:total_chunks 5698 STAT 12:free_chunks 5664 STAT 12:free_chunks_end 0 STAT 13:chunk_size 192 STAT 13:total_chunks 5461 STAT 13:free_chunks 5380 STAT 13:free_chunks_end 0 STAT 14:chunk_size 200 STAT 14:total_chunks 5242 STAT 14:free_chunks 5135 STAT 14:free_chunks_end 0 STAT 15:chunk_size 208 STAT 15:total_chunks 5041 STAT 15:free_chunks 4892 STAT 15:free_chunks_end 0 STAT 16:chunk_size 216 STAT 16:total_chunks 4854 STAT 16:free_chunks 4770 STAT 16:free_chunks_end 0 STAT 17:chunk_size 224 STAT 17:total_chunks 4681 STAT 17:free_chunks 4655 STAT 17:free_chunks_end 0 STAT 18:chunk_size 232 STAT 18:total_chunks 4519 STAT 18:free_chunks 4489 STAT 18:free_chunks_end 0 STAT 19:chunk_size 240 STAT 19:total_chunks 4369 STAT 19:free_chunks 4345 STAT 19:free_chunks_end 0 STAT 20:chunk_size 248 STAT 20:total_chunks 4228 STAT 20:free_chunks 4217 STAT 20:free_chunks_end 0 STAT 21:chunk_size 256 STAT 21:total_chunks 4096 STAT 21:free_chunks 4091 STAT 21:free_chunks_end 0 STAT 22:chunk_size 264 STAT 22:total_chunks 3971 STAT 22:free_chunks 3966 STAT 22:free_chunks_end 0 STAT 23:chunk_size 272 STAT 23:total_chunks 3855 STAT 23:free_chunks 3851 STAT 23:free_chunks_end 0 STAT 24:chunk_size 280 STAT 24:total_chunks 3744 STAT 24:free_chunks 3738 STAT
Re: Question Regarding Slab Imbalance
1.5.what? also yes, chunk_size_growth_factor. don't set it to 1.02. set it to something so your slab classes are more evenly distributed. the max is 63. On Thu, 12 Nov 2020, Tony Wu wrote: > Hi Dormando, > > Thanks for the reply, I believe the engine version is 1.5. By multiplier > I assumed you meant “chunk_size_growth_factor”, which is currently set > at 1.02. > > Best, > > Tony Wu > > > On Nov 12, 2020, at 2:26 PM, dormando wrote: > > > > What version are they running now? That stat output looks pretty sparse. > > > > Unfortunately when it comes to elasticache the answer is probably to just > > beg them to upgrade to a newer version. they tend to run really old and > > newer verisons do a lot better at balancing memory. > > > > This is also suspect: > >> STAT 62:chunk_size 712 > >> STAT 62:total_chunks 4416 > >> STAT 62:free_chunks 631 > >> STAT 62:free_chunks_end 0 > >> STAT 63:chunk_size 524288 > > > > anything you store that's larger than 712 bytes will use half a megabyte > > of memory. If you're setting the slab multiplier (-f) to something really > > aggressive, you should stop that :) > > > > On Thu, 12 Nov 2020, Tony Wu wrote: > > > >> We are currently using memcached provided by AWS ElastiCache service in > >> modern mode to store web session keys > >> among other things. We are observing session keys being evicted before TTL > >> even though the cluster seems to have > >> ample free memory, which hints at slab imbalance. I've included a print > >> out of slab stats below in terms of > >> sizing and total / free pages. > >> What would be the best way to smooth out this imbalance? We already > >> enabled slab reassign / auto-move under > >> modern mode. > >> > >> Thanks. > >> > >> STAT 1:chunk_size 96 > >> STAT 1:total_chunks 131064 > >> STAT 1:free_chunks 1664 > >> STAT 1:free_chunks_end 0 > >> STAT 2:chunk_size 104 > >> STAT 2:total_chunks 60492 > >> STAT 2:free_chunks 5327 > >> STAT 2:free_chunks_end 0 > >> STAT 3:chunk_size 112 > >> STAT 3:total_chunks 46810 > >> STAT 3:free_chunks 79 > >> STAT 3:free_chunks_end 0 > >> STAT 4:chunk_size 120 > >> STAT 4:total_chunks 26214 > >> STAT 4:free_chunks 48 > >> STAT 4:free_chunks_end 0 > >> STAT 5:chunk_size 128 > >> STAT 5:total_chunks 32768 > >> STAT 5:free_chunks 6193 > >> STAT 5:free_chunks_end 0 > >> STAT 6:chunk_size 136 > >> STAT 6:total_chunks 23130 > >> STAT 6:free_chunks 4594 > >> STAT 6:free_chunks_end 0 > >> STAT 7:chunk_size 144 > >> STAT 7:total_chunks 7281 > >> STAT 7:free_chunks 5894 > >> STAT 7:free_chunks_end 0 > >> STAT 8:chunk_size 152 > >> STAT 8:total_chunks 13796 > >> STAT 8:free_chunks 12008 > >> STAT 8:free_chunks_end 0 > >> STAT 9:chunk_size 160 > >> STAT 9:total_chunks 6553 > >> STAT 9:free_chunks 6316 > >> STAT 9:free_chunks_end 0 > >> STAT 10:chunk_size 168 > >> STAT 10:total_chunks 6241 > >> STAT 10:free_chunks 6186 > >> STAT 10:free_chunks_end 0 > >> STAT 11:chunk_size 176 > >> STAT 11:total_chunks 5957 > >> STAT 11:free_chunks 5946 > >> STAT 11:free_chunks_end 0 > >> STAT 12:chunk_size 184 > >> STAT 12:total_chunks 5698 > >> STAT 12:free_chunks 5664 > >> STAT 12:free_chunks_end 0 > >> STAT 13:chunk_size 192 > >> STAT 13:total_chunks 5461 > >> STAT 13:free_chunks 5380 > >> STAT 13:free_chunks_end 0 > >> STAT 14:chunk_size 200 > >> STAT 14:total_chunks 5242 > >> STAT 14:free_chunks 5135 > >> STAT 14:free_chunks_end 0 > >> STAT 15:chunk_size 208 > >> STAT 15:total_chunks 5041 > >> STAT 15:free_chunks 4892 > >> STAT 15:free_chunks_end 0 > >> STAT 16:chunk_size 216 > >> STAT 16:total_chunks 4854 > >> STAT 16:free_chunks 4770 > >> STAT 16:free_chunks_end 0 > >> STAT 17:chunk_size 224 > >> STAT 17:total_chunks 4681 > >> STAT 17:free_chunks 4655 > >> STAT 17:free_chunks_end 0 > >> STAT 18:chunk_size 232 > >> STAT 18:total_chunks 4519 > >> STAT 18:free_chunks 4489 > >> STAT 18:free_chunks_end 0 > >> STAT 19:chunk_size 240 > >> STAT 19:total_chunks 4369 > >> STAT 19:free_chunks 4345 > >> STAT 19:free_chunks_end 0 > >> STAT 20:chunk_size 248 > >> STAT 20:total_chunks 4228 > >> STAT 20:free_chunks 4217 > >> STAT 20:free_chunks_end 0 > >> STAT 21:chunk_size 256 > >> STAT 21:total_chunks 4096 > >> STAT 21:free_chunks 4091 > >> STAT 21:free_chunks_end 0 > >> STAT 22:chunk_size 264 > >> STAT 22:total_chunks 3971 > >> STAT 22:free_chunks 3966 > >> STAT 22:free_chunks_end 0 > >> STAT 23:chunk_size 272 > >> STAT 23:total_chunks 3855 > >> STAT 23:free_chunks 3851 > >> STAT 23:free_chunks_end 0 > >> STAT 24:chunk_size 280 > >> STAT 24:total_chunks 3744 > >> STAT 24:free_chunks 3738 > >> STAT 24:free_chunks_end 0 > >> STAT 25:chunk_size 288 > >> STAT 25:total_chunks 3640 > >> STAT 25:free_chunks 3638 > >> STAT 25:free_chunks_end 0 > >> STAT 26:chunk_size 296 > >> STAT 26:total_chunks 3542 > >> STAT 26:free_chunks 3533 > >> STAT 26:free_chunks_end 0 > >> STAT 27:chunk_size 304 > >> STAT 27:total_chunks 3449 > >> STAT 27:free_chunks 3448 > >> STAT 27:free_chunks_end 0 > >> STAT 28:chunk_size 312 > >> STAT 28:total_chunks 3360 > >>
Re: Question Regarding Slab Imbalance
Hi Dormando, Thanks for the reply, I believe the engine version is 1.5. By multiplier I assumed you meant “chunk_size_growth_factor”, which is currently set at 1.02. Best, Tony Wu > On Nov 12, 2020, at 2:26 PM, dormando wrote: > > What version are they running now? That stat output looks pretty sparse. > > Unfortunately when it comes to elasticache the answer is probably to just > beg them to upgrade to a newer version. they tend to run really old and > newer verisons do a lot better at balancing memory. > > This is also suspect: >> STAT 62:chunk_size 712 >> STAT 62:total_chunks 4416 >> STAT 62:free_chunks 631 >> STAT 62:free_chunks_end 0 >> STAT 63:chunk_size 524288 > > anything you store that's larger than 712 bytes will use half a megabyte > of memory. If you're setting the slab multiplier (-f) to something really > aggressive, you should stop that :) > > On Thu, 12 Nov 2020, Tony Wu wrote: > >> We are currently using memcached provided by AWS ElastiCache service in >> modern mode to store web session keys >> among other things. We are observing session keys being evicted before TTL >> even though the cluster seems to have >> ample free memory, which hints at slab imbalance. I've included a print out >> of slab stats below in terms of >> sizing and total / free pages. >> What would be the best way to smooth out this imbalance? We already enabled >> slab reassign / auto-move under >> modern mode. >> >> Thanks. >> >> STAT 1:chunk_size 96 >> STAT 1:total_chunks 131064 >> STAT 1:free_chunks 1664 >> STAT 1:free_chunks_end 0 >> STAT 2:chunk_size 104 >> STAT 2:total_chunks 60492 >> STAT 2:free_chunks 5327 >> STAT 2:free_chunks_end 0 >> STAT 3:chunk_size 112 >> STAT 3:total_chunks 46810 >> STAT 3:free_chunks 79 >> STAT 3:free_chunks_end 0 >> STAT 4:chunk_size 120 >> STAT 4:total_chunks 26214 >> STAT 4:free_chunks 48 >> STAT 4:free_chunks_end 0 >> STAT 5:chunk_size 128 >> STAT 5:total_chunks 32768 >> STAT 5:free_chunks 6193 >> STAT 5:free_chunks_end 0 >> STAT 6:chunk_size 136 >> STAT 6:total_chunks 23130 >> STAT 6:free_chunks 4594 >> STAT 6:free_chunks_end 0 >> STAT 7:chunk_size 144 >> STAT 7:total_chunks 7281 >> STAT 7:free_chunks 5894 >> STAT 7:free_chunks_end 0 >> STAT 8:chunk_size 152 >> STAT 8:total_chunks 13796 >> STAT 8:free_chunks 12008 >> STAT 8:free_chunks_end 0 >> STAT 9:chunk_size 160 >> STAT 9:total_chunks 6553 >> STAT 9:free_chunks 6316 >> STAT 9:free_chunks_end 0 >> STAT 10:chunk_size 168 >> STAT 10:total_chunks 6241 >> STAT 10:free_chunks 6186 >> STAT 10:free_chunks_end 0 >> STAT 11:chunk_size 176 >> STAT 11:total_chunks 5957 >> STAT 11:free_chunks 5946 >> STAT 11:free_chunks_end 0 >> STAT 12:chunk_size 184 >> STAT 12:total_chunks 5698 >> STAT 12:free_chunks 5664 >> STAT 12:free_chunks_end 0 >> STAT 13:chunk_size 192 >> STAT 13:total_chunks 5461 >> STAT 13:free_chunks 5380 >> STAT 13:free_chunks_end 0 >> STAT 14:chunk_size 200 >> STAT 14:total_chunks 5242 >> STAT 14:free_chunks 5135 >> STAT 14:free_chunks_end 0 >> STAT 15:chunk_size 208 >> STAT 15:total_chunks 5041 >> STAT 15:free_chunks 4892 >> STAT 15:free_chunks_end 0 >> STAT 16:chunk_size 216 >> STAT 16:total_chunks 4854 >> STAT 16:free_chunks 4770 >> STAT 16:free_chunks_end 0 >> STAT 17:chunk_size 224 >> STAT 17:total_chunks 4681 >> STAT 17:free_chunks 4655 >> STAT 17:free_chunks_end 0 >> STAT 18:chunk_size 232 >> STAT 18:total_chunks 4519 >> STAT 18:free_chunks 4489 >> STAT 18:free_chunks_end 0 >> STAT 19:chunk_size 240 >> STAT 19:total_chunks 4369 >> STAT 19:free_chunks 4345 >> STAT 19:free_chunks_end 0 >> STAT 20:chunk_size 248 >> STAT 20:total_chunks 4228 >> STAT 20:free_chunks 4217 >> STAT 20:free_chunks_end 0 >> STAT 21:chunk_size 256 >> STAT 21:total_chunks 4096 >> STAT 21:free_chunks 4091 >> STAT 21:free_chunks_end 0 >> STAT 22:chunk_size 264 >> STAT 22:total_chunks 3971 >> STAT 22:free_chunks 3966 >> STAT 22:free_chunks_end 0 >> STAT 23:chunk_size 272 >> STAT 23:total_chunks 3855 >> STAT 23:free_chunks 3851 >> STAT 23:free_chunks_end 0 >> STAT 24:chunk_size 280 >> STAT 24:total_chunks 3744 >> STAT 24:free_chunks 3738 >> STAT 24:free_chunks_end 0 >> STAT 25:chunk_size 288 >> STAT 25:total_chunks 3640 >> STAT 25:free_chunks 3638 >> STAT 25:free_chunks_end 0 >> STAT 26:chunk_size 296 >> STAT 26:total_chunks 3542 >> STAT 26:free_chunks 3533 >> STAT 26:free_chunks_end 0 >> STAT 27:chunk_size 304 >> STAT 27:total_chunks 3449 >> STAT 27:free_chunks 3448 >> STAT 27:free_chunks_end 0 >> STAT 28:chunk_size 312 >> STAT 28:total_chunks 3360 >> STAT 28:free_chunks 3350 >> STAT 28:free_chunks_end 0 >> STAT 29:chunk_size 320 >> STAT 29:total_chunks 3276 >> STAT 29:free_chunks 3273 >> STAT 29:free_chunks_end 0 >> STAT 30:chunk_size 328 >> STAT 30:total_chunks 3196 >> STAT 30:free_chunks 3174 >> STAT 30:free_chunks_end 0 >> STAT 31:chunk_size 336 >> STAT 31:total_chunks 3120 >> STAT 31:free_chunks 3118 >> STAT 31:free_chunks_end 0 >> STAT 32:chunk_size 344 >> STAT 32:total_chunks 3048 >> STAT 32:free_chunks 3043 >> STAT
Re: Question Regarding Slab Imbalance
What version are they running now? That stat output looks pretty sparse. Unfortunately when it comes to elasticache the answer is probably to just beg them to upgrade to a newer version. they tend to run really old and newer verisons do a lot better at balancing memory. This is also suspect: > STAT 62:chunk_size 712 > STAT 62:total_chunks 4416 > STAT 62:free_chunks 631 > STAT 62:free_chunks_end 0 > STAT 63:chunk_size 524288 anything you store that's larger than 712 bytes will use half a megabyte of memory. If you're setting the slab multiplier (-f) to something really aggressive, you should stop that :) On Thu, 12 Nov 2020, Tony Wu wrote: > We are currently using memcached provided by AWS ElastiCache service in > modern mode to store web session keys > among other things. We are observing session keys being evicted before TTL > even though the cluster seems to have > ample free memory, which hints at slab imbalance. I've included a print out > of slab stats below in terms of > sizing and total / free pages. > What would be the best way to smooth out this imbalance? We already enabled > slab reassign / auto-move under > modern mode. > > Thanks. > > STAT 1:chunk_size 96 > STAT 1:total_chunks 131064 > STAT 1:free_chunks 1664 > STAT 1:free_chunks_end 0 > STAT 2:chunk_size 104 > STAT 2:total_chunks 60492 > STAT 2:free_chunks 5327 > STAT 2:free_chunks_end 0 > STAT 3:chunk_size 112 > STAT 3:total_chunks 46810 > STAT 3:free_chunks 79 > STAT 3:free_chunks_end 0 > STAT 4:chunk_size 120 > STAT 4:total_chunks 26214 > STAT 4:free_chunks 48 > STAT 4:free_chunks_end 0 > STAT 5:chunk_size 128 > STAT 5:total_chunks 32768 > STAT 5:free_chunks 6193 > STAT 5:free_chunks_end 0 > STAT 6:chunk_size 136 > STAT 6:total_chunks 23130 > STAT 6:free_chunks 4594 > STAT 6:free_chunks_end 0 > STAT 7:chunk_size 144 > STAT 7:total_chunks 7281 > STAT 7:free_chunks 5894 > STAT 7:free_chunks_end 0 > STAT 8:chunk_size 152 > STAT 8:total_chunks 13796 > STAT 8:free_chunks 12008 > STAT 8:free_chunks_end 0 > STAT 9:chunk_size 160 > STAT 9:total_chunks 6553 > STAT 9:free_chunks 6316 > STAT 9:free_chunks_end 0 > STAT 10:chunk_size 168 > STAT 10:total_chunks 6241 > STAT 10:free_chunks 6186 > STAT 10:free_chunks_end 0 > STAT 11:chunk_size 176 > STAT 11:total_chunks 5957 > STAT 11:free_chunks 5946 > STAT 11:free_chunks_end 0 > STAT 12:chunk_size 184 > STAT 12:total_chunks 5698 > STAT 12:free_chunks 5664 > STAT 12:free_chunks_end 0 > STAT 13:chunk_size 192 > STAT 13:total_chunks 5461 > STAT 13:free_chunks 5380 > STAT 13:free_chunks_end 0 > STAT 14:chunk_size 200 > STAT 14:total_chunks 5242 > STAT 14:free_chunks 5135 > STAT 14:free_chunks_end 0 > STAT 15:chunk_size 208 > STAT 15:total_chunks 5041 > STAT 15:free_chunks 4892 > STAT 15:free_chunks_end 0 > STAT 16:chunk_size 216 > STAT 16:total_chunks 4854 > STAT 16:free_chunks 4770 > STAT 16:free_chunks_end 0 > STAT 17:chunk_size 224 > STAT 17:total_chunks 4681 > STAT 17:free_chunks 4655 > STAT 17:free_chunks_end 0 > STAT 18:chunk_size 232 > STAT 18:total_chunks 4519 > STAT 18:free_chunks 4489 > STAT 18:free_chunks_end 0 > STAT 19:chunk_size 240 > STAT 19:total_chunks 4369 > STAT 19:free_chunks 4345 > STAT 19:free_chunks_end 0 > STAT 20:chunk_size 248 > STAT 20:total_chunks 4228 > STAT 20:free_chunks 4217 > STAT 20:free_chunks_end 0 > STAT 21:chunk_size 256 > STAT 21:total_chunks 4096 > STAT 21:free_chunks 4091 > STAT 21:free_chunks_end 0 > STAT 22:chunk_size 264 > STAT 22:total_chunks 3971 > STAT 22:free_chunks 3966 > STAT 22:free_chunks_end 0 > STAT 23:chunk_size 272 > STAT 23:total_chunks 3855 > STAT 23:free_chunks 3851 > STAT 23:free_chunks_end 0 > STAT 24:chunk_size 280 > STAT 24:total_chunks 3744 > STAT 24:free_chunks 3738 > STAT 24:free_chunks_end 0 > STAT 25:chunk_size 288 > STAT 25:total_chunks 3640 > STAT 25:free_chunks 3638 > STAT 25:free_chunks_end 0 > STAT 26:chunk_size 296 > STAT 26:total_chunks 3542 > STAT 26:free_chunks 3533 > STAT 26:free_chunks_end 0 > STAT 27:chunk_size 304 > STAT 27:total_chunks 3449 > STAT 27:free_chunks 3448 > STAT 27:free_chunks_end 0 > STAT 28:chunk_size 312 > STAT 28:total_chunks 3360 > STAT 28:free_chunks 3350 > STAT 28:free_chunks_end 0 > STAT 29:chunk_size 320 > STAT 29:total_chunks 3276 > STAT 29:free_chunks 3273 > STAT 29:free_chunks_end 0 > STAT 30:chunk_size 328 > STAT 30:total_chunks 3196 > STAT 30:free_chunks 3174 > STAT 30:free_chunks_end 0 > STAT 31:chunk_size 336 > STAT 31:total_chunks 3120 > STAT 31:free_chunks 3118 > STAT 31:free_chunks_end 0 > STAT 32:chunk_size 344 > STAT 32:total_chunks 3048 > STAT 32:free_chunks 3043 > STAT 32:free_chunks_end 0 > STAT 33:chunk_size 352 > STAT 33:total_chunks 2978 > STAT 33:free_chunks 2954 > STAT 33:free_chunks_end 0 > STAT 34:chunk_size 360 > STAT 34:total_chunks 2912 > STAT 34:free_chunks 2903 > STAT 34:free_chunks_end 0 > STAT 35:chunk_size 368 > STAT 35:total_chunks 2849 > STAT 35:free_chunks 2828 > STAT 35:free_chunks_end 0 > STAT 36:chunk_size 376 > STAT 36:total_chunks 2788 > STAT 36:free_chunks
Question Regarding Slab Imbalance
We are currently using memcached provided by AWS ElastiCache service in modern mode to store web session keys among other things. We are observing session keys being evicted before TTL even though the cluster seems to have ample free memory, which hints at slab imbalance. I've included a print out of slab stats below in terms of sizing and total / free pages. What would be the best way to smooth out this imbalance? We already enabled slab reassign / auto-move under modern mode. Thanks. STAT 1:chunk_size 96 STAT 1:total_chunks 131064 STAT 1:free_chunks 1664 STAT 1:free_chunks_end 0 STAT 2:chunk_size 104 STAT 2:total_chunks 60492 STAT 2:free_chunks 5327 STAT 2:free_chunks_end 0 STAT 3:chunk_size 112 STAT 3:total_chunks 46810 STAT 3:free_chunks 79 STAT 3:free_chunks_end 0 STAT 4:chunk_size 120 STAT 4:total_chunks 26214 STAT 4:free_chunks 48 STAT 4:free_chunks_end 0 STAT 5:chunk_size 128 STAT 5:total_chunks 32768 STAT 5:free_chunks 6193 STAT 5:free_chunks_end 0 STAT 6:chunk_size 136 STAT 6:total_chunks 23130 STAT 6:free_chunks 4594 STAT 6:free_chunks_end 0 STAT 7:chunk_size 144 STAT 7:total_chunks 7281 STAT 7:free_chunks 5894 STAT 7:free_chunks_end 0 STAT 8:chunk_size 152 STAT 8:total_chunks 13796 STAT 8:free_chunks 12008 STAT 8:free_chunks_end 0 STAT 9:chunk_size 160 STAT 9:total_chunks 6553 STAT 9:free_chunks 6316 STAT 9:free_chunks_end 0 STAT 10:chunk_size 168 STAT 10:total_chunks 6241 STAT 10:free_chunks 6186 STAT 10:free_chunks_end 0 STAT 11:chunk_size 176 STAT 11:total_chunks 5957 STAT 11:free_chunks 5946 STAT 11:free_chunks_end 0 STAT 12:chunk_size 184 STAT 12:total_chunks 5698 STAT 12:free_chunks 5664 STAT 12:free_chunks_end 0 STAT 13:chunk_size 192 STAT 13:total_chunks 5461 STAT 13:free_chunks 5380 STAT 13:free_chunks_end 0 STAT 14:chunk_size 200 STAT 14:total_chunks 5242 STAT 14:free_chunks 5135 STAT 14:free_chunks_end 0 STAT 15:chunk_size 208 STAT 15:total_chunks 5041 STAT 15:free_chunks 4892 STAT 15:free_chunks_end 0 STAT 16:chunk_size 216 STAT 16:total_chunks 4854 STAT 16:free_chunks 4770 STAT 16:free_chunks_end 0 STAT 17:chunk_size 224 STAT 17:total_chunks 4681 STAT 17:free_chunks 4655 STAT 17:free_chunks_end 0 STAT 18:chunk_size 232 STAT 18:total_chunks 4519 STAT 18:free_chunks 4489 STAT 18:free_chunks_end 0 STAT 19:chunk_size 240 STAT 19:total_chunks 4369 STAT 19:free_chunks 4345 STAT 19:free_chunks_end 0 STAT 20:chunk_size 248 STAT 20:total_chunks 4228 STAT 20:free_chunks 4217 STAT 20:free_chunks_end 0 STAT 21:chunk_size 256 STAT 21:total_chunks 4096 STAT 21:free_chunks 4091 STAT 21:free_chunks_end 0 STAT 22:chunk_size 264 STAT 22:total_chunks 3971 STAT 22:free_chunks 3966 STAT 22:free_chunks_end 0 STAT 23:chunk_size 272 STAT 23:total_chunks 3855 STAT 23:free_chunks 3851 STAT 23:free_chunks_end 0 STAT 24:chunk_size 280 STAT 24:total_chunks 3744 STAT 24:free_chunks 3738 STAT 24:free_chunks_end 0 STAT 25:chunk_size 288 STAT 25:total_chunks 3640 STAT 25:free_chunks 3638 STAT 25:free_chunks_end 0 STAT 26:chunk_size 296 STAT 26:total_chunks 3542 STAT 26:free_chunks 3533 STAT 26:free_chunks_end 0 STAT 27:chunk_size 304 STAT 27:total_chunks 3449 STAT 27:free_chunks 3448 STAT 27:free_chunks_end 0 STAT 28:chunk_size 312 STAT 28:total_chunks 3360 STAT 28:free_chunks 3350 STAT 28:free_chunks_end 0 STAT 29:chunk_size 320 STAT 29:total_chunks 3276 STAT 29:free_chunks 3273 STAT 29:free_chunks_end 0 STAT 30:chunk_size 328 STAT 30:total_chunks 3196 STAT 30:free_chunks 3174 STAT 30:free_chunks_end 0 STAT 31:chunk_size 336 STAT 31:total_chunks 3120 STAT 31:free_chunks 3118 STAT 31:free_chunks_end 0 STAT 32:chunk_size 344 STAT 32:total_chunks 3048 STAT 32:free_chunks 3043 STAT 32:free_chunks_end 0 STAT 33:chunk_size 352 STAT 33:total_chunks 2978 STAT 33:free_chunks 2954 STAT 33:free_chunks_end 0 STAT 34:chunk_size 360 STAT 34:total_chunks 2912 STAT 34:free_chunks 2903 STAT 34:free_chunks_end 0 STAT 35:chunk_size 368 STAT 35:total_chunks 2849 STAT 35:free_chunks 2828 STAT 35:free_chunks_end 0 STAT 36:chunk_size 376 STAT 36:total_chunks 2788 STAT 36:free_chunks 2766 STAT 36:free_chunks_end 0 STAT 37:chunk_size 384 STAT 37:total_chunks 2730 STAT 37:free_chunks 2684 STAT 37:free_chunks_end 0 STAT 38:chunk_size 392 STAT 38:total_chunks 2674 STAT 38:free_chunks 2621 STAT 38:free_chunks_end 0 STAT 39:chunk_size 400 STAT 39:total_chunks 2621 STAT 39:free_chunks 2564 STAT 39:free_chunks_end 0 STAT 40:chunk_size 408 STAT 40:total_chunks 2570 STAT 40:free_chunks 2527 STAT 40:free_chunks_end 0 STAT 41:chunk_size 416 STAT 41:total_chunks 2520 STAT 41:free_chunks 2216 STAT 41:free_chunks_end 0 STAT 42:chunk_size 424 STAT 42:total_chunks 2473 STAT 42:free_chunks 2256 STAT 42:free_chunks_end 0 STAT 43:chunk_size 432 STAT 43:total_chunks 2427 STAT 43:free_chunks 2299 STAT 43:free_chunks_end 0 STAT 44:chunk_size 440 STAT 44:total_chunks 2383 STAT 44:free_chunks 1898 STAT 44:free_chunks_end 0 STAT 45:chunk_size 448 STAT 45:total_chunks 2340 STAT 45:free_chunks 1962 STAT 45:free_chunks_end 0 STAT 46:chunk_size 456 STAT
