On Jul 25, 4:38 pm, dormando <[email protected]> wrote: > Uhhh. Can you post the output of `memcached -vvv` with your -I > adjustments? If you reduce the max page size it most certainly reduces the > number of slabs. It will increase the number of slab *pages* available. > Which doesn't affect anything. > > > > Also, I do not understand the warning, "It is not recommended to raise > > this limit above 1MB due just to performance reasons." What exactly > > are the performance issues? > > > If my default chunk size is 480 bytes and if I am storing items in 416 > > byte chunks and 448 byte chunks, then, I can store more chunks in 10 > > megabytes pages than I can in 10 kilobyte pages. So, why wouldn't I > > opt to store my chunks in 10 megabyte pages (rather than 10 kilobyte > > pages or even 1 megabyte pages)? The vast majority of my chunks are > > 448 byte chunks. So, it seems to me that I can use my memory more > > efficiently by opting for 10 megabyte slab pages. What, if anything, > > is behind the "peformance" warning? > > IF ANYTHING. So ominous. > > Using a non-brokeassed build of memcached, start it with the following > examples: > > memcached -vvv -I 128k > memcached -vvv -I 512k > memcached -vvv -I 1M > memcached -vvv -I 5M > memcached -vvv -I 10M > memcached -vvv -I 20M > > You'll see that the slab sizes get further apart. You're missunderstanding > what everything is inside the slabber. > > - slab page is set (1M by default) > - slab classes are created by starting at a minimum, multiplying by a > number (1.2 by default?) and repeating until the slab class size is equal > to the page size (1M by default) > - when you want to store an item: > - finds slab class (item size 416 bytes would go into class 8, so long > as the key isn't too long) > - try to find memory in class 8. No memory? Pull in one *page* (1M) > - divide that page into chunks of size 480 bytes (from class 8) > - hand back one page chunk for the item > - repeat > - memory between your actual item size, and the slab chunk size, is wasted > overhead > - the further apart slab classes are, the more memory you waste (perf > issue #1) > > If you give memcached a memory limit of 60 megs, and a max item size of > 10 megs, it only has enough pages to give one page to 6 slab classes. > Others will starve (tho it's actually a little more complicated than > that, but that's the idea). > > -Dormando
Hi Dormando, Thank you for your detailed explanation. I am storing roughly the same size items. All of the items get written to slab class 8, which has a chuck size of 480 bytes. Most of my items are 448 bytes. Some are 416 bytes. I can store roughly 8 million items with 4 GB of total memory. My point above was that when I store the same size items, namely 480 byte chunks, it does not matter very much what value that I assign to my slab page size. If I set my slab page size to 128K, I get 273 chunks (480 bytes per chunk) in slab class 8, and I have more slab pages. If I set the slab page size to 1MB, I get 2,184 chunks (480 bytes per chunk) in slab class 8, and I have fewer slab pages. Test 1: 1M slab page size 3700 max mem 3376314121 bytes 3219.9 megabytes 8,080,800 curr_items 3762.7 MB of resident memory used Test 2: 1K slab page size 3700 max mem 3382172689 bytes 3225.5 megabytes 8,082,772 curr items 3768.4 MB of resident memory used Test 3: 10M slab page size 3690 max mem 3370641668 bytes 3214.5 megabytes 8,060,805 curr items 3754.8 MB of resident memory used With a 1K slab page size (i.e., test 2), I only get 2 chunks per page, so I have 4,041,386 total pages (8,082,772 total chunks). With 1M slab page size (i.e., test 1), I get 2,184 chunks per page, so I have 3,700 total pages (8,080,800 total chunks). I did notice that in test 2 that the system was using more virtual memory--I had 160 MB of swap spaced used in test 2, and no swap space used in test 1. So, I am assuming that a 1MB slab page size is more efficient, but I was questioning the "peformance" warning. So, if I am ONLY using slab class 8 (480 bytes), is there any advantage in setting my slab page size to 1 KB or to 10 MB? I am seeing a very slight edge to a 10 MB slab page size. In test 3 (10 MB slab page size), memcached used 8 megabytes less resident memory than in test 1 (1 MB slab page size), and test 3 stored only 10,000 less items, which is about 4.6 megabytes of data. David
