Re: memory estimate for each key in the key cache
It is not telling you to multiply your key size by 10-12, it is telling you to multiply the output of the nodetool cfstats reported key cache size by 10-12. The key cache size reported is actually the number of keys in the key cache. So, it is the same thing as suggesting each key takes 10-12 bytes.
Re: memory estimate for each key in the key cache
If you get the sizing wrong there area couple of emergency pressure valves in the config… https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L113 Watch for log messages with Cassandra is now reducing cache sizes to free up memory. in them. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 21/12/2011, at 5:21 AM, Kent Tong wrote: It is not telling you to multiply your key size by 10-12, it is telling you to multiply the output of the nodetool cfstats reported key cache size by 10-12. The key cache size reported is actually the number of keys in the key cache. So, it is the same thing as suggesting each key takes 10-12 bytes.
Re: memory estimate for each key in the key cache
It is not telling you to multiply your key size by 10-12, it is telling you to multiply the output of the nodetool cfstats reported key cache size by 10-12. -Jeremiah On Dec 18, 2011, at 6:37 PM, Guy Incognito wrote: to be blunt, this doesn't sound right to me, unless it's doing something rather more clever to manage the memory. i mocked up a simple class containing a byte[], ByteBuffer and long, and the shallow size alone is 32 bytes. deep size with a byte[16], 1-byte bytebuffer and long is 132. this is a on a 64-bit jvm on win x64, but is consistent(ish) with what i've seen in the past on linux jvms. the actual code has rather more objects than this (it's a map, it has a pair, decoratedKey) so would be quite a bit bigger per key. On 17/12/2011 03:42, Brandon Williams wrote: On Fri, Dec 16, 2011 at 9:31 PM, Dave Brosiusdbros...@mebigfatguy.com wrote: Wow, Java is a lot better than I thought if it can perform that kind of magic. I'm guessing the wiki information is just old and out of date. It's probably more like 60 + sizeof(key) With jamm and MAT it's fairly easy to test. The number is accurate last I checked. -Brandon
Re: memory estimate for each key in the key cache
to be blunt, this doesn't sound right to me, unless it's doing something rather more clever to manage the memory. i mocked up a simple class containing a byte[], ByteBuffer and long, and the shallow size alone is 32 bytes. deep size with a byte[16], 1-byte bytebuffer and long is 132. this is a on a 64-bit jvm on win x64, but is consistent(ish) with what i've seen in the past on linux jvms. the actual code has rather more objects than this (it's a map, it has a pair, decoratedKey) so would be quite a bit bigger per key. On 17/12/2011 03:42, Brandon Williams wrote: On Fri, Dec 16, 2011 at 9:31 PM, Dave Brosiusdbros...@mebigfatguy.com wrote: Wow, Java is a lot better than I thought if it can perform that kind of magic. I'm guessing the wiki information is just old and out of date. It's probably more like 60 + sizeof(key) With jamm and MAT it's fairly easy to test. The number is accurate last I checked. -Brandon
Re: memory estimate for each key in the key cache
On Fri, Dec 16, 2011 at 8:52 PM, Kent Tong freemant2...@yahoo.com wrote: Hi, From the source code I can see that for each key, the hash (token), the key itself (ByteBuffer) and the position (long. offset in the sstable) are stored into the key cache. The hash is an MD5 hash, so it is 16 bytes. So, the total size required is at least 16+size-of(key)+4 which is 20 bytes. If we consider the overhead of the object references, then it will be even larger. Then, why the wiki recommends multiplying the number of keys cached with 10-12 to get the memory requirement? In a word: java. -Brandon
Re: memory estimate for each key in the key cache
On 12/16/2011 10:13 PM, Brandon Williams wrote: On Fri, Dec 16, 2011 at 8:52 PM, Kent Tongfreemant2...@yahoo.com wrote: Hi, From the source code I can see that for each key, the hash (token), the key itself (ByteBuffer) and the position (long. offset in the sstable) are stored into the key cache. The hash is an MD5 hash, so it is 16 bytes. So, the total size required is at least 16+size-of(key)+4 which is 20 bytes. If we consider the overhead of the object references, then it will be even larger. Then, why the wiki recommends multiplying the number of keys cached with 10-12 to get the memory requirement? In a word: java. -Brandon Wow, Java is a lot better than I thought if it can perform that kind of magic. I'm guessing the wiki information is just old and out of date. It's probably more like 60 + sizeof(key)
Re: memory estimate for each key in the key cache
On Fri, Dec 16, 2011 at 9:31 PM, Dave Brosius dbros...@mebigfatguy.com wrote: Wow, Java is a lot better than I thought if it can perform that kind of magic. I'm guessing the wiki information is just old and out of date. It's probably more like 60 + sizeof(key) With jamm and MAT it's fairly easy to test. The number is accurate last I checked. -Brandon
