Re: Calculate memory used for keycache
One additionnal question, I don't really understand what is in the key cache. I have a column family with only one key, and the keycache size is 118 ... ? Any idea. Thks. Jean-Yves
Re: Calculate memory used for keycache
One additionnal question, I don't really understand what is in the key cache. I have a column family with only one key, and the keycache size is 118 ... ? The key cache is basically a hash table mapping row keys to sstable offsets. It avoids the need to read from the index portion of the sstable for specific keys that have recently been accessed. (Normally the index portion is seeked into, a bit of data is streamed from disk, de-serialized, and used to find the offset for that key). -- / Peter Schuller
Calculate memory used for keycache
Hello How is it possible calculate this value? I think that key size, if we use RandomPartitioner will 16 bytes so keycache will took 16*(num of keycache elements) bytes ??
Re: Calculate memory used for keycache
How is it possible calculate this value? I think that key size, if we use RandomPartitioner will 16 bytes so keycache will took 16*(num of keycache elements) bytes ?? The easiest way right now is probably empirical testing. The issue is that the memory use must include overhead associated with the data structures involved, as well as longer-term effects like fragmentation in old-space in the GC (similarly to malloc/free). -- / Peter Schuller
Re: Calculate memory used for keycache
Sometime back I looked at the code to find that out. Following is the result. There will be some additional overhead for internal DS for ConcurrentLinkedHashMap. Keycache size * (8 bytes for position i.e. value + X bytes for key + 16 bytes for token (RP) + 8 byte reference for DecoratedKey + 8 bytes for descriptor reference) Thanks, Naren On Mon, Mar 14, 2011 at 1:29 PM, ruslan usifov ruslan.usi...@gmail.comwrote: Hello How is it possible calculate this value? I think that key size, if we use RandomPartitioner will 16 bytes so keycache will took 16*(num of keycache elements) bytes ??
Re: Calculate memory used for keycache
On Mon, Mar 14, 2011 at 1:19 PM, Peter Schuller peter.schul...@infidyne.com wrote: How is it possible calculate this value? I think that key size, if we use RandomPartitioner will 16 bytes so keycache will took 16*(num of keycache elements) bytes ?? The easiest way right now is probably empirical testing. The issue is that the memory use must include overhead associated with the data structures involved, as well as longer-term effects like fragmentation in old-space in the GC (similarly to malloc/free). In 0.7 and trunk, there is nodetool clear[row|key]cache, so your testing could be of the form : a) start cluster b) read enough keys to populate your caches per CF c) measure memory usage d) clear cache e) measure memory usage =Rob
