Hi Nico,
Thanks for the awesome analysis, the one part I was at first confused
about was this part
On Wed, May 25, 2011 at 06:22:53PM +0200, Nico Meyer wrote:
> Your actual RAM consumption of 70 GB seems to be at odd with the
> output of erlang:memory/0 that you sent:
>
> {total,7281790968} => RAM: 7281790968 * 8 = 54.3 GB
Because at first I thought your were multiplying by wordsize which is 8 as
well, but then I realized you were multiplying by number of nodes (also 8).
Anyway, things make a lot more sense now, and I'm thinking I may need
to fork bitcask and get rid of some of that extra overhead. For instance
13 bytes of overhead to store a tuple of binaries seems unnecessary, it's
probably better to just have a single binary with the bucket size as a
prefix, so something like
<<BucketSize:16,Bucket,Key>>
That way you turn 13 bytes of overhead to 2.
Of course I'd need some way to work with old data, but a one time migration
shouldn't be too bad.
It also seems like there should be some way to trim down some of that on
disk usage. I mean 300+ bytes to store 36 bytes is a lot.
Again, many thanks this is an awesome analysis.
-Anthony
--
------------------------------------------------------------------------
Anthony Molinaro <[email protected]>
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
