Re: total # of zknodes
Thanks, I see Patrick has replied in the archives but I don't have it in my mail (yet). I'd probably use 2 EC2 High-mem instances (17GB/instance), and I have no watches at all, so I should be able to store between 5-10M data, but I'll test that over the summer. I'll post the results here (and will publish my simple sync, no-watch Scala client as well). Best, Maarten Op 15 jul 2010, om 17:57 heeft Benjamin Reed het volgende geschreven: > i think there is a wiki page on this, but for the short answer: > > the number of znodes impact two things: memory footprint and recovery time. > there is a base overhead to znodes to store its path, pointers to the data, > pointers to the acl, etc. i believe that is around 100 bytes. you cant just > divide your memory by 100+1K (for data) though, because the GC needs to be > able to run and collect things and maintain a free space. if you use 3/4 of > your available memory, that would mean with 4G you can store about three > million znodes. when there is a crash and you recover, servers may need to > read this data back off the disk or over the network. that means it will take > about a minute to read 3G from the disk and perhaps a bit more to read it > over the network, so you will need to adjust your initLimit accordingly. > > of course this is all back-of-the-envelope. i would suggest doing some quick > benchmarks to test and make sure your results are in line with expectation. > > ben > > On 07/15/2010 02:56 AM, Maarten Koopmans wrote: >> Hi, >> >> I am mapping a filesystem to ZooKeeper, and use it for locking and mapping a >> filesystem namespace to a flat data object space (like S3). So assuming >> proper nesting and small ZooKeeper nodes (< 1KB), how many nodes could a >> cluster with a few GBs of memory per instance realistically hold totally? >> >> Thanks, Maarten > >
Re: total # of zknodes
I've done some tests with ~600 clients creating 5 million znodes (size 100bytes iirc) and 25million watches. I was using 8gb of memory for this, however --- in this scenario it's critical that you tune the GC, in particular you need to turn on CMS and incremental GC options. Otw when the GC collects it will collect for long periods of time and all of your clients will then time out. Keep an eye on the max latency of your servers, that's usually the most obvious indication of GC hits (it will spike up). You can use the latency tester from here to do the quick benchmarks Ben suggested: http://github.com/phunt/zk-smoketest also see: http://bit.ly/4ekN8G Patrick On 07/15/2010 08:57 AM, Benjamin Reed wrote: i think there is a wiki page on this, but for the short answer: the number of znodes impact two things: memory footprint and recovery time. there is a base overhead to znodes to store its path, pointers to the data, pointers to the acl, etc. i believe that is around 100 bytes. you cant just divide your memory by 100+1K (for data) though, because the GC needs to be able to run and collect things and maintain a free space. if you use 3/4 of your available memory, that would mean with 4G you can store about three million znodes. when there is a crash and you recover, servers may need to read this data back off the disk or over the network. that means it will take about a minute to read 3G from the disk and perhaps a bit more to read it over the network, so you will need to adjust your initLimit accordingly. of course this is all back-of-the-envelope. i would suggest doing some quick benchmarks to test and make sure your results are in line with expectation. ben On 07/15/2010 02:56 AM, Maarten Koopmans wrote: Hi, I am mapping a filesystem to ZooKeeper, and use it for locking and mapping a filesystem namespace to a flat data object space (like S3). So assuming proper nesting and small ZooKeeper nodes (< 1KB), how many nodes could a cluster with a few GBs of memory per instance realistically hold totally? Thanks, Maarten
Re: total # of zknodes
i think there is a wiki page on this, but for the short answer: the number of znodes impact two things: memory footprint and recovery time. there is a base overhead to znodes to store its path, pointers to the data, pointers to the acl, etc. i believe that is around 100 bytes. you cant just divide your memory by 100+1K (for data) though, because the GC needs to be able to run and collect things and maintain a free space. if you use 3/4 of your available memory, that would mean with 4G you can store about three million znodes. when there is a crash and you recover, servers may need to read this data back off the disk or over the network. that means it will take about a minute to read 3G from the disk and perhaps a bit more to read it over the network, so you will need to adjust your initLimit accordingly. of course this is all back-of-the-envelope. i would suggest doing some quick benchmarks to test and make sure your results are in line with expectation. ben On 07/15/2010 02:56 AM, Maarten Koopmans wrote: Hi, I am mapping a filesystem to ZooKeeper, and use it for locking and mapping a filesystem namespace to a flat data object space (like S3). So assuming proper nesting and small ZooKeeper nodes (< 1KB), how many nodes could a cluster with a few GBs of memory per instance realistically hold totally? Thanks, Maarten
