Good points Ted. Aniket, you might try using visualvm or something similar to monitor heap usage during your test: http://visualvm.java.net/ It will give you some insight into how quickly the heap is being used up.
You're reporting approx 4k/znode. That seems way high to me. Are you sure that the data for each znode is small? (you initially reported it being a digit). I just ran a quick test using yourkit and I'm seeing more like 400 bytes per znode. You might start bin/zkCli.sh and use the "stat" command to get more insight into your znodes (for example the length). Patrick On Wed, Nov 9, 2011 at 9:07 PM, Ted Dunning <[email protected]> wrote: > Each node can take a fair bit of overhead in memory. That 2G you > allocated may not go as far as you might like. I am a bit surprised if > this eats as much memory as your test seems to imply, but it isn't > impossible. > > Have you tried a row or column compressed format? Do you really need to > store elements separately? > > Also, Zookeeper is intended primarily as a coordination service. That > tends to allow/encourage design decisions that may have a negative impact > on using it as a data store. The fact is, it tends to do much better than > you might expect at datastore tasks, but that doesn't change the fact that > it really isn't a great platform for that. > > On Wed, Nov 9, 2011 at 4:14 PM, Aniket Chakrabarti < > [email protected]> wrote: > >> Hi, >> >> I am trying to load a huge matrix(100,000 by 500) to my zookeeper >> instance. Each element of the matrix is a ZNODE and value of each element >> is a digit(0-9). >> >> But I'm only able to load around 1000 x 500 nodes. Zookeeper is throwing >> an error after that. Mostly it is throwing a "-5" error code which is a >> marshalling/unmarshalling error. I'm using the perl interface of the >> zookeeper. >> >> My question is: Is there a limit to the maximum number of ZNODES a >> zookeeper instance can hold or this is limited by the system memory? >> >> Any pointers on how to avoid the error would be very helpful. >> >> Thanks, >> Aniket Chakrabarti >> PhD student >> The Ohio State University >> >
