On Wed, Nov 9, 2011 at 1:14 PM, Aniket Chakrabarti <[email protected]> wrote: > I am trying to load a huge matrix(100,000 by 500) to my zookeeper instance. > Each element of the matrix is a ZNODE and value of each element is a > digit(0-9). > > But I'm only able to load around 1000 x 500 nodes. Zookeeper is throwing an > error after that. Mostly it is throwing a "-5" error code which is a > marshalling/unmarshalling error. I'm using the perl interface of the > zookeeper. > > My question is: Is there a limit to the maximum number of ZNODES a zookeeper > instance can hold or this is limited by the system memory? > > Any pointers on how to avoid the error would be very helpful.
Available heap memory is really the only limit. Try $ JVMFLAGS=-D-Xmx<mem in gig>g bin/zkServer.sh also $ sudo jmap -heap <jvm pid> will give you some insight into whether it was set correctly or not (ie MaxHeapSize) The most I've tried is 5 million znodes with 25 million watches (using zkpython, but zkperl should be fine). iirc that was 8gig heap, but ymmv depending on data size. You may also need to tune the GC at some point (would suggest turning on cms and parallel collector) to limit stop the world pauses. Regards, Patrick
