Brian, How much state are you storing in ZK? Can you check the size of the snapshots?
One common problem when folks are testing is that they forget to delete the data from previous tests, so the state keeps accumulating and the server keeps crashing because the state is too large. Also, consider trying 3.4.5 just to see if it is a problem with 3.4.6 alone. -Flavio On Friday, September 5, 2014 2:23 PM, Brian C. Huffman <[email protected]> wrote: > > >We're running the latest version of the stable 3.4 branch (3.4.6) and >have been consistently having problems running out of heap space. > >We're running a single server (redundancy isn't a concern at this point) >and I've tried the defaults (which seems to use Java's default heap of >8GB) as well as limiting to 3GB. Either way the Zookeeper server >eventually dies. With larger heap size it seems to take longer to die. > >Here's the latest trace: >2014-09-05 00:51:11,419 [myid:] - ERROR >[SyncThread:0:SyncRequestProcessor@183] - Severe unrecoverable error, >exiting >java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:2271) > at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) > at >java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) > at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) > at java.io.DataOutputStream.write(DataOutputStream.java:107) > at java.io.FilterOutputStream.write(FilterOutputStream.java:97) > at >org.apache.jute.BinaryOutputArchive.writeBuffer(BinaryOutputArchive.java:119) > at org.apache.zookeeper.txn.Txn.serialize(Txn.java:49) > at >org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123) > at org.apache.zookeeper.txn.MultiTxn.serialize(MultiTxn.java:44) > at >org.apache.zookeeper.server.persistence.Util.marshallTxnEntry(Util.java:263) > at >org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:216) > at >org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:314) > at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:476) > at >org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:140) >2014-09-05 00:51:07,866 [myid:] - WARN >[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught >end of stream exception >EndOfStreamException: Unable to read additional data from client >sessionid 0x14837ac98960071, likely client has closed socket > at >org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) > at >org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) > at java.lang.Thread.run(Thread.java:745) > >Here's my configuration: >[user@xyz conf]$ grep -v '^#' zoo.cfg >tickTime=2000 >initLimit=10 >syncLimit=5 >dataDir=/usr/local/var/zookeeper >clientPort=2181 >autopurge.snapRetainCount=3 >autopurge.purgeInterval=1 > >Can anyone suggest what the issue could be? > >Thanks, >Brian > > > >
