We're running the latest version of the stable 3.4 branch (3.4.6) and
have been consistently having problems running out of heap space.
We're running a single server (redundancy isn't a concern at this point)
and I've tried the defaults (which seems to use Java's default heap of
8GB) as well as limiting to 3GB. Either way the Zookeeper server
eventually dies. With larger heap size it seems to take longer to die.
Here's the latest trace:
2014-09-05 00:51:11,419 [myid:] - ERROR
[SyncThread:0:SyncRequestProcessor@183] - Severe unrecoverable error,
exiting
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
at
org.apache.jute.BinaryOutputArchive.writeBuffer(BinaryOutputArchive.java:119)
at org.apache.zookeeper.txn.Txn.serialize(Txn.java:49)
at
org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
at org.apache.zookeeper.txn.MultiTxn.serialize(MultiTxn.java:44)
at
org.apache.zookeeper.server.persistence.Util.marshallTxnEntry(Util.java:263)
at
org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:216)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:314)
at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:476)
at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:140)
2014-09-05 00:51:07,866 [myid:] - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught
end of stream exception
EndOfStreamException: Unable to read additional data from client
sessionid 0x14837ac98960071, likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
Here's my configuration:
[user@xyz conf]$ grep -v '^#' zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/var/zookeeper
clientPort=2181
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
Can anyone suggest what the issue could be?
Thanks,
Brian
