Hi guys,
I have a small setup with 3 macs running os x snow leopard and the
native java 1.6.0.something.something ;-)
I am doing a lot of inserts, without any problems and I can read the
data again.
But....
At some point, around 1G Commitlog size, cassandra starts a rise in
ram usage, untill it just stops responding - if I restart the server,
with or without -b, it will just usae 100% cpu for 10-15 mins and then
stop with this message:
Another problem, is that it seems to kill other servers too, when I
try to restart :-(
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid53353.hprof ...
Heap dump file created [1164687070 bytes in 89.912 secs]
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
(TCPTransport.java:679)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
ERROR - Fatal exception in thread Thread[main,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
at java.lang.Class.getDeclaredMethod(Class.java:1935)
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:
1382)
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1106)
at java.io.ObjectOutputStream.defaultWriteFields
(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:
1474)
at java.io.ObjectOutputStream.writeOrdinaryObject
(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:
945)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:
1461)
at java.io.ObjectOutputStream.writeOrdinaryObject
(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages
(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
(TCPTransport.java:790)
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
5,RMI Runtime]
[ABOVE REPEATS A COUPLE OF TIMES]
If I try to connect with nodeprobe, after it halts, I will just get
this:
Error connecting to remote JMX agent!
java.io.IOException: Failed to retrieve RMIServer stub:
javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host:
192.168.100.203; nested exception is:
java.net.ConnectException: Connection refused]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:
338)
at javax.management.remote.JMXConnectorFactory.connect
(JMXConnectorFactory.java:248)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
Caused by: javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host:
192.168.100.203; nested exception is:
java.net.ConnectException: Connection refused]
at com.sun.jndi.rmi.registry.RegistryContext.lookup
(RegistryContext.java:101)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup
(GenericURLContext.java:185)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
(RMIConnector.java:1886)
at javax.management.remote.rmi.RMIConnector.findRMIServer
(RMIConnector.java:1856)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:
257)
... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host:
192.168.100.203; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:
198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup
(RegistryContext.java:97)
... 9 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
at java.net.Socket.connect(Socket.java:525)
at java.net.Socket.connect(Socket.java:475)
at java.net.Socket.<init>(Socket.java:372)
at java.net.Socket.<init>(Socket.java:186)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 14 more
I know i responds as supposed, before the crash.
Hope somebody can help! :-)
Best regards
Dan Larsen
