Re: Help! Serious crash!

Wed, 07 Oct 2009 11:24:14 -0700

I don't know why... But for some reason, the commitlogs grew bigger than 128MB - so I suppose a manual compact was needed...?!?! Anyways... I restarted with the 2GB value in cassandra.in.sh, everything got compacted, the commitlog rotated - and I could stop, change the settings back and start everything up, with no probles... 

Thanks for your suggestions Igor! :-)

Best regards
Dan

On 07/10/2009, at 20.16, Dan Larsen wrote:


Sorry about that... running 0.4 taken from trunk.
I didn't change CommitLogRotationThresholdInMB - it's still 128

Thanks!

On 07/10/2009, at 20.12, Igor Katkov wrote:


What version are you running? Take 0.4 from trunk.
Default config has
<CommitLogRotationThresholdInMB>128</ CommitLogRotationThresholdInMB> 
Did you change it allow your commit log chunks be 1Gb?

On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <[email protected]> wrote:
Hi guys,
I have a small setup with 3 macs running os x snow leopard and the native java 1.6.0.something.something ;-) I am doing a lot of inserts, without any problems and I can read the data again. 

But....
At some point, around 1G Commitlog size, cassandra starts a rise in ram usage, untill it just stops responding - if I restart the server, with or without -b, it will just usae 100% cpu for 10-15 mins and then stop with this message: Another problem, is that it seems to kill other servers too, when I try to restart :-( 

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid53353.hprof ...
Heap dump file created [1164687070 bytes in 89.912 secs]
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle), 5,RMI Runtime] 
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedInputStream.<init> (BufferedInputStream.java:178) at java.io.BufferedInputStream.<init> (BufferedInputStream.java:158) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0 (TCPTransport.java:679) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run (TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask (ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:908) 
       at java.lang.Thread.run(Thread.java:637)
ERROR - Fatal exception in thread Thread[main,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle), 5,RMI Runtime] 
java.lang.OutOfMemoryError: Java heap space
       at java.lang.Class.getDeclaredMethods0(Native Method)
       at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
       at java.lang.Class.getDeclaredMethod(Class.java:1935)
at java.io.ObjectStreamClass.getPrivateMethod (ObjectStreamClass.java:1382) at java.io.ObjectStreamClass.access$1700 (ObjectStreamClass.java:52) 
       at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
       at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java: 413) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java: 310) at java.io.ObjectOutputStream.writeObject0 (ObjectOutputStream.java:1106) at java.io.ObjectOutputStream.defaultWriteFields (ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeSerialData (ObjectOutputStream.java:1474) at java.io.ObjectOutputStream.writeOrdinaryObject (ObjectOutputStream.java:1392) at java.io.ObjectOutputStream.writeObject0 (ObjectOutputStream.java:1150) at java.io.ObjectOutputStream.writeObject (ObjectOutputStream.java:326) 
       at java.util.ArrayList.writeObject(ArrayList.java:570)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:25) 
       at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject (ObjectStreamClass.java:945) at java.io.ObjectOutputStream.writeSerialData (ObjectOutputStream.java:1461) at java.io.ObjectOutputStream.writeOrdinaryObject (ObjectOutputStream.java:1392) at java.io.ObjectOutputStream.writeObject0 (ObjectOutputStream.java:1150) at java.io.ObjectOutputStream.writeObject (ObjectOutputStream.java:326) 
       at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
at sun.rmi.server.UnicastServerRef.dispatch (UnicastServerRef.java:315) 
       at sun.rmi.transport.Transport$1.run(Transport.java:159)
       at java.security.AccessController.doPrivileged(Native Method)
       at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages (TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0 (TCPTransport.java:790) ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle), 5,RMI Runtime] 

[ABOVE REPEATS A COUPLE OF TIMES]
If I try to connect with nodeprobe, after it halts, I will just get this: 

Error connecting to remote JMX agent!
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.100.203; nested exception is: 
       java.net.ConnectException: Connection refused]
at javax.management.remote.rmi.RMIConnector.connect (RMIConnector.java:338) at javax.management.remote.JMXConnectorFactory.connect (JMXConnectorFactory.java:248) at org.apache.cassandra.tools.NodeProbe.connect (NodeProbe.java:152) at org.apache.cassandra.tools.NodeProbe.<init> (NodeProbe.java:114) at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java: 545) Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.100.203; nested exception is: 
       java.net.ConnectException: Connection refused]
at com.sun.jndi.rmi.registry.RegistryContext.lookup (RegistryContext.java:101) at com.sun.jndi.toolkit.url.GenericURLContext.lookup (GenericURLContext.java:185) 
       at javax.naming.InitialContext.lookup(InitialContext.java:392)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI (RMIConnector.java:1886) at javax.management.remote.rmi.RMIConnector.findRMIServer (RMIConnector.java:1856) at javax.management.remote.rmi.RMIConnector.connect (RMIConnector.java:257) 
       ... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host: 192.168.100.203; nested exception is: 
       java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket (TCPEndpoint.java:601) at sun.rmi.transport.tcp.TCPChannel.createConnection (TCPChannel.java:198) at sun.rmi.transport.tcp.TCPChannel.newConnection (TCPChannel.java:184) 
       at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
       at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup (RegistryContext.java:97) 
       ... 9 more
Caused by: java.net.ConnectException: Connection refused
       at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java: 333) at java.net.PlainSocketImpl.connectToAddress (PlainSocketImpl.java:195) 
       at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
       at java.net.Socket.connect(Socket.java:525)
       at java.net.Socket.connect(Socket.java:475)
       at java.net.Socket.<init>(Socket.java:372)
       at java.net.Socket.<init>(Socket.java:186)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket (RMIDirectSocketFactory.java:22) at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket (RMIMasterSocketFactory.java:128) at sun.rmi.transport.tcp.TCPEndpoint.newSocket (TCPEndpoint.java:595) 
       ... 14 more


I know i responds as supposed, before the crash.

Hope somebody can help! :-)

Best regards
Dan Larsen

Reply via email to