For me the timeout occurs when I've a lot of request to the server. It was working at 100% CPU and it seems that cassandra was receiving request and keep them into the queue for more than 5000 millis then all the request was rejected with timeout.
I can't think about another solution or to increase the timeout (not ideally) or add more cassandra instance to answer high requests number faster. On Tue, Nov 10, 2009 at 7:22 AM, Jonathan Ellis <[email protected]> wrote: > What's causing the timeout? An error on the source node, or just > slowness? If the latter, how many rows are in your multiget? > > On Mon, Nov 9, 2009 at 10:25 PM, Chris Were <[email protected]> wrote: > > > > I'm getting a Timeout Exception every now and again (currently every > couple > > of minutes or so). > > Using revision 833288. Quorum set to ONE. My cassandra instance has been > > running for two days and the data directory is around 16GB. I'm not sure > > what the problem is, but let me know of any tests I can do to help reduce > > the problem further. There are two variations on the exception, I have > > pasted them both below. > > ERROR [pool-1-thread-63] 2009-11-09 20:17:27,579 Cassandra.java (line > > org.apache.cassandra.service.Cassandra$Processor) Internal error > processing > > get_slice > > java.lang.RuntimeException: java.util.concurrent.TimeoutException: > Operation > > timed out - received only 0 responses from . > > at > > > org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:103) > > at > > > org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:177) > > at > > > org.apache.cassandra.service.CassandraServer.multigetSliceInternal(CassandraServer.java:252) > > at > > > org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:215) > > at > > > org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:668) > > at > > > org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:624) > > at > > > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:636) > > Caused by: java.util.concurrent.TimeoutException: Operation timed out - > > received only 0 responses from . > > at > > > org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:79) > > at > > > org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:408) > > at > > > org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:333) > > at > > > org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:95) > > ... 9 more > > ERROR [pool-1-thread-19] 2009-11-09 11:29:18,731 Cassandra.java (line > > org.apache.cassandra.service.Cassandra$Processor) Internal error > processing > > get_slice > > java.lang.RuntimeException: java.util.concurrent.TimeoutException: > Operation > > timed out - received only 1 responses from /10.121.217.5 . > > at > > > org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:103) > > at > > > org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:177) > > at > > > org.apache.cassandra.service.CassandraServer.multigetSliceInternal(CassandraServer.java:252) > > at > > > org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:215) > > at > > > org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:668) > > at > > > org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:624) > > at > > > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:636) > > Caused by: java.util.concurrent.TimeoutException: Operation timed out - > > received only 1 responses from /10.121.217.5 . > > at > > > org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:79) > > at > > > org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:408) > > at > > > org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:333) > > at > > > org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:95) > > ... 9 more > > Cheers, > > Chris >
