Quorum read timeout stemming from TcpConnectionManager initialization
---------------------------------------------------------------------
Key: CASSANDRA-381
URL: https://issues.apache.org/jira/browse/CASSANDRA-381
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.4
Environment: ubuntu linux (#42-Ubuntu SMP Fri Apr 17 02:48:10 UTC
2009) running in VMWare Fusion virtual machine.
Reporter: Phillip Michalak
A quorum read from the Cassandra-remote tool times out on certain queries,
depending on the key and node at which the query is directed. An example of a
failed query and its output:
vad...@vadmin:~/cassandra$ interface/gen-py/cassandra/Cassandra-remote -h
192.168.133.130:9160 get 'MockElementLibrary' '0401318uuuuruepwdcznr'
"ColumnPath('strings', None, 'id')" 2
/usr/local/lib/python2.6/dist-packages/thrift/Thrift.py:58: DeprecationWarning:
BaseException.message has been deprecated as of Python 2.6
self.message = message
/usr/local/lib/python2.6/dist-packages/thrift/Thrift.py:99: DeprecationWarning:
BaseException.message has been deprecated as of Python 2.6
self.message = iprot.readString();
Traceback (most recent call last):
File "interface/gen-py/cassandra/Cassandra-remote", line 93, in <module>
pp.pprint(client.get(args[0],args[1],eval(args[2]),eval(args[3]),))
File
"/home/vadmin/cassandra-0.4.0-beta1/interface/gen-py/cassandra/Cassandra.py",
line 182, in get
return self.recv_get()
File
"/home/vadmin/cassandra-0.4.0-beta1/interface/gen-py/cassandra/Cassandra.py",
line 201, in recv_get
raise x
thrift.Thrift.TApplicationException/usr/local/lib/python2.6/dist-packages/thrift/Thrift.py:76:
DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
if self.message:
/usr/local/lib/python2.6/dist-packages/thrift/Thrift.py:77: DeprecationWarning:
BaseException.message has been deprecated as of Python 2.6
return self.message
: Internal error processing get
The relevant portion of the log file generated from this request:
DEBUG [pool-1-thread-22] 2009-08-19 16:54:57,618 CassandraServer.java (line
221) get
DEBUG [pool-1-thread-22] 2009-08-19 16:54:57,618 StorageProxy.java (line 420)
strongread reading data for SliceByNamesReadCommand(table='MockElementLibrary',
key='0401318uuuuruepwdcznr',
columnParent='QueryPath(columnFamilyName='strings', superColumnName='null',
columnName='null')', columns=[id,]) from 38...@null
DEBUG [pool-1-thread-22] 2009-08-19 16:54:57,619 StorageProxy.java (line 427)
strongread reading digest for
SliceByNamesReadCommand(table='MockElementLibrary',
key='0401318uuuuruepwdcznr',
columnParent='QueryPath(columnFamilyName='strings', superColumnName='null',
columnName='null')', columns=[id,]) from [email protected]:7000
WARN [MESSAGE-SERIALIZER-POOL:4] 2009-08-19 16:54:57,619
MessageSerializationTask.java (line 81) Exception was generated at : 08/19/2009
16:54:57 on thread MESSAGE-SERIALIZER-POOL:4
java.lang.NullPointerException
at org.apache.cassandra.net.TcpConnection.<init>(TcpConnection.java:83)
at
org.apache.cassandra.net.TcpConnectionManager.getConnection(TcpConnectionManager.java:64)
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:306)
at
org.apache.cassandra.net.MessageSerializationTask.run(MessageSerializationTask.java:66)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
DEBUG [RESPONSE-STAGE:4] 2009-08-19 16:54:57,622 ResponseVerbHandler.java (line
38) Processing response on a callback from
[email protected]:7000
ERROR [pool-1-thread-22] 2009-08-19 16:55:02,619 Cassandra.java (line 608)
Internal error processing get
java.lang.RuntimeException: java.util.concurrent.TimeoutException: Operation
timed out - received only 1 responses from 192.168.133.129:7000 .
at
org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:100)
at
org.apache.cassandra.service.CassandraServer.get(CassandraServer.java:226)
at
org.apache.cassandra.service.Cassandra$Processor$get.process(Cassandra.java:602)
at
org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:560)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.concurrent.TimeoutException: Operation timed out -
received only 1 responses from 192.168.133.129:7000 .
at
org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:86)
at
org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:435)
at
org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:330)
at
org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:92)
... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
