I'm running three Cassandra nodes in virtual machines.
During a 'get' operation from Cassandra-remote directed at one of
these nodes, I'm receiving the following output
vad...@vadmin:~/cassandra$ interface/gen-py/cassandra/Cassandra-remote
-h 192.168.133.130:9160 get 'MockElementLibrary'
'0401318uuuuruepwdcznr' "ColumnPath('strings', None, 'id')" 2
/usr/local/lib/python2.6/dist-packages/thrift/Thrift.py:58:
DeprecationWarning: BaseException.message has been deprecated as of
Python 2.6
self.message = message
/usr/local/lib/python2.6/dist-packages/thrift/Thrift.py:99:
DeprecationWarning: BaseException.message has been deprecated as of
Python 2.6
self.message = iprot.readString();
Traceback (most recent call last):
File "interface/gen-py/cassandra/Cassandra-remote", line 93, in
<module>
pp.pprint(client.get(args[0],args[1],eval(args[2]),eval(args[3]),))
File "/home/vadmin/cassandra-0.4.0-beta1/interface/gen-py/cassandra/
Cassandra.py", line 182, in get
return self.recv_get()
File "/home/vadmin/cassandra-0.4.0-beta1/interface/gen-py/cassandra/
Cassandra.py", line 201, in recv_get
raise x
thrift.Thrift.TApplicationException/usr/local/lib/python2.6/dist-
packages/thrift/Thrift.py:76: DeprecationWarning:
BaseException.message has been deprecated as of Python 2.6
if self.message:
/usr/local/lib/python2.6/dist-packages/thrift/Thrift.py:77:
DeprecationWarning: BaseException.message has been deprecated as of
Python 2.6
return self.message
: Internal error processing get
The same 'get' operation from Cassandra-remote directed at another of
these nodes, yields 'normal' output
vad...@vadmin:~/cassandra$ interface/gen-py/cassandra/Cassandra-remote
-h 192.168.133.129:9160 get 'MockElementLibrary'
'0401318uuuuruepwdcznr' "ColumnPath('strings', None, 'id')" 2
Traceback (most recent call last):
File "interface/gen-py/cassandra/Cassandra-remote", line 93, in
<module>
pp.pprint(client.get(args[0],args[1],eval(args[2]),eval(args[3]),))
File "/home/vadmin/cassandra-0.4.0-beta1/interface/gen-py/cassandra/
Cassandra.py", line 182, in get
return self.recv_get()
File "/home/vadmin/cassandra-0.4.0-beta1/interface/gen-py/cassandra/
Cassandra.py", line 210, in recv_get
raise result.nfe
ttypes.NotFoundException: NotFoundException()
Furthermore, querying the same column for (some) other keys is
successful when no matter which node it is directed at.
Looking at the log for the node that produced the error from the query
above:
DEBUG [pool-1-thread-22] 2009-08-19 16:54:57,618
CassandraServer.java (line 221) get
DEBUG [pool-1-thread-22] 2009-08-19 16:54:57,618 StorageProxy.java
(line 420) strongread reading data for
SliceByNamesReadCommand(table='MockElementLibrary',
key='0401318uuuuruepwdcznr',
columnParent='QueryPath(columnFamilyName='strings',
superColumnName='null', columnName='null')', columns=[id,]) from
38...@null
DEBUG [pool-1-thread-22] 2009-08-19 16:54:57,619 StorageProxy.java
(line 427) strongread reading digest for
SliceByNamesReadCommand(table='MockElementLibrary',
key='0401318uuuuruepwdcznr',
columnParent='QueryPath(columnFamilyName='strings',
superColumnName='null', columnName='null')', columns=[id,]) from [email protected]
:7000
WARN [MESSAGE-SERIALIZER-POOL:4] 2009-08-19 16:54:57,619
MessageSerializationTask.java (line 81) Exception was generated at :
08/19/2009 16:54:57 on thread MESSAGE-SERIALIZER-POOL:4
java.lang.NullPointerException
at
org.apache.cassandra.net.TcpConnection.<init>(TcpConnection.java:83)
at
org
.apache
.cassandra
.net.TcpConnectionManager.getConnection(TcpConnectionManager.java:64)
at
org
.apache
.cassandra.net.MessagingService.getConnection(MessagingService.java:
306)
at
org
.apache
.cassandra
.net.MessageSerializationTask.run(MessageSerializationTask.java:66)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
DEBUG [RESPONSE-STAGE:4] 2009-08-19 16:54:57,622
ResponseVerbHandler.java (line 38) Processing response on a callback
from [email protected]:7000
ERROR [pool-1-thread-22] 2009-08-19 16:55:02,619 Cassandra.java
(line 608) Internal error processing get
java.lang.RuntimeException: java.util.concurrent.TimeoutException:
Operation timed out - received only 1 responses from
192.168.133.129:7000 .
at
org
.apache
.cassandra
.service.CassandraServer.readColumnFamily(CassandraServer.java:100)
at
org
.apache.cassandra.service.CassandraServer.get(CassandraServer.java:
226)
at org.apache.cassandra.service.Cassandra$Processor
$get.process(Cassandra.java:602)
at org.apache.cassandra.service.Cassandra
$Processor.process(Cassandra.java:560)
at org.apache.thrift.server.TThreadPoolServer
$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.concurrent.TimeoutException: Operation timed
out - received only 1 responses from 192.168.133.129:7000 .
at
org
.apache
.cassandra
.service.QuorumResponseHandler.get(QuorumResponseHandler.java:86)
at
org
.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:
435)
at
org
.apache
.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:330)
at
org
.apache
.cassandra
.service.CassandraServer.readColumnFamily(CassandraServer.java:92)
... 7 more
It appears to me that there is a timeout during the
QuorumResponseHandler processing, stemming from a NullPointerException
that occurs as part of the read process. I suspect that this
NullPointerException has something to do with the second DEBUG [pool-1-
thread-22] comment regarding strongread ... from 38...@null.
Does anyone know why this might be happening?
Thanks for any insight,
Phil
