[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13166904#comment-13166904 ] Jonathan Ellis commented on CASSANDRA-3585: --- Figured it out. This is actually a second manifestation of CASSANDRA-3577, a bug in the multi-DC write optimization. Pasting from there: bq. Node A (DC1) sends a write to node B (DC2), which forwards to node C (DC2). Node C replies to node A with the message ID it received from node B. If the message generation on A and B is far enough apart, then A will not have a callback for the reply and all you will see happen is the write timeout (at CL ONE). But if A *does* have a callback (for a different operation) waiting, then A will try to apply the mutation response to that callback, which (if the callback is for a read) will result in the error see in CASSANDRA-3585. For 1.0.6 we've disabled that optimization; for 1.1 we've fixed it by pre-generating extra callback IDs on the coordinator (node A in this example) and forwarding those cross-DC as well. Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585-v2.txt, 3585.txt, CassandraLogs.tar.bz2, metap_system.log.zip, metap_system.log.zip In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13166055#comment-13166055 ] Sylvain Lebresne commented on CASSANDRA-3585: - Are you using some internode encryption ? Also, if you could provide all the logs for all the nodes involved, that could help (with Jonathan patch). And of course, if you have an easy way to reproduce, that would be even better. Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585-v2.txt, 3585.txt, metap_system.log.zip, metap_system.log.zip In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13166340#comment-13166340 ] Jonathan Ellis commented on CASSANDRA-3585: --- In particular, logs from 192.168.136.234 would be useful since that is what sends the first bad response. Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585-v2.txt, 3585.txt, metap_system.log.zip, metap_system.log.zip In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[
https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13166449#comment-13166449
]
Jonathan Ellis commented on CASSANDRA-3585:
---
Some interesting things from the trace log:
The request that errors out:
{noformat}
DEBUG [pool-2-thread-39] 2011-12-08 21:07:21,815 StorageProxy.java (line 627)
Command/ConsistencyLevel is SliceByNamesReadCommand(table='KeyspaceMetadata',
key=313030303237383033,
columnParent='QueryPath(columnFamilyName='CF_MailboxSummary',
superColumnName='null', columnName='null')',
columns=[666f6c6465725f73756d6d6172795f4c057f31b9ba31138e6c1ff57f552f66,666f6c6465725f73756d6d6172795f5112276526f2330392edfc4b47cf4725,666f6c6465725f73756d6d6172795ff75e02920bc73235abb48ead4be65444,])/QUORUM
{noformat}
The data read, and the digest read that errors out.
{noformat}
DEBUG [pool-2-thread-39] 2011-12-08 21:07:21,836 StorageProxy.java (line 648)
reading data from /192.168.136.231
TRACE [pool-2-thread-39] 2011-12-08 21:07:21,836 MessagingService.java (line
412) /192.168.128.229 sending READ version 3 to 3242@/192.168.136.231
DEBUG [pool-2-thread-39] 2011-12-08 21:07:21,869 StorageProxy.java (line 668)
reading digest from /192.168.128.226
TRACE [pool-2-thread-39] 2011-12-08 21:07:21,870 MessagingService.java (line
412) /192.168.128.229 sending READ version 3 to 3280@/192.168.128.226
{noformat}
Failing to parse the reply:
{noformat}
DEBUG [Thread-8] 2011-12-08 21:07:21,877 IncomingTcpConnection.java (line 149)
Message body is 47 bytes
TRACE [Thread-8] 2011-12-08 21:07:21,885 MessagingService.java (line 502)
/192.168.128.229 received REQUEST_RESPONSE from 3280@/192.168.136.234
DEBUG [RequestResponseStage:18] 2011-12-08 21:07:21,885
ResponseVerbHandler.java (line 44) Processing response on a callback from
3280@/192.168.136.234
DEBUG [RequestResponseStage:18] 2011-12-08 21:07:21,891
AbstractRowResolver.java (line 64) Preprocessing 47 bytes
ERROR [RequestResponseStage:18] 2011-12-08 21:07:21,903
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[RequestResponseStage:18,5,main]
{noformat}
It sure looks like something fishy is going on with the message IDs -- we sent
3280@ to .226, but here we are processing a response from .234. It's also
worth noting that the message size (47 bytes) is neither a digest reply (21
bytes) or the data for the read here (16460 bytes), BUT we do see other 47 byte
replies in the log around this time, that are replies to a Mutation sent
earlier:
{noformat}
DEBUG [Thread-15] 2011-12-08 21:07:21,891 IncomingTcpConnection.java (line 149)
Message body is 47 bytes
TRACE [Thread-15] 2011-12-08 21:07:21,902 MessagingService.java (line 502)
/192.168.128.229 received REQUEST_RESPONSE from 3145@/192.168.128.227
{noformat}
I think we have a race in the MessagingService ID/Callback Map somewhere.
Intermittent exceptions seen in cassandra 1.0.5 during Reads.
-
Key: CASSANDRA-3585
URL: https://issues.apache.org/jira/browse/CASSANDRA-3585
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.5
Environment: RHEL 2.6.32-71.el6.x86_64.
RAM - 28GB
11 CPUs of 2.6GHz
Reporter: Shantanu
Attachments: 3585-v2.txt, 3585.txt, metap_system.log.zip,
metap_system.log.zip
In my test setup I have cassandra db provisioned with cassandra 0.8.7. The
setup is of two data centers. I have upgraded the cassandra to the latest
version 1.0.5. I'm seeing following exceptions in cassandra logs -
ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[RequestResponseStage:32,5,main]
java.io.IOError: java.io.EOFException
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126)
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81)
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
... 6 more
RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum.
I have run nodetool scrub on all the
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[
https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13166453#comment-13166453
]
Jonathan Ellis commented on CASSANDRA-3585:
---
Later on, we do get the reply we were expecting from .226, but by then the
callback mapping has been destroyed:
{noformat}
DEBUG [Thread-17] 2011-12-08 21:07:22,140 IncomingTcpConnection.java (line 149)
Message body is 21 bytes
TRACE [Thread-17] 2011-12-08 21:07:22,145 MessagingService.java (line 502)
/192.168.128.229 received REQUEST_RESPONSE from 3280@/192.168.128.226
{noformat}
Intermittent exceptions seen in cassandra 1.0.5 during Reads.
-
Key: CASSANDRA-3585
URL: https://issues.apache.org/jira/browse/CASSANDRA-3585
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.5
Environment: RHEL 2.6.32-71.el6.x86_64.
RAM - 28GB
11 CPUs of 2.6GHz
Reporter: Shantanu
Attachments: 3585-v2.txt, 3585.txt, metap_system.log.zip,
metap_system.log.zip
In my test setup I have cassandra db provisioned with cassandra 0.8.7. The
setup is of two data centers. I have upgraded the cassandra to the latest
version 1.0.5. I'm seeing following exceptions in cassandra logs -
ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[RequestResponseStage:32,5,main]
java.io.IOError: java.io.EOFException
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126)
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81)
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
... 6 more
RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum.
I have run nodetool scrub on all the nodes in the ring to verify if it solves
the issue but it didn't.
Thanks,
Shantanu
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[
https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13166466#comment-13166466
]
Jonathan Ellis commented on CASSANDRA-3585:
---
There's one more instance of the 3280@ id in the log:
{noformat}
TRACE [Thread-5] 2011-12-08 21:07:28,659 MessagingService.java (line 502)
/192.168.128.229 received GOSSIP_DIGEST_ACK from 3280@/192.168.128.225
{noformat}
But GOSSIP_DIGEST_ACK is a command, not a response, i.e., that 3280 is an id
generated on .225, so there's nothing abnormal there.
Intermittent exceptions seen in cassandra 1.0.5 during Reads.
-
Key: CASSANDRA-3585
URL: https://issues.apache.org/jira/browse/CASSANDRA-3585
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.5
Environment: RHEL 2.6.32-71.el6.x86_64.
RAM - 28GB
11 CPUs of 2.6GHz
Reporter: Shantanu
Attachments: 3585-v2.txt, 3585.txt, metap_system.log.zip,
metap_system.log.zip
In my test setup I have cassandra db provisioned with cassandra 0.8.7. The
setup is of two data centers. I have upgraded the cassandra to the latest
version 1.0.5. I'm seeing following exceptions in cassandra logs -
ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[RequestResponseStage:32,5,main]
java.io.IOError: java.io.EOFException
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126)
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81)
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
... 6 more
RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum.
I have run nodetool scrub on all the nodes in the ring to verify if it solves
the issue but it didn't.
Thanks,
Shantanu
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13166673#comment-13166673 ] Jonathan Ellis commented on CASSANDRA-3585: --- Brandon wrote a script (http://aep.appspot.com/display/PWH-9bzElErPO9ydpC2ay3I5Jys/) to check for the counterpart of this broken response -- a request sent to .234 that did NOT get a response. But nothing like that is in the logs, everything else matches up. Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585-v2.txt, 3585.txt, CassandraLogs.tar.bz2, metap_system.log.zip, metap_system.log.zip In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13166674#comment-13166674 ] Jonathan Ellis commented on CASSANDRA-3585: --- Shantanu, do you have encryption enabled? Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585-v2.txt, 3585.txt, CassandraLogs.tar.bz2, metap_system.log.zip, metap_system.log.zip In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13166677#comment-13166677 ] Shantanu commented on CASSANDRA-3585: - No, the internode encryption is set to none. Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585-v2.txt, 3585.txt, CassandraLogs.tar.bz2, metap_system.log.zip, metap_system.log.zip In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165323#comment-13165323 ] Jonathan Ellis commented on CASSANDRA-3585: --- The only place I can see any changes here from 0.8 is in the treatment of counters, are those involved in the exception you are seeing? Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585.txt In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165457#comment-13165457 ] Shantanu commented on CASSANDRA-3585: - We are not using counters Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585.txt In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165456#comment-13165456 ] Shantanu commented on CASSANDRA-3585: - Following is the stack trace for this patch - ERROR [RequestResponseStage:25] 2011-12-08 11:14:56,289 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:25,5,main] java.lang.AssertionError: 1067877 at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:99) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585.txt In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165637#comment-13165637 ] Jonathan Ellis commented on CASSANDRA-3585: --- Interesting. The exception seems to happen in clusters, and it's always the same incorrect digest length, 1067877. Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585.txt, metap_system.log.zip In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165831#comment-13165831 ] Jonathan Ellis commented on CASSANDRA-3585: --- The first cluster starts at 11:12:26 and finishes a few ms later. The next starts at 11:31:14 11:34:12. It's *almost* like what I'd expect to see from version negotiation but it's a little too slow to finish. Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585.txt, metap_system.log.zip In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165835#comment-13165835 ] Jonathan Ellis commented on CASSANDRA-3585: --- Another point: this node doesn't serve as request coordinator until much LATER than the exceptions. So why is there a ReadCallback in the stacktrace? Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585.txt, metap_system.log.zip In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3585) Intermittent exceptions seen in cassandra 1.0.5 during Reads.
[ https://issues.apache.org/jira/browse/CASSANDRA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165860#comment-13165860 ] Shantanu commented on CASSANDRA-3585: - Hi Jonathan, As of now the issue is only observed with multi DC deployment of cassandra1.0.5. The issue can be seen with a clean db created with 1.0.5 as well. Thanks, Shantanu Intermittent exceptions seen in cassandra 1.0.5 during Reads. - Key: CASSANDRA-3585 URL: https://issues.apache.org/jira/browse/CASSANDRA-3585 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.5 Environment: RHEL 2.6.32-71.el6.x86_64. RAM - 28GB 11 CPUs of 2.6GHz Reporter: Shantanu Attachments: 3585-v2.txt, 3585.txt, metap_system.log.zip, metap_system.log.zip In my test setup I have cassandra db provisioned with cassandra 0.8.7. The setup is of two data centers. I have upgraded the cassandra to the latest version 1.0.5. I'm seeing following exceptions in cassandra logs - ERROR [RequestResponseStage:32] 2011-12-06 14:46:08,150 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[RequestResponseStage:32,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:126) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64) ... 6 more RF is set to DC1:3,DC2:3 and I/m doing the operations with CL=Local_Quorum. I have run nodetool scrub on all the nodes in the ring to verify if it solves the issue but it didn't. Thanks, Shantanu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
