[jira] [Commented] (CASSANDRA-2604) EOFException on commitlogs
[ https://issues.apache.org/jira/browse/CASSANDRA-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029874#comment-13029874 ] Terje Marthinussen commented on CASSANDRA-2604: --- Not able to reproduce the problem with the patch in this ticket. Currently using CASSANDRA-2401 and so far so good. I think you can just close this ticket for now and I will reopen if I see the problem again. Thanks! EOFException on commitlogs -- Key: CASSANDRA-2604 URL: https://issues.apache.org/jira/browse/CASSANDRA-2604 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Reporter: Terje Marthinussen Assignee: Sylvain Lebresne Fix For: 0.8.0 Attachments: 0001-avoid-modifying-original-mutation-during-apply.patch I have seen this occasionally since we started testing 0.8. It happens when reading commitlogs on startups. However, I have seen it a lot less on 0.8 beta2 (although this is from beta 2) ERROR [main] 2011-05-04 18:02:38,134 AbstractCassandraDaemon.java (line 330) Exception encountered during startup. java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:250) at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:357) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:252) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:43) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136) at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:368) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:256) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:157) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:313) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80) Note that the line numbers on columnserializer may be off due to some local changes, but those changes are in code not executed in this case and I am 100% sure they do not trigger this problem. I looked on this in the debugger in eclipse on a trunk from 0.8 2 weeks ago, and the interesting thing I saw was that according to the debugger, the offset of the inputstream to the deserializer was already at the end (very last byte) of the underlying bytebuffer but according to the stack, it was trying to read the length of the column name (first read done in the deserialized). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2604) EOFException on commitlogs
[ https://issues.apache.org/jira/browse/CASSANDRA-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029345#comment-13029345 ] Terje Marthinussen commented on CASSANDRA-2604: --- Secondary indexes - Check (the error at least in the case I studied now is on a CF with secondary index) Not in batch mode - Check I am not 100% sure I understand in what cases a column may be removed. There should be no deletes here and I think no updates to columns that already exists (I will have to check that...), but I am messing a bit around with the system while a couple of feeders are continuously running and they will refeed if there are errors. It is very likely that there are are quite a few retries going on at times which I guess could trigger what you are referring too? I will apply the patch at once and test. Thanks! EOFException on commitlogs -- Key: CASSANDRA-2604 URL: https://issues.apache.org/jira/browse/CASSANDRA-2604 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Reporter: Terje Marthinussen Assignee: Sylvain Lebresne Fix For: 0.8.0 Attachments: 0001-avoid-modifying-original-mutation-during-apply.patch I have seen this occasionally since we started testing 0.8. It happens when reading commitlogs on startups. However, I have seen it a lot less on 0.8 beta2 (although this is from beta 2) ERROR [main] 2011-05-04 18:02:38,134 AbstractCassandraDaemon.java (line 330) Exception encountered during startup. java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:250) at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:357) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:252) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:43) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136) at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:368) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:256) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:157) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:313) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80) Note that the line numbers on columnserializer may be off due to some local changes, but those changes are in code not executed in this case and I am 100% sure they do not trigger this problem. I looked on this in the debugger in eclipse on a trunk from 0.8 2 weeks ago, and the interesting thing I saw was that according to the debugger, the offset of the inputstream to the deserializer was already at the end (very last byte) of the underlying bytebuffer but according to the stack, it was trying to read the length of the column name (first read done in the deserialized). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2604) EOFException on commitlogs
[ https://issues.apache.org/jira/browse/CASSANDRA-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029388#comment-13029388 ] Jonathan Ellis commented on CASSANDRA-2604: --- I don't think it's actually necessary to even modify the original mutation, since applying obsolete columns to the data path is harmless. I removed that entirely in the CASSANDRA-2401 patch (not yet applied). EOFException on commitlogs -- Key: CASSANDRA-2604 URL: https://issues.apache.org/jira/browse/CASSANDRA-2604 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Reporter: Terje Marthinussen Assignee: Sylvain Lebresne Fix For: 0.8.0 Attachments: 0001-avoid-modifying-original-mutation-during-apply.patch I have seen this occasionally since we started testing 0.8. It happens when reading commitlogs on startups. However, I have seen it a lot less on 0.8 beta2 (although this is from beta 2) ERROR [main] 2011-05-04 18:02:38,134 AbstractCassandraDaemon.java (line 330) Exception encountered during startup. java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:250) at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:357) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:252) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:43) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136) at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:368) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:256) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:157) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:313) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80) Note that the line numbers on columnserializer may be off due to some local changes, but those changes are in code not executed in this case and I am 100% sure they do not trigger this problem. I looked on this in the debugger in eclipse on a trunk from 0.8 2 weeks ago, and the interesting thing I saw was that according to the debugger, the offset of the inputstream to the deserializer was already at the end (very last byte) of the underlying bytebuffer but according to the stack, it was trying to read the length of the column name (first read done in the deserialized). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2604) EOFException on commitlogs
[ https://issues.apache.org/jira/browse/CASSANDRA-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029467#comment-13029467 ] Sylvain Lebresne commented on CASSANDRA-2604: - I propose to see if Terje can reproduce. If not, this was the actual problem and I guess we'll close this on the account that CASSANDRA-2401 fixed it as a side effect. EOFException on commitlogs -- Key: CASSANDRA-2604 URL: https://issues.apache.org/jira/browse/CASSANDRA-2604 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Reporter: Terje Marthinussen Assignee: Sylvain Lebresne Fix For: 0.8.0 Attachments: 0001-avoid-modifying-original-mutation-during-apply.patch I have seen this occasionally since we started testing 0.8. It happens when reading commitlogs on startups. However, I have seen it a lot less on 0.8 beta2 (although this is from beta 2) ERROR [main] 2011-05-04 18:02:38,134 AbstractCassandraDaemon.java (line 330) Exception encountered during startup. java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:250) at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:357) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:252) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:43) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136) at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:368) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:256) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:157) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:313) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80) Note that the line numbers on columnserializer may be off due to some local changes, but those changes are in code not executed in this case and I am 100% sure they do not trigger this problem. I looked on this in the debugger in eclipse on a trunk from 0.8 2 weeks ago, and the interesting thing I saw was that according to the debugger, the offset of the inputstream to the deserializer was already at the end (very last byte) of the underlying bytebuffer but according to the stack, it was trying to read the length of the column name (first read done in the deserialized). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2604) EOFException on commitlogs
[ https://issues.apache.org/jira/browse/CASSANDRA-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029495#comment-13029495 ] Terje Marthinussen commented on CASSANDRA-2604: --- After stressing the system a bit with a bunch of restarts, I have not manage to reproduce the problem again, so looks good so far. I will leave it feeding overnight and try one more time in the morning and then I will try to reproduce with 2401 instead. On a side notice... I sometimes see quite a few commitlog segments. Right now, one node had 110 of them and it took 20 minutes to read all the logs. This seems a bit excessive? EOFException on commitlogs -- Key: CASSANDRA-2604 URL: https://issues.apache.org/jira/browse/CASSANDRA-2604 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Reporter: Terje Marthinussen Assignee: Sylvain Lebresne Fix For: 0.8.0 Attachments: 0001-avoid-modifying-original-mutation-during-apply.patch I have seen this occasionally since we started testing 0.8. It happens when reading commitlogs on startups. However, I have seen it a lot less on 0.8 beta2 (although this is from beta 2) ERROR [main] 2011-05-04 18:02:38,134 AbstractCassandraDaemon.java (line 330) Exception encountered during startup. java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:250) at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:357) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:252) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:43) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136) at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:368) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:256) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:157) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:313) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80) Note that the line numbers on columnserializer may be off due to some local changes, but those changes are in code not executed in this case and I am 100% sure they do not trigger this problem. I looked on this in the debugger in eclipse on a trunk from 0.8 2 weeks ago, and the interesting thing I saw was that according to the debugger, the offset of the inputstream to the deserializer was already at the end (very last byte) of the underlying bytebuffer but according to the stack, it was trying to read the length of the column name (first read done in the deserialized). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2604) EOFException on commitlogs
[ https://issues.apache.org/jira/browse/CASSANDRA-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029503#comment-13029503 ] Jonathan Ellis commented on CASSANDRA-2604: --- We're working on CASSANDRA-2427 to deal with commitlog segment proliferation. EOFException on commitlogs -- Key: CASSANDRA-2604 URL: https://issues.apache.org/jira/browse/CASSANDRA-2604 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8 beta 1 Reporter: Terje Marthinussen Assignee: Sylvain Lebresne Fix For: 0.8.0 Attachments: 0001-avoid-modifying-original-mutation-during-apply.patch I have seen this occasionally since we started testing 0.8. It happens when reading commitlogs on startups. However, I have seen it a lot less on 0.8 beta2 (although this is from beta 2) ERROR [main] 2011-05-04 18:02:38,134 AbstractCassandraDaemon.java (line 330) Exception encountered during startup. java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:250) at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:357) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:252) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:43) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136) at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:368) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:256) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:157) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:313) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80) Note that the line numbers on columnserializer may be off due to some local changes, but those changes are in code not executed in this case and I am 100% sure they do not trigger this problem. I looked on this in the debugger in eclipse on a trunk from 0.8 2 weeks ago, and the interesting thing I saw was that according to the debugger, the offset of the inputstream to the deserializer was already at the end (very last byte) of the underlying bytebuffer but according to the stack, it was trying to read the length of the column name (first read done in the deserialized). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2604) EOFException on commitlogs
[
https://issues.apache.org/jira/browse/CASSANDRA-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029033#comment-13029033
]
Terje Marthinussen commented on CASSANDRA-2604:
---
Time for some sleep, but I stepped through this code with a bad commit table
(not the same which caused the above error though, but produces same stack).
public void deserializeColumns(DataInput dis, ColumnFamily cf, boolean
intern, boolean fromRemote) throws IOException
{
int size = dis.readInt();
ColumnFamilyStore interner = intern ?
Table.open(CFMetaData.getCF(cf.id()).left).getColumnFamilyStore(cf.id()) : null;
for (int i = 0; i size; ++i)
{
IColumn column = cf.getColumnSerializer().deserialize(dis,
interner, fromRemote, (int) (System.currentTimeMillis() / 1000));
cf.addColumn(column);
}
}
The size of the buffer underlying dis is 153 byte.
size in the above code is 4.
With i == 2 finished, dis has just reached position 153 (perfectly) in the
underlying buffer, although I cannot guarantee that the value is completely
read.
Obviously, when trying to read at i==3, it crashes as it is already at the end
of the buffer.
I would expect the entry which it is trying to decode in this case to have 4,
not 3 columns.
I do see some of my own code is actually involved here, but unless the
commitlog does its own serialization fully or partly and uses the standard
deserializer my code is unlikely to cause any problems like this.
EOFException on commitlogs
--
Key: CASSANDRA-2604
URL: https://issues.apache.org/jira/browse/CASSANDRA-2604
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8 beta 1
Reporter: Terje Marthinussen
Assignee: Sylvain Lebresne
I have seen this occasionally since we started testing 0.8.
It happens when reading commitlogs on startups.
However, I have seen it a lot less on 0.8 beta2 (although this is from beta 2)
ERROR [main] 2011-05-04 18:02:38,134 AbstractCassandraDaemon.java (line 330)
Exception encountered during startup.
java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:250)
at
org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:357)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:252)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:43)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:368)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:256)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:157)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:313)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
Note that the line numbers on columnserializer may be off due to some local
changes, but those changes are in code not executed in this case and I am
100% sure they do not trigger this problem.
I looked on this in the debugger in eclipse on a trunk from 0.8 2 weeks ago,
and the interesting thing I saw was that according to the debugger, the
offset of the inputstream to the deserializer was already at the end (very
last byte) of the underlying bytebuffer but according to the stack, it was
trying to read the length of the column name (first read done in the
deserialized).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
