Re: Cannot recover SSTable with version f (current version g)
2011/7/6 Héctor Izquierdo Seliva izquie...@strands.com: Hi, i've been struggling to repair my failed node for the past few days, and I've seen this erros a few times. java.lang.RuntimeException: Cannot recover SSTable with version f (current version g). If it can read the sstables, why can't they be used to repair? After a sstable is streamed (by repair in particular), we first have to recreate a number of data structures. To do that, a specific part of the code is used that do not know how to cope with older sstable version (hence the message). This does not mean that this won't even get fixed, it will, but it is a current limitation. Is there anything I can do besides running scrub or compact on all the cluster? Not really no. You can wait enough time so that all old sstables have been compacted to newer version before running a repair, but since that could take a long time if you have lots of data, that may not always be an option. Now probably the most efficient way (the quote are because it's not the most efficient in time and investment it will require from you) is probably something like that: For each of the nodes of the cluster: 1) look the sstables on the node (do a 'ls' on the data repository). The sstable will look something like Standard1-g-105-Data.db where the 'g' may be a 'f' for some (all?) of the sstable. The 'f' means old sstable, the 'g' means new ones. 2) if all or almost all the big sstables are 'f', then run scrub on that node, that'll be simpler. 3) if only a handfull of sstable are on 'f' (typically, if the node has not just been updated), then what you can do is to force a compaction on those only by using JMX-CompactionManager-forceUserDefinedCompaction, giving it the keyspace name as first argument and the full path to the sstable as second argument. Regards Hector Izquierdo
Re: Cannot recover SSTable with version f (current version g)
Thanks Sylvain. I'll try option 3 when the current repair ends so I can fix the remaining CFs. I'm also finding a few of this while opening sstables that have been build with repair (SSTable build compactions) ERROR [CompactionExecutor:2] 2011-07-06 10:09:16,054 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[CompactionExecutor:2,1,main] java.lang.NullPointerException at org.apache.cassandra.io.sstable.SSTableWriter $RowIndexer.close(SSTableWriter.java:382) at org.apache.cassandra.io.sstable.SSTableWriter $RowIndexer.index(SSTableWriter.java:370) at org.apache.cassandra.io.sstable.SSTableWriter $Builder.build(SSTableWriter.java:315) at org.apache.cassandra.db.compaction.CompactionManager $9.call(CompactionManager.java:1103) at org.apache.cassandra.db.compaction.CompactionManager $9.call(CompactionManager.java:1094) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) El mié, 06-07-2011 a las 11:22 +0200, Sylvain Lebresne escribió: 2011/7/6 Héctor Izquierdo Seliva izquie...@strands.com: Hi, i've been struggling to repair my failed node for the past few days, and I've seen this erros a few times. java.lang.RuntimeException: Cannot recover SSTable with version f (current version g). If it can read the sstables, why can't they be used to repair? After a sstable is streamed (by repair in particular), we first have to recreate a number of data structures. To do that, a specific part of the code is used that do not know how to cope with older sstable version (hence the message). This does not mean that this won't even get fixed, it will, but it is a current limitation. Is there anything I can do besides running scrub or compact on all the cluster? Not really no. You can wait enough time so that all old sstables have been compacted to newer version before running a repair, but since that could take a long time if you have lots of data, that may not always be an option. Now probably the most efficient way (the quote are because it's not the most efficient in time and investment it will require from you) is probably something like that: For each of the nodes of the cluster: 1) look the sstables on the node (do a 'ls' on the data repository). The sstable will look something like Standard1-g-105-Data.db where the 'g' may be a 'f' for some (all?) of the sstable. The 'f' means old sstable, the 'g' means new ones. 2) if all or almost all the big sstables are 'f', then run scrub on that node, that'll be simpler. 3) if only a handfull of sstable are on 'f' (typically, if the node has not just been updated), then what you can do is to force a compaction on those only by using JMX-CompactionManager-forceUserDefinedCompaction, giving it the keyspace name as first argument and the full path to the sstable as second argument. Regards Hector Izquierdo
Re: Cannot recover SSTable with version f (current version g)
2011/7/6 Héctor Izquierdo Seliva izquie...@strands.com: I'm also finding a few of this while opening sstables that have been build with repair (SSTable build compactions) ERROR [CompactionExecutor:2] 2011-07-06 10:09:16,054 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[CompactionExecutor:2,1,main] java.lang.NullPointerException at org.apache.cassandra.io.sstable.SSTableWriter $RowIndexer.close(SSTableWriter.java:382) at org.apache.cassandra.io.sstable.SSTableWriter $RowIndexer.index(SSTableWriter.java:370) at org.apache.cassandra.io.sstable.SSTableWriter $Builder.build(SSTableWriter.java:315) at org.apache.cassandra.db.compaction.CompactionManager $9.call(CompactionManager.java:1103) at org.apache.cassandra.db.compaction.CompactionManager $9.call(CompactionManager.java:1094) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) That would be a bug. Can you open a ticket with the exact version you're using and the circumstance where this happens. Thanks. -- Sylvain El mié, 06-07-2011 a las 11:22 +0200, Sylvain Lebresne escribió: 2011/7/6 Héctor Izquierdo Seliva izquie...@strands.com: Hi, i've been struggling to repair my failed node for the past few days, and I've seen this erros a few times. java.lang.RuntimeException: Cannot recover SSTable with version f (current version g). If it can read the sstables, why can't they be used to repair? After a sstable is streamed (by repair in particular), we first have to recreate a number of data structures. To do that, a specific part of the code is used that do not know how to cope with older sstable version (hence the message). This does not mean that this won't even get fixed, it will, but it is a current limitation. Is there anything I can do besides running scrub or compact on all the cluster? Not really no. You can wait enough time so that all old sstables have been compacted to newer version before running a repair, but since that could take a long time if you have lots of data, that may not always be an option. Now probably the most efficient way (the quote are because it's not the most efficient in time and investment it will require from you) is probably something like that: For each of the nodes of the cluster: 1) look the sstables on the node (do a 'ls' on the data repository). The sstable will look something like Standard1-g-105-Data.db where the 'g' may be a 'f' for some (all?) of the sstable. The 'f' means old sstable, the 'g' means new ones. 2) if all or almost all the big sstables are 'f', then run scrub on that node, that'll be simpler. 3) if only a handfull of sstable are on 'f' (typically, if the node has not just been updated), then what you can do is to force a compaction on those only by using JMX-CompactionManager-forceUserDefinedCompaction, giving it the keyspace name as first argument and the full path to the sstable as second argument. Regards Hector Izquierdo
Cannot recover SSTable with version f (current version g).
Hi guys, I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following exceptions on others nodes. ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main] java.lang.RuntimeException: Cannot recover SSTable with version f (current version g). at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240) at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088) at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108) at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61) at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main] java.lang.RuntimeException: Cannot recover SSTable with version f (current version g). at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240) at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088) at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108) at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61) at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) Do you have any ideas about this? Thanks. -- Dikang Gu 0086 - 18611140205
Re: Cannot recover SSTable with version f (current version g).
Could you please create a bug report for this in Jira https://issues.apache.org/jira/browse/CASSANDRA Please check the data directory on the node that encountered the error and include any files names that have tmp in them. Can you also check for log messages on the node you decommissioned at the INFO level that start with Stream context metadata. it looks like the file version from the old files on the decommissioning node are included in the stream header and it's used to create the new temp file on the node that got this error. So when the it tries to build the other SSTable files from the new data it thinks it's an old data file, even though it's a v0.8 file. v0.8 can read v0.7 data files, but it would not expected to see a new v0.7 data file. If this is the case the only work around I can think of would be to nodetool scrub the node before starting decommission as this would write new data files. It's a bit late here so I may have missed something. Thanks for reporting it. - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 3 Jun 2011, at 22:06, Dikang Gu wrote: Hi guys, I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following exceptions on others nodes. ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main] java.lang.RuntimeException: Cannot recover SSTable with version f (current version g). at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240) at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088) at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108) at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61) at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main] java.lang.RuntimeException: Cannot recover SSTable with version f (current version g). at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240) at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088) at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108) at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61) at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) Do you have any ideas about this? Thanks. -- Dikang Gu 0086 - 18611140205
Re: Cannot recover SSTable with version f (current version g).
Hi Aaron, Thank you for your reply, I've reported a bug on the jira https://issues.apache.org/jira/browse/CASSANDRA-2739 . Since I'm not in the office now, so I will come up with more information and try your methods when I'm in the office again, maybe next Monday. Thanks. -- Dikang Gu 0086 - 18611140205 On Friday, June 3, 2011 at 7:00 PM, aaron morton wrote: Could you please create a bug report for this in Jira https://issues.apache.org/jira/browse/CASSANDRA Please check the data directory on the node that encountered the error and include any files names that have tmp in them. Can you also check for log messages on the node you decommissioned at the INFO level that start with Stream context metadata. it looks like the file version from the old files on the decommissioning node are included in the stream header and it's used to create the new temp file on the node that got this error. So when the it tries to build the other SSTable files from the new data it thinks it's an old data file, even though it's a v0.8 file. v0.8 can read v0.7 data files, but it would not expected to see a new v0.7 data file. If this is the case the only work around I can think of would be to nodetool scrub the node before starting decommission as this would write new data files. It's a bit late here so I may have missed something. Thanks for reporting it. - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 3 Jun 2011, at 22:06, Dikang Gu wrote: Hi guys, I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following exceptions on others nodes. ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main] java.lang.RuntimeException: Cannot recover SSTable with version f (current version g). at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240) at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088) at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108) at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61) at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main] java.lang.RuntimeException: Cannot recover SSTable with version f (current version g). at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240) at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088) at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108) at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61) at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) Do you have any ideas about this? Thanks. -- Dikang Gu 0086 - 18611140205
