Re: 3.0.6 - CorruptSSTableException
system.local only contains a single partition with key='local'. There won't be anything to repair for it since it only contains data about that node, e.g. saved tokens, cluster name, etc. Cheers! On Wed, Nov 8, 2017 at 5:03 AM, Riccardo Ferrari wrote: > Thanks you Adama, > > Even though it refers to an older C* version it is still a good starting > point, thanks! > > I ended up restoring the table from a previous snapshot. The server booted > up ok and joined the ring. Hopefully the repair process will take care of > synching back the node. > > Best, > > On Tue, Nov 7, 2017 at 6:54 PM, wrote: > >> Hi Riccardo, >> >> >> >> The following may help me, as the case described there is similar to >> yours ! >> >> https://engineering.gosquared.com/dealing-corrupt-sstable-cassandra >> >> >> >> Regards. >> >> >> >> *De :* Riccardo Ferrari [mailto:ferra...@gmail.com] >> *Envoyé :* lundi 6 novembre 2017 16:16 >> *À :* user@cassandra.apache.org >> *Objet :* 3.0.6 - CorruptSSTableException >> >> >> >> Hi list, >> >> >> >> It happened that one of our EC2 instance of our cluster got rebooted. >> Unfortunately when back Cassandra 3.0.6 failed to restart complaining about: >> >> >> >> ERROR [NonPeriodicTasks:1] 2017-11-04 03:44:20,019 >> LogTransaction.java:204 - Unable to delete >> //system/local/ma-292-big-Data.db >> as it does not exist >> >> ERROR [CompactionExecutor:1] 2017-11-04 03:44:20,136 >> CassandraDaemon.java:195 - Exception in thread >> Thread[CompactionExecutor:1,1,main] >> >> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: >> //system/local/ma-294-big-Data.db >> >> at org.apache.cassandra.io.sstable.format.big.BigTableScanner$K >> eyScanningIterator.computeNext(BigTableScanner.java:351) >> ~[apache-cassandra-3.0.6.jar:3.0.6] >> >> >> >> I tried to run an sstablescrub system local with no luck as it seems to >> get stuck in an endless loop: >> >> >> >> Scrubbing BigTableReader(path='//system/local/ma-293-big-Data.db') >> (5862 bytes) >> >> WARNING: Error reading row (stacktrace follows): >> >> Retrying from row index; data is 5855 bytes starting at 7 >> >> WARNING: Retry failed too. Skipping to next row (retry's stacktrace >> follows) >> >> WARNING: Error reading row (stacktrace follows): >> >> WARNING: Row starting at position 0 is unreadable; skipping to next >> >> WARNING: Data file row position 7 differs from index file row position -1 >> >> WARNING: Error reading row (stacktrace follows): >> >> >> >> I do have a snapshot taken 3 hours before the crash. >> >> Is it safe to restore that snapshot and run a repair? >> >> Is there anything else I can try to recover ? >> >> >> >> Any help much appreciated, >> >> regards >> >> _ >> >> Ce message et ses pieces jointes peuvent contenir des informations >> confidentielles ou privilegiees et ne doivent donc >> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu >> ce message par erreur, veuillez le signaler >> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages >> electroniques etant susceptibles d'alteration, >> Orange decline toute responsabilite si ce message a ete altere, deforme ou >> falsifie. Merci. >> >> This message and its attachments may contain confidential or privileged >> information that may be protected by law; >> they should not be distributed, used or copied without authorisation. >> If you have received this email in error, please notify the sender and >> delete this message and its attachments. >> As emails may be altered, Orange is not liable for messages that have been >> modified, changed or falsified. >> Thank you. >> >> >
Re: 3.0.6 - CorruptSSTableException
Thanks you Adama, Even though it refers to an older C* version it is still a good starting point, thanks! I ended up restoring the table from a previous snapshot. The server booted up ok and joined the ring. Hopefully the repair process will take care of synching back the node. Best, On Tue, Nov 7, 2017 at 6:54 PM, wrote: > Hi Riccardo, > > > > The following may help me, as the case described there is similar to yours > ! > > https://engineering.gosquared.com/dealing-corrupt-sstable-cassandra > > > > Regards. > > > > *De :* Riccardo Ferrari [mailto:ferra...@gmail.com] > *Envoyé :* lundi 6 novembre 2017 16:16 > *À :* user@cassandra.apache.org > *Objet :* 3.0.6 - CorruptSSTableException > > > > Hi list, > > > > It happened that one of our EC2 instance of our cluster got rebooted. > Unfortunately when back Cassandra 3.0.6 failed to restart complaining about: > > > > ERROR [NonPeriodicTasks:1] 2017-11-04 03:44:20,019 LogTransaction.java:204 > - Unable to delete //system/local/ma-292-big-Data.db as it > does not exist > > ERROR [CompactionExecutor:1] 2017-11-04 03:44:20,136 > CassandraDaemon.java:195 - Exception in thread > Thread[CompactionExecutor:1,1,main] > > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > //system/local/ma-294-big-Data.db > > at org.apache.cassandra.io.sstable.format.big.BigTableScanner$ > KeyScanningIterator.computeNext(BigTableScanner.java:351) > ~[apache-cassandra-3.0.6.jar:3.0.6] > > > > I tried to run an sstablescrub system local with no luck as it seems to > get stuck in an endless loop: > > > > Scrubbing BigTableReader(path='//system/local/ma-293-big-Data.db') > (5862 bytes) > > WARNING: Error reading row (stacktrace follows): > > Retrying from row index; data is 5855 bytes starting at 7 > > WARNING: Retry failed too. Skipping to next row (retry's stacktrace > follows) > > WARNING: Error reading row (stacktrace follows): > > WARNING: Row starting at position 0 is unreadable; skipping to next > > WARNING: Data file row position 7 differs from index file row position -1 > > WARNING: Error reading row (stacktrace follows): > > > > I do have a snapshot taken 3 hours before the crash. > > Is it safe to restore that snapshot and run a repair? > > Is there anything else I can try to recover ? > > > > Any help much appreciated, > > regards > > _ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete > this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > >
RE: 3.0.6 - CorruptSSTableException
Hi Riccardo, The following may help me, as the case described there is similar to yours ! https://engineering.gosquared.com/dealing-corrupt-sstable-cassandra Regards. De : Riccardo Ferrari [mailto:ferra...@gmail.com] Envoyé : lundi 6 novembre 2017 16:16 À : user@cassandra.apache.org Objet : 3.0.6 - CorruptSSTableException Hi list, It happened that one of our EC2 instance of our cluster got rebooted. Unfortunately when back Cassandra 3.0.6 failed to restart complaining about: ERROR [NonPeriodicTasks:1] 2017-11-04 03:44:20,019 LogTransaction.java:204 - Unable to delete //system/local/ma-292-big-Data.db as it does not exist ERROR [CompactionExecutor:1] 2017-11-04 03:44:20,136 CassandraDaemon.java:195 - Exception in thread Thread[CompactionExecutor:1,1,main] org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: //system/local/ma-294-big-Data.db at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:351) ~[apache-cassandra-3.0.6.jar:3.0.6] I tried to run an sstablescrub system local with no luck as it seems to get stuck in an endless loop: Scrubbing BigTableReader(path='//system/local/ma-293-big-Data.db') (5862 bytes) WARNING: Error reading row (stacktrace follows): Retrying from row index; data is 5855 bytes starting at 7 WARNING: Retry failed too. Skipping to next row (retry's stacktrace follows) WARNING: Error reading row (stacktrace follows): WARNING: Row starting at position 0 is unreadable; skipping to next WARNING: Data file row position 7 differs from index file row position -1 WARNING: Error reading row (stacktrace follows): I do have a snapshot taken 3 hours before the crash. Is it safe to restore that snapshot and run a repair? Is there anything else I can try to recover ? Any help much appreciated, regards _ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
