Re: ghost table is breaking compactions and won't go away… even during a drop.
In my experience, SSTable FileNotFoundException, not only caused by recreate a table but also other operations or even bug, cannot be solved by any nodetool command. However, restart the node for more than one time can make this Exception disappear. I don't know the reason but it does work... Thanks, Philo Yang 2014-07-17 10:32 GMT+08:00 Kevin Burton bur...@spinn3r.com: you rock… glad it's fixed in 2.1… :) On Wed, Jul 16, 2014 at 7:05 PM, graham sanderson gra...@vast.com wrote: Known issue deleting and recreating a CF with the same name, fixed in 2.1 (manifests in lots of ways) https://issues.apache.org/jira/browse/CASSANDRA-5202 On Jul 16, 2014, at 8:53 PM, Kevin Burton bur...@spinn3r.com wrote: looks like a restart of cassandra and a nodetool compact fixed this… On Wed, Jul 16, 2014 at 6:45 PM, Kevin Burton bur...@spinn3r.com wrote: this is really troubling… I have a ghost table. I dropped it.. but it's not going away. (Cassandra 2.0.8 btw) I ran a 'drop table' on it.. then a 'describe tables' shows that it's not there. However, when I recreated it, with a new schema, all operations on it failed. Looking at why… it seems that cassandra had some old SSTables that I imagine are no longer being used but are now in an inconsistent state? This is popping up in the system.log: Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /d0/cassandra/data/blogindex/content_idx_source_hashcode/blogindex-content_idx_source_hashcode-jb-1447-Data.db (No such file or directory) so I think what happened… is that the original drop table, failed, and then left things in an inconsistent state. I tried a nodetool repair and a nodetool compact… those fail on the same java.io.FileNotFoundException … I moved the directories out of the way, same failure issue. … any advice on resolving this? -- Founder/CEO Spinn3r.com http://spinn3r.com/ Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com/ -- Founder/CEO Spinn3r.com http://spinn3r.com/ Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com/ -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: ghost table is breaking compactions and won't go away… even during a drop.
looks like a restart of cassandra and a nodetool compact fixed this… On Wed, Jul 16, 2014 at 6:45 PM, Kevin Burton bur...@spinn3r.com wrote: this is really troubling… I have a ghost table. I dropped it.. but it's not going away. (Cassandra 2.0.8 btw) I ran a 'drop table' on it.. then a 'describe tables' shows that it's not there. However, when I recreated it, with a new schema, all operations on it failed. Looking at why… it seems that cassandra had some old SSTables that I imagine are no longer being used but are now in an inconsistent state? This is popping up in the system.log: Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /d0/cassandra/data/blogindex/content_idx_source_hashcode/blogindex-content_idx_source_hashcode-jb-1447-Data.db (No such file or directory) so I think what happened… is that the original drop table, failed, and then left things in an inconsistent state. I tried a nodetool repair and a nodetool compact… those fail on the same java.io.FileNotFoundException … I moved the directories out of the way, same failure issue. … any advice on resolving this? -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: ghost table is breaking compactions and won't go away… even during a drop.
Known issue deleting and recreating a CF with the same name, fixed in 2.1 (manifests in lots of ways) https://issues.apache.org/jira/browse/CASSANDRA-5202 On Jul 16, 2014, at 8:53 PM, Kevin Burton bur...@spinn3r.com wrote: looks like a restart of cassandra and a nodetool compact fixed this… On Wed, Jul 16, 2014 at 6:45 PM, Kevin Burton bur...@spinn3r.com wrote: this is really troubling… I have a ghost table. I dropped it.. but it's not going away. (Cassandra 2.0.8 btw) I ran a 'drop table' on it.. then a 'describe tables' shows that it's not there. However, when I recreated it, with a new schema, all operations on it failed. Looking at why… it seems that cassandra had some old SSTables that I imagine are no longer being used but are now in an inconsistent state? This is popping up in the system.log: Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /d0/cassandra/data/blogindex/content_idx_source_hashcode/blogindex-content_idx_source_hashcode-jb-1447-Data.db (No such file or directory) so I think what happened… is that the original drop table, failed, and then left things in an inconsistent state. I tried a nodetool repair and a nodetool compact… those fail on the same java.io.FileNotFoundException … I moved the directories out of the way, same failure issue. … any advice on resolving this? -- Founder/CEO Spinn3r.com Location: San Francisco, CA blog: http://burtonator.wordpress.com … or check out my Google+ profile -- Founder/CEO Spinn3r.com Location: San Francisco, CA blog: http://burtonator.wordpress.com … or check out my Google+ profile smime.p7s Description: S/MIME cryptographic signature
Re: ghost table is breaking compactions and won't go away… even during a drop.
you rock… glad it's fixed in 2.1… :) On Wed, Jul 16, 2014 at 7:05 PM, graham sanderson gra...@vast.com wrote: Known issue deleting and recreating a CF with the same name, fixed in 2.1 (manifests in lots of ways) https://issues.apache.org/jira/browse/CASSANDRA-5202 On Jul 16, 2014, at 8:53 PM, Kevin Burton bur...@spinn3r.com wrote: looks like a restart of cassandra and a nodetool compact fixed this… On Wed, Jul 16, 2014 at 6:45 PM, Kevin Burton bur...@spinn3r.com wrote: this is really troubling… I have a ghost table. I dropped it.. but it's not going away. (Cassandra 2.0.8 btw) I ran a 'drop table' on it.. then a 'describe tables' shows that it's not there. However, when I recreated it, with a new schema, all operations on it failed. Looking at why… it seems that cassandra had some old SSTables that I imagine are no longer being used but are now in an inconsistent state? This is popping up in the system.log: Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /d0/cassandra/data/blogindex/content_idx_source_hashcode/blogindex-content_idx_source_hashcode-jb-1447-Data.db (No such file or directory) so I think what happened… is that the original drop table, failed, and then left things in an inconsistent state. I tried a nodetool repair and a nodetool compact… those fail on the same java.io.FileNotFoundException … I moved the directories out of the way, same failure issue. … any advice on resolving this? -- Founder/CEO Spinn3r.com http://spinn3r.com/ Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com/ -- Founder/CEO Spinn3r.com http://spinn3r.com/ Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com/ -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
