Conan,
Good to see I'm not alone in this! I just set up a fresh test cluster. I first
did a fresh install of 1.1.0 and was able to replicate the issue. I then did a
fresh install using 1.0.10 and didn't see the issue. So it looks like rolling
back to 1.0.10 could be the answer for now.
Jeff
On May 11, 2012, at 10:40 AM, Conan Cook wrote:
> Hi,
>
> OK we're pretty sure we dropped and re-created the keyspace before restarting
> the Cassandra nodes during some testing (we've been migrating to a new
> cluster). The keyspace was created via the cli:
>
>
> create keyspace m7
>
> with placement_strategy = 'NetworkTopologyStrategy'
>
> and strategy_options = {us-east: 3}
>
> and durable_writes = true;
>
> I'm pretty confident that it's a result of the issue I spotted before:
>
> https://issues.apache.org/jira/browse/CASSANDRA-4219
>
> Does anyone know whether this also affected versions before 1.1.0? If not
> then we can just roll back until there's a fix; we're not using our cluster
> in production so we can afford to just bin it all and load it again. +1 for
> this being a major issue though, the fact that you can't see it until you
> restart a node makes it quite dangerous, and that node is lost when it occurs
> (I also haven't been able to restore the schema in any way).
>
> Thanks very much,
>
>
> Conan
>
>
>
> On 10 May 2012 17:15, Conan Cook <conan.c...@amee.com> wrote:
> Hi Aaron,
>
> Thanks for getting back to me! Yes, I believe our keyspace was created prior
> to 1.1, and I think I also understand why you're asking that, having found
> this:
>
> https://issues.apache.org/jira/browse/CASSANDRA-4219
>
> Here's our startup log:
>
> https://gist.github.com/2654155
>
> There isn't much in there of interest however. It may well be the case that
> we created our keyspace, dropped it, then created it again. The dev
> responsible for setting it up is ill today, but I'll get back to you tomorrow
> with exact details of how it was originally created and whether we did
> definitely drop and re-create it.
>
> Ta,
>
> Conan
>
>
> On 10 May 2012 11:43, aaron morton <aa...@thelastpickle.com> wrote:
> Was this a schema that was created prior to 1.1 ?
>
> What process are you using to create the schema ?
>
> Can you share the logs from system startup ? Up until it logs "Listening for
> thrift clients". (if they are long please link to them)
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 10/05/2012, at 1:04 AM, Conan Cook wrote:
>
>> Sorry, forgot to mention we're running Cassandra 1.1.
>>
>> Conan
>>
>> On 8 May 2012 17:51, Conan Cook <conan.c...@amee.com> wrote:
>> Hi Cassandra Folk,
>>
>> We've experienced a problem a couple of times where Cassandra nodes lose a
>> keyspace after a restart. We've restarted 2 out of 3 nodes, and they have
>> both experienced this problem; clearly we're doing something wrong, but
>> don't know what. The data files are all still there, as before, but the
>> node can't see the keyspace (we only have one). Tthe nodetool still says
>> that each one is responsible for 33% of the keys, but the disk usage has
>> dropped to a tiny amount on the nodes that we've restarted. I saw this:
>>
>> http://mail-archives.apache.org/mod_mbox/cassandra-user/201202.mbox/%3c4f3582e7.20...@conga.com%3E
>>
>> Seems to be exactly our problem, but we have not modified the cassandra.yaml
>> - we have overwritten it through an automated process, and that happened
>> just before restarting, but the contents did not change.
>>
>> Any ideas as to what might cause this, or how the keyspace can be restored
>> (like I say, the data is all still in the data directory).
>>
>> We're running in AWS.
>>
>> Thanks,
>>
>>
>> Conan
>>
>
>
>
