Re: Exceptions when upgrade from 2.1.14 to 2.2.5

2017-04-19 Thread Dikang Gu
Thanks Jeff & Hannu, Yeah, that's my guess too, I walk around this by set -Dcassandra.commitlog.ignorereplayerrors=true. Before upgrade, we did run `nodetool drain`, but seems the 2.1 commit log were not cleared, and still got replayed by 2.2. Thanks Dikang. On Tue, Apr 18, 2017 at 11:09 PM,

Re: Exceptions when upgrade from 2.1.14 to 2.2.5

2017-04-19 Thread Jeff Jirsa
On 2017-04-18 18:57 (-0700), Dikang Gu wrote: > Hello there, > > We are upgrading one of our cluster from 2.1.14 to 2.2.5, but cassandra had > problems replaying the commit logs... > > Here is the exception, does anyone experience similar before? > >

Re: Exceptions when upgrade from 2.1.14 to 2.2.5

2017-04-18 Thread Hannu Kröger
Hello, It seems that commit log is broken. One way to fix this would be to remove commit logs and then restart. This will cause you to lose the writes that were in the commit log but hopefully the data is in other nodes. In the future to avoid this: before you kill Cassandra, run “nodetool

Exceptions when upgrade from 2.1.14 to 2.2.5

2017-04-18 Thread Dikang Gu
Hello there, We are upgrading one of our cluster from 2.1.14 to 2.2.5, but cassandra had problems replaying the commit logs... Here is the exception, does anyone experience similar before? 2017-04-19_00:22:21.69943 DEBUG 00:22:21 [main]: Finished reading