The error which keeps it from starting is below.
The files like
"mc_txn_anticompactionafterrepair_19a46410-459f-11e7-91c7-4f4e8666b5c8.log"
are on both disks of a node but are different .
Of course, just renaming (deleting) the two files (or making them equal)
makes cassandra start again. But I would like to know the right way to
handle this.
I start the repair again with increased log level.
thanks for answering,
Michael
ERROR 08:26:20 Mismatched line in file
mc_txn_anticompactionafterrepair_19a46410-459f-11e7-91c7-4f4e8666b5c8.log:
got
'ADD:[/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4348-big,0,8][1394849421]'
expected
'ADD:[/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4349-big,0,8][910462411]',
giving up
ERROR 08:26:20 Failed to read records for transaction log
[mc_txn_anticompactionafterrepair_19a46410-459f-11e7-91c7-4f4e8666b5c8.log
in
/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0,
/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0]
ERROR 08:26:20 Unexpected disk state: failed to read transaction log
[mc_txn_anticompactionafterrepair_19a46410-459f-11e7-91c7-4f4e8666b5c8.log
in
/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0,
/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0]
Files and contents follow:
/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc_txn_anticompactionafterrepair_19a46410-459f-11e7-91c7-4f4e8666b5c8.log
ADD:[/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4349-big,0,8][910462411]
REMOVE:[/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4241-big,1495845618000,8][2443235315]
REMOVE:[/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4249-big,1495856254000,8][681858089]
COMMIT:[,0,0][2613697770]
/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc_txn_anticompactionafterrepair_19a46410-459f-11e7-91c7-4f4e8666b5c8.log
ADD:[/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4348-big,0,8][1394849421]
***Does not match
<ADD:[/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4349-big,0,8][910462411]>
in first replica file
ADD:[/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4349-big,0,8][910462411]
REMOVE:[/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4241-big,1495845618000,8][2443235315]
REMOVE:[/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4249-big,1495856254000,8][681858089]
COMMIT:[,0,0][2613697770]
On 31.05.2017 11:10, Oleksandr Shulgin wrote:
> On Wed, May 31, 2017 at 9:11 AM, Micha <[email protected]
> <mailto:[email protected]>> wrote:
>
> Hi,
>
> after failed repair on a three node cluster all nodes were down.
>
>
> To clarify, was it failed repair that brought the nodes down so that you
> had to start them back? Do you see any error messages or stack trace in
> the logs?
>
>
> It cannot start, since it finds a mismatch in a
> mc_txn_anticompactionafterrepair log file:
> "got ADD ...."
> "expected "ADD:..."
>
>
> The two log files are different:
> one has "ADD, ADD; REMOVE, REMOVE, COMMIT"
> the other is missing an "ADD"
>
>
> I assume this is about commit log. There doesn't seem to be a separate
> log file named "mc_txn_anticompactionafterrepair" in your Cassandra version.
>
> Each of the nodes give this error.
>
> sstableutil -c also gives this error.
>
> How to deal with this?
>
>
> I would try removing the faulty commit log file(s) and try to start the
> node again, until it works. This might mean that you'll have to remove
> all commit logs, but it's better than being completely down, I assume.
>
> --
> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
> 127-59-707
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
