Thank you for the reply Aaron. Unfortunately, I could not seem to find any
additional info in the logs. However, upgrading from 2.0.2 to 2.0.3 seems to
have done the trick!
Best regards,
-David Laube
On Dec 11, 2013, at 6:51 PM, Aaron Morton <aa...@thelastpickle.com> wrote:
>> [2013-12-08 11:04:02,047] Repair session
>> ff16c510-5ff7-11e3-97c0-5973cc397f8f for range
>> (1246984843639507027,1266616572749926276] failed with error
>> org.apache.cassandra.exceptions.RepairException: [repair
>> #ff16c510-5ff7-11e3-97c0-5973cc397f8f on keyspace_name/col_family1,
>> (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48
> the 10.x.x.48 node sent a tree response (merkle tree) to this node that did
> not contain the tree. This node then killed the repair session.
>
> Look for log messages on 10.x.x.48 that correlate with the repair session ID
> above. They may look like
>
> logger.error("Failed creating a merkle tree for " + desc + ", " + initiator +
> " (see log for details)”);
>
> or
>
> logger.info(String.format("[repair #%s] Sending completed merkle tree to %s
> for %s/%s", desc.sessionId, initiator, desc.keyspace, desc.columnFamily));
>
> Hope that helps.
>
> -----------------
> Aaron Morton
> New Zealand
> @aaronmorton
>
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On 10/12/2013, at 12:57 pm, Laing, Michael <michael.la...@nytimes.com> wrote:
>
>> My experience is that you must upgrade to 2.0.3 ASAP to fix this.
>>
>> Michael
>>
>>
>> On Mon, Dec 9, 2013 at 6:39 PM, David Laube <d...@stormpath.com> wrote:
>> Hi All,
>>
>> We are running Cassandra 2.0.2 and have recently stumbled upon an issue with
>> nodetool repair. Upon running nodetool repair on each of the 5 nodes in the
>> ring (one at a time) we observe the following exceptions returned to
>> standard out;
>>
>>
>> [2013-12-08 11:04:02,047] Repair session
>> ff16c510-5ff7-11e3-97c0-5973cc397f8f for range
>> (1246984843639507027,1266616572749926276] failed with error
>> org.apache.cassandra.exceptions.RepairException: [repair
>> #ff16c510-5ff7-11e3-97c0-5973cc397f8f on keyspace_name/col_family1,
>> (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48
>> [2013-12-08 11:04:02,063] Repair session
>> 284c8b40-5ff8-11e3-97c0-5973cc397f8f for range
>> (-109256956528331396,-89316884701275697] failed with error
>> org.apache.cassandra.exceptions.RepairException: [repair
>> #284c8b40-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family2,
>> (-109256956528331396,-89316884701275697]] Validation failed in /10.x.x.103
>> [2013-12-08 11:04:02,070] Repair session
>> 399e7160-5ff8-11e3-97c0-5973cc397f8f for range
>> (8901153810410866970,8915879751739915956] failed with error
>> org.apache.cassandra.exceptions.RepairException: [repair
>> #399e7160-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1,
>> (8901153810410866970,8915879751739915956]] Validation failed in /10.x.x.103
>> [2013-12-08 11:04:02,072] Repair session
>> 3ea73340-5ff8-11e3-97c0-5973cc397f8f for range
>> (1149084504576970235,1190026362216198862] failed with error
>> org.apache.cassandra.exceptions.RepairException: [repair
>> #3ea73340-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1,
>> (1149084504576970235,1190026362216198862]] Validation failed in /10.x.x.103
>> [2013-12-08 11:04:02,091] Repair session
>> 6f0da460-5ff8-11e3-97c0-5973cc397f8f for range
>> (-5407189524618266750,-5389231566389960750] failed with error
>> org.apache.cassandra.exceptions.RepairException: [repair
>> #6f0da460-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1,
>> (-5407189524618266750,-5389231566389960750]] Validation failed in /10.x.x.103
>> [2013-12-09 23:16:36,962] Repair session
>> 7efc2740-6127-11e3-97c0-5973cc397f8f for range
>> (1246984843639507027,1266616572749926276] failed with error
>> org.apache.cassandra.exceptions.RepairException: [repair
>> #7efc2740-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family1,
>> (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48
>> [2013-12-09 23:16:36,986] Repair session
>> a8c44260-6127-11e3-97c0-5973cc397f8f for range
>> (-109256956528331396,-89316884701275697] failed with error
>> org.apache.cassandra.exceptions.RepairException: [repair
>> #a8c44260-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family2,
>> (-109256956528331396,-89316884701275697]] Validation failed in /10.x.x.210
>>
>> The /var/log/cassandra/system.log shows similar info as above with no real
>> explanation as to the root cause behind the exception(s). There also does
>> not appear to be any additional info in /var/log/cassandra/cassandra.log. We
>> have tried restoring a recent snapshot of the keyespace in question to a
>> separate staging ring and the repair runs successfully and without exception
>> there. This is even after we tried insert/delete on the keyspace in the
>> separate staging ring. Has anyone seen this behavior before and what can we
>> do to resolve this? Any assistance would be greatly appreciated.
>>
>> Best regards,
>> -Dave
>>
>
