Re: repair cause large number of SSTABLEs

2011-01-27 Thread aaron morton
The ArrayIndexOutOfBounds in the ReadStage looks like it can happen if a key is not of the expected type. Could the comparator for the CF have changed ? The error in the RequestResponseStage may be the race condition identified here https://issues.apache.org/jira/browse/CASSANDRA-1959 Aaron

Re: repair cause large number of SSTABLEs

2011-01-27 Thread Todd Burruss
The comparator has not changed. Sent from my Android phone using TouchDown (www.nitrodesk.com) -Original Message- From: aaron morton [aa...@thelastpickle.com] Received: Thursday, 27 Jan 2011, 1:10am To: user@cassandra.apache.org [user@cassandra.apache.org] Subject: Re: repair cause large

Re: repair cause large number of SSTABLEs

2011-01-27 Thread Matthew Conway
Maybe related to https://issues.apache.org/jira/browse/CASSANDRA-1992 ? On Jan 27, 2011, at Thu Jan 27, 1:22 AM, B. Todd Burruss wrote: i ran out of file handles on the repairing node after doing nodetool repair - strange as i have never had this issue until using 0.7.0 (but i should say

RE: repair cause large number of SSTABLEs

2011-01-27 Thread Todd Burruss
Subject: Re: repair cause large number of SSTABLEs Maybe related to https://issues.apache.org/jira/browse/CASSANDRA-1992 ? On Jan 27, 2011, at Thu Jan 27, 1:22 AM, B. Todd Burruss wrote: i ran out of file handles on the repairing node after doing nodetool repair - strange as i have never had

Re: repair cause large number of SSTABLEs

2011-01-27 Thread Brandon Williams
On Thu, Jan 27, 2011 at 10:21 AM, Todd Burruss bburr...@real.com wrote: thx, but i didn't do anything like removing/adding nodes. just did a nodetool repair after running for an hour or so on a clean install It affects anything that involves streaming. -Brandon

Re: repair cause large number of SSTABLEs

2011-01-27 Thread B. Todd Burruss
ok thx. what about the repair creating hundreds of new sstables and lsof showing cassandra using currently over 800 Data.db files? is this normal? On 01/27/2011 08:40 AM, Brandon Williams wrote: On Thu, Jan 27, 2011 at 10:21 AM, Todd Burruss bburr...@real.com mailto:bburr...@real.com wrote:

Re: repair cause large number of SSTABLEs

2011-01-27 Thread Stu Hood
When the destination node fails to open the streamed SSTable, we assume it was corrupted during transfer, and retry the stream. Independent of the exception posted above, it is a problem that the failed transfers were not cleaned up. How many of the data files are marked as -tmp-? On Jan 27, 2011

Re: repair cause large number of SSTABLEs

2011-01-27 Thread B. Todd Burruss
[cassandra@kv-app02 ~]$ ls -l /data/cassandra-data/data/Queues/*Data.db | grep -c -v \-tmp\- 824 [cassandra@kv-app02 ~]$ ls -l /data/cassandra-data/data/Queues/*-tmp-*Data.db | wc -l 829 [cassandra@kv-app02 ~]$ ls -l /data/cassandra-data/data/Queues/*Comp* | wc -l 247 On 01/27/2011 11:14