Re: Cassandra HANGS after some writes
Naresh, are you deploying cassandra in windows? If that is the case you may need to change the data and commitlog directories in cassandra.yaml. Also you should check the log directories. See the section 2.1 http://wiki.apache.org/cassandra/GettingStarted On Tue, Aug 13, 2013 at 8:28 AM, Naresh Yadav nyadav@gmail.com wrote: Thanks Alain, will avoid capsi am newbie to cassandra, just started using 2 weeks back.. Here are JConsole screenshots just 5mins after cassandra freezed : http://i.imgur.com/3oUBjKU.png http://i.imgur.com/2O4PrKb.png http://i.imgur.com/zxhFzr1.png 4:05 is time cassandra freezed thats why decline in no of threads http://i.imgur.com/ScgAciv.png Uploaded complete system.log of cassandra till freeze : http://www.scribd.com/doc/159949231/Cassandrasystem-log Observation : As in my usecase i am storing 1lakh combinations(527insert,506update,954select) each parallel by 100 threads in batch of 1000... Sometimes it works till 1000 batch then hangs but sometimes it completes 1 then hangs and once even worked for more than lakh Same hardware Same settings of cassandra i see random behaviour of performance.. Thanks Naresh On Tue, Aug 13, 2013 at 3:48 PM, Alain RODRIGUEZ arodr...@gmail.comwrote: Hi Naresh. First thing, there is no need of caps in here. People reading this ML is here to help when they have time and skills enough to do so. So please, chill out and do not use caps to show how much desperate you are. Concerning your problem, the only abnormal thing I was able to find in your logs is 1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106 SSTableDeletingTask.java (line 72) Unable to delete \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db (it will be removed on server restart; we'll also retry after GC) I don't think this should keep C* hanging. Do you have something on kernel logs ? Do you have monitor any metrics like disk throughput / heap used / cpu load / iowait which are known as being bottlenecks / pertinent metrics ? Alain 2013/8/13 Naresh Yadav nyadav@gmail.com Hi All, I have single node cassandra using CQL using datastax java driver 1.0.1 and cassandra verison 1.2.6. *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7 processor.. DEFAULT cassandra.yaml no change done by me. -Xms1G^ -Xmx12G^ no other change in cassandra.bat *Problem : _ *cassandra Freezes after some writes and i see no action on cassandra console for an hour...all Native_Transport threads are also killedmy program keeps running NO ERROR comes...when i connect with cql that works In start it creates 16 NativeTransport threads and after 10-15 minutes Total threads goes to 128...Just before it hangs, With JCONSOLE when i see Native_Transport threads then i find most of them in state as : http://pastebin.com/DeShpHtP *Load on cassandra : * ___ i am running a usecase which stores Combinations(my project terminology) in cassandraCurrently testing storing 2.5 lakh combinations with 100 parallel threads..each thread storing one combination...real case i need to support of many CRORES but that would need different hardware and multi node cluster... In Storing ONE combination takes around 2sec and involves : 527 INSERT INTO queries 506 UPDATE queries 954 SELECT queries 100 parallel threads parallely storing 100 combinations *MY CASSANDRA LOGS :* http://pastebin.com/CnNvA9x3 Please look last 100-200 lines of log because that is time it freezed PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
Re: Data disappear immediately after reading?
Carlo, Do you read/write with the consistency levels according to your needs [1]? Have you tried to see if it happens when using the cassandra-cli to get that data? [1] http://wiki.apache.org/cassandra/ArchitectureOverview On Wed, Jul 24, 2013 at 5:34 PM, cbert...@libero.it cbert...@libero.itwrote: Hi all, I know the subject is not saying much but this is what I'm experiencing now with my cluster. After some years without any problem now I'm experiencing problems with counters but, the most serious problem, is data loss immediately after a read. I have some webservices that I use to query data on Cassandra but in the last month happened 2 times the following problem: I call my WS, it shows data. I refresh the page -- data are no more available! I can call then 200 times the WS but I won't see data anymore ... today my colleague experienced the same problem. The WS are ABSOLUTELY read only on the DB and there are no write to erase these data. Anyone understand wth is going on? I have no idea but most of all I don't know how to fix. Any help would really be appreciated. Kind Regards, Carlo
Re: [deletion in the future]
Richard, That's exactly what is happening with my row, but not what I was trying to do. It seems that I misunderstood the stackoverflow post. I was trying to schedule a delete for an entire row, is using ttl for columns the only way? Thanks for the reply. On Sat, Jul 20, 2013 at 5:36 AM, Richard Low rich...@wentnet.com wrote: On 19 July 2013 23:31, Alexis Rodríguez arodrig...@inconcertcc.comwrote: Hi guys, I've read here [1] that you can make a deletion mutation for the future. That mechanism operates as a schedule for deletions according to the stackoverflow post. But, I've been having problems to make it work with my thrift c++ client. I believe it's related to this paragraph of the thrift api documentation: What problem are you having? If you insert a tombstone with a future timestamp, it means delete what is there now, as well as anything that is inserted in the future up to the tombstone timestamp. It does not mean for the tombstone to only kick in in the future, that is what TTLs do. Richard.
Re: Problem with libcassandra
That repo for libcassandra works for cassandra 0.7.x due to changes in the
thrift interface we have faced some problems in the past.
May be you can take a look to my fork of libcassandra https://github.com/axs
-mvd/libcassandra that we are using with cassandra 1.1.11.
Besides that, I recommend that you use the cassandra-cli to create
keyspacesand column families because that operations fails if some of
your nodes is
down, so by using cassandra-cli you force, in a way, the supervision of a
human being :D.
good luck!
On Wed, Jul 3, 2013 at 9:12 AM, Shubham Mittal smsmitta...@gmail.comwrote:
Hey,
I found out that the problem is caused by this line :
c-createKeyspace(ks_def);
because the below code works fine.
#include string.h
#include sstream
#include iostream
#include stdlib.h
#include set
#include string
#include stdio.h
#include vector
#include libcassandra/cassandra_factory.h
#include libcassandra/cassandra.h
#include libcassandra/column_family_definition.h
#include libcassandra/keyspace.h
#include libcassandra/keyspace_definition.h
using namespace std;
using namespace libcassandra;
static string host(127.0.0.1);
static int port= 9160;
int main()
{
CassandraFactory cf(host, port);
tr1::shared_ptrCassandra c(cf.create());
KeyspaceDefinition ks_def;
ks_def.setName(demo);
return 0;
}
On Wed, Jul 3, 2013 at 5:22 PM, Shubham Mittal smsmitta...@gmail.comwrote:
no Jordan, the cassandra version I have is Cassandra 1.1.12
On Wed, Jul 3, 2013 at 5:21 PM, Shubham Mittal smsmitta...@gmail.comwrote:
This is the gdb output
[Thread debugging using libthread_db enabled]
terminate called after throwing an instance of
'org::apache::cassandra::InvalidRequestException'
what(): Default TException.
Program received signal SIGABRT, Aborted.
0x770a0b25 in raise () from /lib/libc.so.6
On Wed, Jul 3, 2013 at 8:38 AM, Jeremiah D Jordan
jeremiah.jor...@gmail.com wrote:
If you are using 1.2, I would checkout https://github.com/mstump/libcql
-Jeremiah
On Jul 2, 2013, at 5:18 AM, Shubham Mittal smsmitta...@gmail.com
wrote:
I am trying to run below code, but it gives this error. It compiles
without any errors. Kindly help me.
(source of the code :
http://posulliv.github.io/2011/02/27/libcassandra-sec-indexes/)
terminate called after throwing an instance of
'org::apache::cassandra::InvalidRequestException'
what(): Default TException.
Aborted
#include string.h
#include sstream
#include iostream
#include stdlib.h
#include set
#include string
#include stdio.h
#include vector
#include libcassandra/cassandra_factory.h
#include libcassandra/cassandra.h
#include libcassandra/column_family_definition.h
#include libcassandra/keyspace.h
#include libcassandra/keyspace_definition.h
using namespace std;
using namespace libcassandra;
static string host(127.0.0.1);
static int port= 9160;
int main()
{
CassandraFactory cf(host, port);
tr1::shared_ptrCassandra c(cf.create());
KeyspaceDefinition ks_def;
ks_def.setName(demo);
c-createKeyspace(ks_def);
ColumnFamilyDefinition cf_def;
cf_def.setName(users);
cf_def.setKeyspaceName(ks_def.getName());
ColumnDefinition name_col;
name_col.setName(full_name);
name_col.setValidationClass(UTF8Type);
ColumnDefinition sec_col;
sec_col.setName(birth_date);
sec_col.setValidationClass(LongType);
sec_col.setIndexType(org::apache::cassandra::IndexType::KEYS);
ColumnDefinition third_col;
third_col.setName(state);
third_col.setValidationClass(UTF8Type);
third_col.setIndexType(org::apache::cassandra::IndexType::KEYS);
cf_def.addColumnMetadata(name_col);
cf_def.addColumnMetadata(sec_col);
cf_def.addColumnMetadata(third_col);
c-setKeyspace(ks_def.getName());
c-createColumnFamily(cf_def);
return 0;
}
Re: Problem with libcassandra
Shubham,
You are right, my point is that with non schema-update thrift calls you can
tune the consistency level used.
bye.
On Wed, Jul 3, 2013 at 10:10 AM, Shubham Mittal smsmitta...@gmail.comwrote:
hi Alexis,
Even if I create keyspaces, column families using cassandra-cli, the
column creation and insertion work will still need thrift calls.
On Wed, Jul 3, 2013 at 6:05 PM, Alexis Rodríguez
arodrig...@inconcertcc.com wrote:
That repo for libcassandra works for cassandra 0.7.x due to changes in
the thrift interface we have faced some problems in the past.
May be you can take a look to my fork of libcassandra https://github.com/
axs-mvd/libcassandra that we are using with cassandra 1.1.11.
Besides that, I recommend that you use the cassandra-cli to create
keyspaces and column families because that operations fails if some of
your nodes is down, so by using cassandra-cli you force, in a way, the
supervision of a human being :D.
good luck!
On Wed, Jul 3, 2013 at 9:12 AM, Shubham Mittal smsmitta...@gmail.comwrote:
Hey,
I found out that the problem is caused by this line :
c-createKeyspace(ks_def);
because the below code works fine.
#include string.h
#include sstream
#include iostream
#include stdlib.h
#include set
#include string
#include stdio.h
#include vector
#include libcassandra/cassandra_factory.h
#include libcassandra/cassandra.h
#include libcassandra/column_family_definition.h
#include libcassandra/keyspace.h
#include libcassandra/keyspace_definition.h
using namespace std;
using namespace libcassandra;
static string host(127.0.0.1);
static int port= 9160;
int main()
{
CassandraFactory cf(host, port);
tr1::shared_ptrCassandra c(cf.create());
KeyspaceDefinition ks_def;
ks_def.setName(demo);
return 0;
}
On Wed, Jul 3, 2013 at 5:22 PM, Shubham Mittal smsmitta...@gmail.comwrote:
no Jordan, the cassandra version I have is Cassandra 1.1.12
On Wed, Jul 3, 2013 at 5:21 PM, Shubham Mittal
smsmitta...@gmail.comwrote:
This is the gdb output
[Thread debugging using libthread_db enabled]
terminate called after throwing an instance of
'org::apache::cassandra::InvalidRequestException'
what(): Default TException.
Program received signal SIGABRT, Aborted.
0x770a0b25 in raise () from /lib/libc.so.6
On Wed, Jul 3, 2013 at 8:38 AM, Jeremiah D Jordan
jeremiah.jor...@gmail.com wrote:
If you are using 1.2, I would checkout
https://github.com/mstump/libcql
-Jeremiah
On Jul 2, 2013, at 5:18 AM, Shubham Mittal smsmitta...@gmail.com
wrote:
I am trying to run below code, but it gives this error. It compiles
without any errors. Kindly help me.
(source of the code :
http://posulliv.github.io/2011/02/27/libcassandra-sec-indexes/)
terminate called after throwing an instance of
'org::apache::cassandra::InvalidRequestException'
what(): Default TException.
Aborted
#include string.h
#include sstream
#include iostream
#include stdlib.h
#include set
#include string
#include stdio.h
#include vector
#include libcassandra/cassandra_factory.h
#include libcassandra/cassandra.h
#include libcassandra/column_family_definition.h
#include libcassandra/keyspace.h
#include libcassandra/keyspace_definition.h
using namespace std;
using namespace libcassandra;
static string host(127.0.0.1);
static int port= 9160;
int main()
{
CassandraFactory cf(host, port);
tr1::shared_ptrCassandra c(cf.create());
KeyspaceDefinition ks_def;
ks_def.setName(demo);
c-createKeyspace(ks_def);
ColumnFamilyDefinition cf_def;
cf_def.setName(users);
cf_def.setKeyspaceName(ks_def.getName());
ColumnDefinition name_col;
name_col.setName(full_name);
name_col.setValidationClass(UTF8Type);
ColumnDefinition sec_col;
sec_col.setName(birth_date);
sec_col.setValidationClass(LongType);
sec_col.setIndexType(org::apache::cassandra::IndexType::KEYS);
ColumnDefinition third_col;
third_col.setName(state);
third_col.setValidationClass(UTF8Type);
third_col.setIndexType(org::apache::cassandra::IndexType::KEYS);
cf_def.addColumnMetadata(name_col);
cf_def.addColumnMetadata(sec_col);
cf_def.addColumnMetadata(third_col);
c-setKeyspace(ks_def.getName());
c-createColumnFamily(cf_def);
return 0;
}
Re: Sudden increase in diskspace usage
Nicolai, Perhaps you can check the system.log to see if there are any errors on compaction. Also, I believe C* 1.2.0 it's not a stable version. On Thu, May 9, 2013 at 2:43 AM, Nicolai Gylling n...@issuu.com wrote: Hi I have a 3-node SSD-based cluster, with around 1 TB data, RF:3, C* v.1.2.0, vnodes. One large CF, LCS. Everything was running smooth, until one of the nodes crashed and was restarted. At the time of normal operation there was 800 gb free space on each node. After the crash, C* started using a lot more, resulting in an out-of-diskspace situation on 2 nodes, eg. C* used up the 800 gb in just 2 days, giving us very little time to do anything about it, since repairs/joins takes a considerable amount of time. What can make C* suddenly use this amount of disk-space? We did see a lot of pending compactions on one node (7k). Any tips on recovering from an out-of-diskspace on multiple nodes, situation? I've tried moving some SStables away, but C* seems to use whatever space I free up in no time. I'm not sure if any of the nodes is fully updated as 'nodetool status' reports 3 different loads -- Address Load Tokens Owns (effective) Host ID Rack UN 10.146.145.26 1.4 TB 256 100.0% 1261717d-ddc1-457e-9c93-431b3d3b5c5b rack1 UN 10.148.149.1411.03 TB256 100.0% f80bfa31-e19d-4346-9a14-86ae87f06356 rack1 DN 10.146.146.4 1.11 TB256 100.0% 85d4cd28-93f4-4b96-8140-3605302e90a9 rack1 -- Sincerely, *Nicolai Gylling*
Re: How to make compaction run faster?
Jay, Do you have metrics of disk usage on the disks that contains your data directories? Compaction operates over those files, may be your problems are with those disks and not with the disks that have the commitlog. On Thu, Apr 18, 2013 at 1:33 PM, Jay Svc jaytechg...@gmail.com wrote: Hi Alexis, Yes compaction happens on data files - 1. What my disk latency is high for SSDs which been only used for commit log 2. Why my compaction is not catching up with my write traffic in spite of low CPU, low memory and low JVM usage. I am adding more details to this thread. Thanks, Jayant K Kenjale On Wed, Apr 17, 2013 at 6:58 PM, Alexis Rodríguez arodrig...@inconcertcc.com wrote: Jay, I believe that compaction occurs on the data directories and not in the commitlog. http://wiki.apache.org/cassandra/MemtableSSTable On Wed, Apr 17, 2013 at 7:58 PM, Jay Svc jaytechg...@gmail.com wrote: Hi Alexis, Thank you for your response. My commit log is on SSD. which shows me 30 to 40 ms of disk latency. When I ran iostat; I see await 26ms to 30 ms for my commit log disk. My CPU is less than 18% used. How I reduce the disk latency for my commit log disk. They are SSDs. Thank you in advance, Jay On Wed, Apr 17, 2013 at 3:58 PM, Alexis Rodríguez arodrig...@inconcertcc.com wrote: :D Jay, check if your disk(s) utilization allows you to change the configuration the way Edward suggest. iostat -xkcd 1 will show you how much of your disk(s) are in use. On Wed, Apr 17, 2013 at 5:26 PM, Edward Capriolo edlinuxg...@gmail.com wrote: three things: 1) compaction throughput is fairly low (yaml nodetool) 2) concurrent compactions is fairly low (yaml) 3) multithreaded compaction might be off in your version Try raising these things. Otherwise consider option 4. 4)$$$ RAID,RAMCPU$$ On Wed, Apr 17, 2013 at 4:01 PM, Jay Svc jaytechg...@gmail.comwrote: Hi Team, I have a high write traffic to my Cassandra cluster. I experience a very high number of pending compactions. As I expect higher writes, The pending compactions keep increasing. Even when I stop my writes it takes several hours to finishing pending compactions. My CF is configured with LCS, with sstable_size_mb=20M. My CPU is below 20%, JVM memory usage is between 45%-55%. I am using Cassandra 1.1.9. How can I increase the compaction rate so it will run bit faster to match my write speed? Your inputs are appreciated. Thanks, Jay
Re: How to make compaction run faster?
some information on how many writes you are processing the the size of rows as well. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 18/04/2013, at 11:58 AM, Alexis Rodríguez arodrig...@inconcertcc.com wrote: Jay, I believe that compaction occurs on the data directories and not in the commitlog. http://wiki.apache.org/cassandra/MemtableSSTable On Wed, Apr 17, 2013 at 7:58 PM, Jay Svc jaytechg...@gmail.com wrote: Hi Alexis, Thank you for your response. My commit log is on SSD. which shows me 30 to 40 ms of disk latency. When I ran iostat; I see await 26ms to 30 ms for my commit log disk. My CPU is less than 18% used. How I reduce the disk latency for my commit log disk. They are SSDs. Thank you in advance, Jay On Wed, Apr 17, 2013 at 3:58 PM, Alexis Rodríguez arodrig...@inconcertcc.com wrote: :D Jay, check if your disk(s) utilization allows you to change the configuration the way Edward suggest. iostat -xkcd 1 will show you how much of your disk(s) are in use. On Wed, Apr 17, 2013 at 5:26 PM, Edward Capriolo edlinuxg...@gmail.com wrote: three things: 1) compaction throughput is fairly low (yaml nodetool) 2) concurrent compactions is fairly low (yaml) 3) multithreaded compaction might be off in your version Try raising these things. Otherwise consider option 4. 4)$$$ RAID,RAMCPU$$ On Wed, Apr 17, 2013 at 4:01 PM, Jay Svc jaytechg...@gmail.com wrote: Hi Team, I have a high write traffic to my Cassandra cluster. I experience a very high number of pending compactions. As I expect higher writes, The pending compactions keep increasing. Even when I stop my writes it takes several hours to finishing pending compactions. My CF is configured with LCS, with sstable_size_mb=20M. My CPU is below 20%, JVM memory usage is between 45%-55%. I am using Cassandra 1.1.9. How can I increase the compaction rate so it will run bit faster to match my write speed? Your inputs are appreciated. Thanks, Jay
Re: How to make compaction run faster?
:D Jay, check if your disk(s) utilization allows you to change the configuration the way Edward suggest. iostat -xkcd 1 will show you how much of your disk(s) are in use. On Wed, Apr 17, 2013 at 5:26 PM, Edward Capriolo edlinuxg...@gmail.comwrote: three things: 1) compaction throughput is fairly low (yaml nodetool) 2) concurrent compactions is fairly low (yaml) 3) multithreaded compaction might be off in your version Try raising these things. Otherwise consider option 4. 4)$$$ RAID,RAMCPU$$ On Wed, Apr 17, 2013 at 4:01 PM, Jay Svc jaytechg...@gmail.com wrote: Hi Team, I have a high write traffic to my Cassandra cluster. I experience a very high number of pending compactions. As I expect higher writes, The pending compactions keep increasing. Even when I stop my writes it takes several hours to finishing pending compactions. My CF is configured with LCS, with sstable_size_mb=20M. My CPU is below 20%, JVM memory usage is between 45%-55%. I am using Cassandra 1.1.9. How can I increase the compaction rate so it will run bit faster to match my write speed? Your inputs are appreciated. Thanks, Jay
Re: CorruptedBlockException
Lanny,
We will try that.
Thanks a lot
On Thu, Apr 11, 2013 at 11:13 PM, Lanny Ripple la...@spotright.com wrote:
Saw this in earlier versions. Our workaround was disable; drain; snap;
shutdown; delete; link from snap; restart;
-ljr
On Apr 11, 2013, at 9:45, moshe.kr...@barclays.com wrote:
I have formulated the following theory regarding C* 1.2.2 which may be
relevant: Whenever there is a disk error during compaction of an SS table
(e.g., bad block, out of disk space), that SStable’s files stick around
forever after, and do not subsequently get deleted by normal compaction
(minor or major), long after all its records have been deleted. This causes
disk usage to rise dramatically. The only way to make the SStable files
disappear is to run “nodetool cleanup” (which takes hours to run).
** **
Just a theory so far….
** **
*From:* Alexis Rodríguez
[mailto:arodrig...@inconcertcc.comarodrig...@inconcertcc.com]
*Sent:* Thursday, April 11, 2013 5:31 PM
*To:* user@cassandra.apache.org
*Subject:* Re: CorruptedBlockException
** **
Aaron,
** **
It seems that we are in the same situation as Nury, we are storing a lot
of files of ~5MB in a CF.
** **
This happens in a test cluster, with one node using cassandra 1.1.5, we
have commitlog in a different partition than the data directory. Normally
our tests use nearly 13 GB in data, but when the exception on compaction
appears our disk space ramp up to:
** **
# df -h
FilesystemSize Used Avail Use% Mounted on
/dev/sda1 440G 330G 89G 79% /
tmpfs 7.9G 0 7.9G 0% /lib/init/rw
udev 7.9G 160K 7.9G 1% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
/dev/sdb1 459G 257G 179G 59% /cassandra
** **
# cd /cassandra/data/Repository/
** **
# ls Files/*tmp* | wc -l
1671
** **
# du -ch Files | tail -1
257Gtotal
** **
# du -ch Files/*tmp* | tail -1
34G total
** **
We are using cassandra 1.1.5 with one node, our schema for that keyspace
is:
** **
[default@unknown] use Repository;
Authenticated to keyspace: Repository
[default@Repository] show schema;
create keyspace Repository
with placement_strategy = 'NetworkTopologyStrategy'
and strategy_options = {datacenter1 : 1}
and durable_writes = true;
** **
use Repository;
** **
create column family Files
with column_type = 'Standard'
and comparator = 'UTF8Type'
and default_validation_class = 'BytesType'
and key_validation_class = 'BytesType'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy =
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'
and caching = 'KEYS_ONLY'
and compaction_strategy_options = {'sstable_size_in_mb' : '120'}
and compression_options = {'sstable_compression' :
'org.apache.cassandra.io.compress.SnappyCompressor'};
** **
In our logs:
** **
ERROR [CompactionExecutor:1831] 2013-04-11 09:12:41,725
AbstractCassandraDaemon.java (line 135) Exception in thread
Thread[CompactionExecutor:1831,1,main]
java.io.IOError: org.apache.cassandra.io.compress.CorruptedBlockException:
(/cassandra/data/Repository/Files/Repository-Files-he-4533-Data.db):
corruption detected, chunk at 43325354 of length 65545.
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116)
at
org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:99)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:176)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:83)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:68)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135
Re: Repair hanges on 1.1.4
Adeel, It may be a problem in the remote node, could you check the system.log? Also you might want to check the rpc_timeout_in_ms in both nodes, maybe an increase in this parameter helps. On Fri, Apr 12, 2013 at 9:17 AM, adeel.ak...@panasiangroup.com wrote: Hi, I have started repair on newly added node with -pr and this nodes exist on another data center. I have 5MB internet connection and configured setstreamthroughput 1. After some time repair goes hang and following meesage found in logs; # /opt/apache-cassandra-1.1.4/**bin/nodetool -h localhost ring Address DC RackStatus State Load Effective-Ownership Token 169417178424467235000914166253**263322299 10.0.0.3DC1 RAC1Up Normal 93.26 GB 66.67% 0 10.0.0.4DC1 RAC1Up Normal 89.1 GB 66.67% 567137278201564105772291012386**28035242 10.0.0.15 DC1 RAC1Up Normal 72.87 GB 66.67% 113427455640312821154458202477**256070484 10.40.1.103 DC2 RAC1Up Normal 48.59 GB 100.00% 169417178424467235000914166253**263322299 INFO [HintedHandoff:1] 2013-04-12 17:05:49,411 HintedHandOffManager.java (line 372) Timed out replaying hints to /10.40.1.103; aborting further deliveries INFO [HintedHandoff:1] 2013-04-12 17:05:49,411 HintedHandOffManager.java (line 390) Finished hinted handoff of 0 rows to endpoint /10.40.1.103 Why we getting this message and how I prevent repair from this error. Regards, Adeel Akbar
Re: CorruptedBlockException
Aaron,
It seems that we are in the same situation as Nury, we are storing a lot of
files of ~5MB in a CF.
This happens in a test cluster, with one node using cassandra 1.1.5, we
have commitlog in a different partition than the data directory. Normally
our tests use nearly 13 GB in data, but when the exception on compaction
appears our disk space ramp up to:
# df -h
FilesystemSize Used Avail Use% Mounted on
/dev/sda1 440G 330G 89G 79% /
tmpfs 7.9G 0 7.9G 0% /lib/init/rw
udev 7.9G 160K 7.9G 1% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
/dev/sdb1 459G 257G 179G 59% /cassandra
# cd /cassandra/data/Repository/
# ls Files/*tmp* | wc -l
1671
# du -ch Files | tail -1
257Gtotal
# du -ch Files/*tmp* | tail -1
34G total
We are using cassandra 1.1.5 with one node, our schema for that keyspace is:
[default@unknown] use Repository;
Authenticated to keyspace: Repository
[default@Repository] show schema;
create keyspace Repository
with placement_strategy = 'NetworkTopologyStrategy'
and strategy_options = {datacenter1 : 1}
and durable_writes = true;
use Repository;
create column family Files
with column_type = 'Standard'
and comparator = 'UTF8Type'
and default_validation_class = 'BytesType'
and key_validation_class = 'BytesType'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy =
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'
and caching = 'KEYS_ONLY'
and compaction_strategy_options = {'sstable_size_in_mb' : '120'}
and compression_options = {'sstable_compression' :
'org.apache.cassandra.io.compress.SnappyCompressor'};
In our logs:
ERROR [CompactionExecutor:1831] 2013-04-11 09:12:41,725
AbstractCassandraDaemon.java (line 135) Exception in thread
Thread[CompactionExecutor:1831,1,main]
java.io.IOError: org.apache.cassandra.io.compress.CorruptedBlockException:
(/cassandra/data/Repository/Files/Repository-Files-he-4533-Data.db):
corruption detected, chunk at 43325354 of length 65545.
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116)
at
org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:99)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:176)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:83)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:68)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173)
at
org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
at
org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
On Thu, Jul 5, 2012 at 7:42 PM, aaron morton aa...@thelastpickle.comwrote:
But I don't understand, how was all the available space taken away.
Take a look on disk at /var/lib/cassandra/data/your_keyspace and
/var/lib/cassandra/commitlog to see what is taking up a lot of space.
Cassandra stores the column names as well as the values, so that can take
up some space.
it says that while compaction a CorruptedBlockException has occured.
Are you able to reproduce this error ?
Thanks
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 6/07/2012, at 12:04 AM, Nury Redjepow wrote:
Hello to all,
I have cassandra instance I'm trying to use to store millions of file
with
invalid counter shard detected
Hi guys! We are getting the following message in our logs ERROR [CompactionExecutor:535] 2012-10-31 12:14:14,254 CounterContext.java (line 381) invalid counter shard detected; (ea9feac0-ec3b-11e1--fea7847157bf, 1, 60) and (ea9feac0-ec3b-11e1--fea7847157bf, 1, -60) differ only in count; will pick highest to self-heal; this indicates a bug or corruption generated a bad counter shard Googling about it we found the issue CASSANDRA-4417 so we know this is an open bug. We have verified that the value of those counters is corrupted, we have two nodes and if you request data with CL ONE you get different values depending on what node used. But with CL ALL works ok. There is a way to fix the corrupted counters? thanks
[howto measure disk usage]
cassandra-people,
I'm trying to measure disk usage by cassandra after inserting some columns
in order to plan disk sizes and configurations for future deploys.
My approach is very straightforward:
clean_data (stop_cassandra rm -rf
/var/lib/cassandra/{dara,commitlog,saved_caches}/*)
perform_inserts
measure_disk_usage (nodetool -flush du -ch /var/lib/cassandra)
There are two types of inserts:
- In a simple column with key, name and value a random string of size 100
- In a super-column with key, super-column-name, name and value a random
string of size 100
But surprisingly when I'm inserting 100 million columns on a simple column
it uses more disk than the same amount in a super-column. How can that be
possible?
For simple column 41984 MB and for super-column 29696, the difference is
more than noticeable!
Somebody told me yesterday that super-columns don't have a per-column
timestamp, but... it in my case, even if every data was in the same
super-column-key it will not explain the difference!
ps: sorry, English is not my first language
results.eps
Description: PostScript document
