Re: Does Memtable resides in Heap?
How about the bloom filter and index samples, are they part of off-heap? Starting from C* 1.2 bloom filters are stored off-heap. Index samples are stored on heap. M.
Configuring 2 node Cassandra cluster
Hello Guys, I am trying to setup a 2 node Cassandra clustesr. My parameters are: *Node1(ip1):* initial_token: 0 rpc_address: ip1 listen_address: ip1 seeds: “ip1 *Node2(ip2):* initial_token: 85070591730234615865843651857942052864 rpc_address: ip1 listen_address: ip1 seeds: “ip1 and in Node2 *auto_bootstrap: true;* * * Then when i try to run:* **/usr/lib/apache-cassandra/bin/nodetool -host localhost ring* * * I get this error: *Failed to connect to 'localhost:7199': Connection refused ; * on both nodes. Can you help here please. Also when I create a key space and a column family in Node1, how can I access this data from Node2 ? -- Thanks Regards Venkata Sai Ganji Graduate Student Dept of Computer Science Montana State University - Bzn
Re: Configuring 2 node Cassandra cluster
Could we have the logs after starting each node ? 2013/4/16 Sai Kumar Ganji saikumarganj...@gmail.com Hello Guys, I am trying to setup a 2 node Cassandra clustesr. My parameters are: *Node1(ip1):* initial_token: 0 rpc_address: ip1 listen_address: ip1 seeds: “ip1 *Node2(ip2):* initial_token: 85070591730234615865843651857942052864 rpc_address: ip1 listen_address: ip1 seeds: “ip1 and in Node2 *auto_bootstrap: true;* * * Then when i try to run:* **/usr/lib/apache-cassandra/bin/nodetool -host localhost ring* * * I get this error: *Failed to connect to 'localhost:7199': Connection refused ; * on both nodes. Can you help here please. Also when I create a key space and a column family in Node1, how can I access this data from Node2 ? -- Thanks Regards Venkata Sai Ganji Graduate Student Dept of Computer Science Montana State University - Bzn
Re: Reduce Cassandra GC
How do you calculate the heap / data size ratio? Is this a linear ratio? Each node has slightly more than 12 GB right now though. 2013/4/16 Viktor Jevdokimov viktor.jevdoki...@adform.com For a 40GB of data 1GB of heap is too low. ** ** Best regards / Pagarbiai *Viktor Jevdokimov* Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsider http://twitter.com/#!/adforminsider Take a ride with Adform's Rich Media Suitehttp://vimeo.com/adform/richmedia [image: Adform News] http://www.adform.com [image: Adform awarded the Best Employer 2012] http://www.adform.com/site/blog/adform/adform-takes-top-spot-in-best-employer-survey/ Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. *From:* Joel Samuelsson [mailto:samuelsson.j...@gmail.com] *Sent:* Tuesday, April 16, 2013 10:47 *To:* user@cassandra.apache.org *Subject:* Reduce Cassandra GC ** ** Hi, ** ** We have a small production cluster with two nodes. The load on the nodes is very small, around 20 reads / sec and about the same for writes. There are around 2.5 million keys in the cluster and a RF of 2. ** ** About 2.4 million of the rows are skinny (6 columns) and around 3kb in size (each). Currently, scripts are running, accessing all of the keys in timeorder to do some calculations. ** ** While running the scripts, the nodes go down and then come back up 6-7 minutes later. This seems to be due to GC. I get lines like this in the log: INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java (line 122) GC for ParNew: 338798 ms for 1 collections, 592212416 used; max is 1046937600 ** ** However, the heap is not full. The heap usage has a jagged pattern going from 60% up to 70% during 5 minutes and then back down to 60% the next 5 minutes and so on. I get no Heap is X full... messages. Every once in a while at one of these peaks, I get these stop-the-world GC for 6-7 minutes. Why does GC take up so much time even though the heap isn't full? ** ** I am aware that my access patterns make key caching very unlikely to be high. And indeed, my average key cache hit ratio during the run of the scripts is around 0.5%. I tried disabling key caching on the accessed column family (UPDATE COLUMN FAMILY cf WITH caching=none;) through the cassandra-cli but I get the same behaviour. Is the turning key cache off effective immediately? ** ** Stop-the-world GC is fine if it happens for a few seconds but having them for several minutes doesn't work. Any other suggestions to remove them?*** * ** ** Best regards, Joel Samuelsson signature-logo402b.pngsignature-best-employer-logo72cd.png
MySQL Cluster performing faster than Cassandra cluster on single table
Hi, I was wondering if anybody here had any insight into this. I was running some tests on cassandra and mysql performance, with a two node and three node cassandra cluster, and a five node mysql cluster (mgmt, 2 x api, 2 x data). On the cassandra 2 node cluster vs mysql cluster, I was getting a couple of strange results. For example, on updating a single table in MySQL, with the equivalent super column in Cassandra, I was getting results of 0.231 ms for MySQL and 1.248ms for Cassandra to perform the update 1000 times. Could anybody help explain why this is the case? Thanks, Hannah
Re: MySQL Cluster performing faster than Cassandra cluster on single table
Hi Hannah, mysql-cluster is a in-memory database. In-memory is fast. But I dont think you ever be able to store hundreds of Gigabytes of data on a node, which is something you can do with Cassandra. If your dataset is small, then maybe NDB is the better choice for you. I myself will not even touch it with stick any more, I hate it with a passion. But this might depend on the use-case :-) regards, Christian On Tue, Apr 16, 2013 at 12:56 PM, jrdn hannah j...@jrdnhannah.co.uk wrote: Hi, I was wondering if anybody here had any insight into this. I was running some tests on cassandra and mysql performance, with a two node and three node cassandra cluster, and a five node mysql cluster (mgmt, 2 x api, 2 x data). On the cassandra 2 node cluster vs mysql cluster, I was getting a couple of strange results. For example, on updating a single table in MySQL, with the equivalent super column in Cassandra, I was getting results of 0.231 ms for MySQL and 1.248ms for Cassandra to perform the update 1000 times. Could anybody help explain why this is the case? Thanks, Hannah
Re: Lost data after expanding cluster c* 1.2.3-1
Thanks aaron, I feel that rebuilding indexes went well, but the result of my query (SELECT * FROM userdata WHERE login='kais';) is still emty. INFO [Creating index: userdata.userdata_login_idx] 2013-03-30 01:16:33,110 SecondaryIndex.java (line 175) Submitting index build of userdata.userdata_login_idx INFO [Creating index: userdata.userdata_login_idx] 2013-03-30 01:34:11,667 SecondaryIndex.java (line 202) Index build of userdata.userdata_login_idx complete Thanks, 2013/4/9 aaron morton aa...@thelastpickle.com Look in the logs for messages from the SecondaryIndexManager starts with Submitting index build of end with Index build of Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 7/04/2013, at 12:55 AM, Kais Ahmed k...@neteck-fr.com wrote: hi aaron, nodetool compactionstats on all nodes return 1 pending task : ubuntu@app:~$ nodetool compactionstats host pending tasks: 1 Active compaction remaining time :n/a The command nodetool rebuild_index was launched several days ago. 2013/4/5 aaron morton aa...@thelastpickle.com but nothing's happening, how can i monitor the progress? and how can i know when it's finished? check nodetool compacitonstats Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 4/04/2013, at 2:51 PM, Kais Ahmed k...@neteck-fr.com wrote: Hi aaron, I ran the command nodetool rebuild_index host keyspace cf on all the nodes, in the log i see : INFO [RMI TCP Connection(5422)-10.34.139.xxx] 2013-04-04 08:31:53,641 ColumnFamilyStore.java (line 558) User Requested secondary index re-build for ... but nothing's happening, how can i monitor the progress? and how can i know when it's finished? Thanks, 2013/4/2 aaron morton aa...@thelastpickle.com The problem come from that i don't put auto_boostrap to true for the new nodes, not in this documentation ( http://www.datastax.com/docs/1.2/install/expand_ami) auto_bootstrap defaults to True if not specified in the yaml. can i do that at any time, or when the cluster are not loaded Not sure what the question is. Both those operations are online operations you can do while the node is processing requests. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 1/04/2013, at 9:26 PM, Kais Ahmed k...@neteck-fr.com wrote: At this moment the errors started, we see that members and other data are gone, at this moment the nodetool status return (in red color the 3 new nodes) What errors? The errors was in my side in the application, not cassandra errors I put for each of them seeds = A ip, and start each with two minutes intervals. When I'm making changes I tend to change a single node first, confirm everything is OK and then do a bulk change. Thank you for that advice. I'm not sure what or why it went wrong, but that should get you to a stable place. If you have any problems keep an eye on the logs for errors or warnings. The problem come from that i don't put auto_boostrap to true for the new nodes, not in this documentation ( http://www.datastax.com/docs/1.2/install/expand_ami) if you are using secondary indexes use nodetool rebuild_index to rebuild those. can i do that at any time, or when the cluster are not loaded Thanks aaron, 2013/4/1 aaron morton aa...@thelastpickle.com Please do not rely on colour in your emails, the best way to get your emails accepted by the Apache mail servers is to use plain text. At this moment the errors started, we see that members and other data are gone, at this moment the nodetool status return (in red color the 3 new nodes) What errors? I put for each of them seeds = A ip, and start each with two minutes intervals. When I'm making changes I tend to change a single node first, confirm everything is OK and then do a bulk change. Now the cluster seem to work normally, but i can use the secondary for the moment, the queryanswer are random run nodetool repair -pr on each node, let it finish before starting the next one. if you are using secondary indexes use nodetool rebuild_index to rebuild those. Add one node new node to the cluster and confirm everything is ok, then add the remaining ones. I'm not sure what or why it went wrong, but that should get you to a stable place. If you have any problems keep an eye on the logs for errors or warnings. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 31/03/2013, at 10:01 PM, Kais Ahmed k...@neteck-fr.com wrote: Hi aaron, Thanks for reply, i will try to explain what append exactly I had 4 C* called [A,B,C,D] cluster (1.2.3-1 version) start with ec2 ami
Re: Does Memtable resides in Heap?
Thanks Edward.! On Fri, Apr 12, 2013 at 9:46 AM, Edward Capriolo edlinuxg...@gmail.comwrote: This issue describes the design of the arena allocation of memtabes. https://issues.apache.org/jira/browse/CASSANDRA-2252 On Fri, Apr 12, 2013 at 1:35 AM, Viktor Jevdokimov viktor.jevdoki...@adform.com wrote: Memtables resides in heap, write rate impacts GC, more writes - more frequent and longer ParNew GC pauses. From: Jay Svc [mailto:jaytechg...@gmail.com] Sent: Friday, April 12, 2013 01:03 To: user@cassandra.apache.org Subject: Does Memtable resides in Heap? Hi Team, I have got this 8GB of RAM out of that 4GB allocated to Java Heap. My question is the size of Memtable does it contribute to heap size? or they are part of off-heap? Does bigger Memtable would have impact on GC and overall memory management? I am using DSE 3.0 / Cassandra 1.1.9. Thanks, Jay Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063 Fax: +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
Re: Configuring 2 node Cassandra cluster
*Node2(ip2):* initial_token: 85070591730234615865843651857942052864 rpc_address: ip*1* Should be *ip2* listen_address: ip*1* Should be *ip2* seeds: “ip1 On Tue, Apr 16, 2013 at 5:43 PM, Alain RODRIGUEZ arodr...@gmail.com wrote: Could we have the logs after starting each node ? 2013/4/16 Sai Kumar Ganji saikumarganj...@gmail.com Hello Guys, I am trying to setup a 2 node Cassandra clustesr. My parameters are: *Node1(ip1):* initial_token: 0 rpc_address: ip1 listen_address: ip1 seeds: “ip1 *Node2(ip2):* initial_token: 85070591730234615865843651857942052864 rpc_address: ip1 listen_address: ip1 seeds: “ip1 and in Node2 *auto_bootstrap: true;* * * Then when i try to run:* **/usr/lib/apache-cassandra/bin/nodetool -host localhost ring* * * I get this error: *Failed to connect to 'localhost:7199': Connection refused ; * on both nodes. Can you help here please. Also when I create a key space and a column family in Node1, how can I access this data from Node2 ? -- Thanks Regards Venkata Sai Ganji Graduate Student Dept of Computer Science Montana State University - Bzn
Re: Configuring 2 node Cassandra cluster
If you are using a two node cassandra cluster locally use ccm, it builds all the configuration files for you. https://github.com/pcmanus/ccm On Tue, Apr 16, 2013 at 11:06 AM, Alicia Leong lccali...@gmail.com wrote: *Node2(ip2):* initial_token: 85070591730234615865843651857942052864 rpc_address: ip*1* Should be *ip2* listen_address: ip*1* Should be *ip2* seeds: “ip1 On Tue, Apr 16, 2013 at 5:43 PM, Alain RODRIGUEZ arodr...@gmail.comwrote: Could we have the logs after starting each node ? 2013/4/16 Sai Kumar Ganji saikumarganj...@gmail.com Hello Guys, I am trying to setup a 2 node Cassandra clustesr. My parameters are: *Node1(ip1):* initial_token: 0 rpc_address: ip1 listen_address: ip1 seeds: “ip1 *Node2(ip2):* initial_token: 85070591730234615865843651857942052864 rpc_address: ip1 listen_address: ip1 seeds: “ip1 and in Node2 *auto_bootstrap: true;* * * Then when i try to run:* **/usr/lib/apache-cassandra/bin/nodetool -host localhost ring* * * I get this error: *Failed to connect to 'localhost:7199': Connection refused ; * on both nodes. Can you help here please. Also when I create a key space and a column family in Node1, how can I access this data from Node2 ? -- Thanks Regards Venkata Sai Ganji Graduate Student Dept of Computer Science Montana State University - Bzn
Re: MySQL Cluster performing faster than Cassandra cluster on single table
Ah, I see, that makes sense. Have you got a source for the storing of hundreds of gigabytes? And does Cassandra not store anything in memory? Yeah, my dataset is small at the moment - perhaps I should have chosen something larger for the work I'm doing (University dissertation), however, it is far too late to change now! Thanks for the help, Hannah On 16 Apr 2013, at 12:16, horschi hors...@gmail.com wrote: Hi Hannah, mysql-cluster is a in-memory database. In-memory is fast. But I dont think you ever be able to store hundreds of Gigabytes of data on a node, which is something you can do with Cassandra. If your dataset is small, then maybe NDB is the better choice for you. I myself will not even touch it with stick any more, I hate it with a passion. But this might depend on the use-case :-) regards, Christian On Tue, Apr 16, 2013 at 12:56 PM, jrdn hannah j...@jrdnhannah.co.uk wrote: Hi, I was wondering if anybody here had any insight into this. I was running some tests on cassandra and mysql performance, with a two node and three node cassandra cluster, and a five node mysql cluster (mgmt, 2 x api, 2 x data). On the cassandra 2 node cluster vs mysql cluster, I was getting a couple of strange results. For example, on updating a single table in MySQL, with the equivalent super column in Cassandra, I was getting results of 0.231 ms for MySQL and 1.248ms for Cassandra to perform the update 1000 times. Could anybody help explain why this is the case? Thanks, Hannah
Re: MySQL Cluster performing faster than Cassandra cluster on single table
Yeah, I remember reading about that, but the schema had already been set and submitted. I will have to take that into consideration when discussing the results. Thanks, Hannah On 16 Apr 2013, at 17:42, Robert Coli rc...@eventbrite.com wrote: On Tue, Apr 16, 2013 at 3:56 AM, jrdn hannah j...@jrdnhannah.co.uk wrote: For example, on updating a single table in MySQL, with the equivalent super column in Cassandra, I was getting results of 0.231 ms for MySQL and 1.248ms for Cassandra to perform the update 1000 times. You probably do not want to use a Supercolumn. They have been in a state of quasi-supported semi-deprecation for years now... =Rob
Re: MySQL Cluster performing faster than Cassandra cluster on single table
On Tue, Apr 16, 2013 at 3:56 AM, jrdn hannah j...@jrdnhannah.co.uk wrote: For example, on updating a single table in MySQL, with the equivalent super column in Cassandra, I was getting results of 0.231 ms for MySQL and 1.248ms for Cassandra to perform the update 1000 times. You probably do not want to use a Supercolumn. They have been in a state of quasi-supported semi-deprecation for years now... =Rob
Re: MySQL Cluster performing faster than Cassandra cluster on single table
Ah, I see, that makes sense. Have you got a source for the storing of hundreds of gigabytes? And does Cassandra not store anything in memory? It stores bloom filters and index-samples in memory. But they are much smaller than the actual data and they can be configured. Yeah, my dataset is small at the moment - perhaps I should have chosen something larger for the work I'm doing (University dissertation), however, it is far too late to change now! On paper mysql-cluster looks great. But in daily use its not as nice as Cassandra (where you have machines dying, networks splitting, etc.). cheers, Christian
RE: Reduce Cassandra GC
How one could provide any help without any knowledge about your cluster, node and environment settings? 40GB was calculated from 2 nodes with RF=2 (each has 100% data range), 2.4-2.5M rows * 6 cols * 3kB as a minimum without compression and any overhead (sstable, bloom filters and indexes). With ParNew GC time such as yours even if it is a swapping issue I could say only that heap size is too small. Check Heap, New Heap sizes, memtable and cache sizes. Are you on Linux? Is JNA installed and used? What is total amount of RAM? Just for a DEV environment we use 3 virtual machines with 4GB RAM and use 2GB heap without any GC issue with amount of data from 0 to 16GB compressed on each node. Memtable space sized to 100MB, New Heap 400MB. Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.commailto:viktor.jevdoki...@adform.com Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsiderhttp://twitter.com/#!/adforminsider Take a ride with Adform's Rich Media Suitehttp://vimeo.com/adform/richmedia [Adform News] http://www.adform.com [Adform awarded the Best Employer 2012] http://www.adform.com/site/blog/adform/adform-takes-top-spot-in-best-employer-survey/ Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Joel Samuelsson [mailto:samuelsson.j...@gmail.com] Sent: Tuesday, April 16, 2013 12:52 To: user@cassandra.apache.org Subject: Re: Reduce Cassandra GC How do you calculate the heap / data size ratio? Is this a linear ratio? Each node has slightly more than 12 GB right now though. 2013/4/16 Viktor Jevdokimov viktor.jevdoki...@adform.commailto:viktor.jevdoki...@adform.com For a 40GB of data 1GB of heap is too low. Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.commailto:viktor.jevdoki...@adform.com Phone: +370 5 212 3063tel:%2B370%205%20212%203063, Fax +370 5 261 0453tel:%2B370%205%20261%200453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsiderhttp://twitter.com/#!/adforminsider Take a ride with Adform's Rich Media Suitehttp://vimeo.com/adform/richmedia [Adform News]http://www.adform.com [Adform awarded the Best Employer 2012]http://www.adform.com/site/blog/adform/adform-takes-top-spot-in-best-employer-survey/ Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Joel Samuelsson [mailto:samuelsson.j...@gmail.commailto:samuelsson.j...@gmail.com] Sent: Tuesday, April 16, 2013 10:47 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Reduce Cassandra GC Hi, We have a small production cluster with two nodes. The load on the nodes is very small, around 20 reads / sec and about the same for writes. There are around 2.5 million keys in the cluster and a RF of 2. About 2.4 million of the rows are skinny (6 columns) and around 3kb in size (each). Currently, scripts are running, accessing all of the keys in timeorder to do some calculations. While running the scripts, the nodes go down and then come back up 6-7 minutes later. This seems to be due to GC. I get lines like this in the log: INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java (line 122) GC for ParNew: 338798 ms for 1 collections, 592212416 used; max is 1046937600 However, the heap is not full. The heap usage has a jagged pattern going from 60% up to 70% during 5 minutes and then back down to 60% the next 5 minutes and so on. I get no Heap is X full... messages. Every once in a while at one of these peaks, I get these stop-the-world GC for 6-7 minutes. Why does GC take up so much time even though the heap isn't full? I am aware that my access patterns make key caching very unlikely to be high. And indeed, my average key cache hit ratio during the run of the scripts is around 0.5%. I tried disabling key caching on the accessed column family (UPDATE COLUMN FAMILY cf WITH caching=none;) through the cassandra-cli but I get the same behaviour. Is the turning key cache off effective immediately? Stop-the-world GC is fine if it
Re: Any experience of 20 node mini-itx cassandra cluster
I already have thanks. I'll do the tests with the hardware arrives. Thanks Jabbar Azam On 16 April 2013 22:27, aaron morton aa...@thelastpickle.com wrote: Can't we use LCS? Do some reading and some tests… http://www.datastax.com/dev/blog/leveled-compaction-in-apache-cassandra http://www.datastax.com/dev/blog/when-to-use-leveled-compaction Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 15/04/2013, at 10:44 PM, Jabbar Azam aja...@gmail.com wrote: I know the SSD's are a bit small but they should be enough for our application. Out test data is 1.6 TB(including replication of rf=3). Can't we use LCS? This will give us more space at the expensive of more I/O but SSD's have loads of I/Os. Thanks Jabbar Azam On 14 April 2013 20:20, Jabbar Azam aja...@gmail.com wrote: Thanks Aaron. Thanks Jabbar Azam On 14 April 2013 19:39, aaron morton aa...@thelastpickle.com wrote: That's better. The SSD size is a bit small, and be warned that you will want to leave 50Gb to 100GB free to allow room for compaction (using the default size tiered). On the ram side you will want to run about 4GB (assuming cass 1.2) for the JVM the rest can be off heap Cassandra structures. This may not leave too much free space for the os page cache, but SSD may help there. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 13/04/2013, at 4:47 PM, Jabbar Azam aja...@gmail.com wrote: What about using quad core athlon x4 740 3.2 GHz with 8gb of ram and 256gb ssds? I know it will depend on our workload but will be better than a dual core CPU. I think Jabbar Azam On 13 Apr 2013 01:05, Edward Capriolo edlinuxg...@gmail.com wrote: Duel core not the greatest you might run into GC issues before you run out of IO from your ssd devices. Also cassandra has other concurrency settings that are tuned roughly around the number of processors/cores. It is not uncommon to see 4-6 cores of cpu (600 % in top dealing with young gen garbage managing lots of sockets whatever. On Fri, Apr 12, 2013 at 12:02 PM, Jabbar Azam aja...@gmail.com wrote: That's my guess. My colleague is still looking at CPU's so I'm hoping he can get quad core CPU's for the servers. Thanks Jabbar Azam On 12 April 2013 16:48, Colin Blower cblo...@barracuda.com wrote: If you have not seen it already, checkout the Netflix blog post on their performance testing of AWS SSD instances. http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html My guess, based on very little experience, is that you will be CPU bound. On 04/12/2013 03:05 AM, Jabbar Azam wrote: Hello, I'm going to be building a 20 node cassandra cluster in one datacentre. The spec of the servers will roughly be dual core Celeron CPU, 256 GB SSD, 16GB RAM and two nics. Has anybody done any performance testing with this setup or have any gotcha's I should be aware of wrt to the hardware? I do realise the CPU is fairly low computational power but I'm going to assume the system is going to be IO bound hence the RAM and SSD's. Thanks Jabbar Azam -- *Colin Blower* *Software Engineer* Barracuda Networks Inc. +1 408-342-5576 (o)
Re: Extracting data from SSTable files with MapReduce
I did try to upgrade to 1.2 but it did not work out. Maybe to many versions in between. Newer versions should be able to read older file formats. What was the error? Why would later formats make this easier you think? it will be easier to write against the current code base and you find it easier to get help. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 16/04/2013, at 5:37 AM, Jasper K. jasper.knu...@incentro.com wrote: Hi Aaron, I did try to upgrade to 1.2 but it did not work out. Maybe to many versions in between. Why would later formats make this easier you think? Jasper 2013/4/14 aaron morton aa...@thelastpickle.com The SSTable files are in the -f- format from 0.8.10. If you can upgrade to the latest version it will make things easier. Start a node and use nodetool upgradesstables. The org.apache.cassandra.tools.SSTableExport class provides a blue print for reading rows from disk. hope that helps. - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 13/04/2013, at 7:58 PM, Jasper K. jasper.knu...@incentro.com wrote: Hi, Does anyone have any experience with running a MapReduce directly against a CF's SSTable files? I have a use case where this seems to be an option. I want to export all data from a CF to a flat file format for statistical analysis. Some factors that make it (more) doable in my case: -The Cassandra instance is not 'on-line' (no writes- no reads) -The .db files were exported from another instance. I got them all in one place now The SSTable files are in the -f- format from 0.8.10. Looking at this : http://wiki.apache.org/cassandra/ArchitectureSSTable it should be possible to write a Hadoop RecordReader for Cassandra rowkeys. But maybe I am not fully aware of what I am up to. -- Jasper --
Re: StatusLogger format?
99% sure it's in bytes. +1 to your confidence level. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 16/04/2013, at 6:14 AM, William Oberman ober...@civicscience.com wrote: 99% sure it's in bytes. On Mon, Apr 15, 2013 at 11:25 AM, William Oberman ober...@civicscience.com wrote: Mainly the: ColumnFamilyMemtable ops,data section. Is data in bytes/kb/mb/etc? Example line: StatusLogger.java (line 116) civicscience.sessions4963,1799916 Thanks!
Re: Thrift message length exceeded
Can you confirm the you are using the same thrift version that ships 1.2.3 ? Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 16/04/2013, at 10:17 AM, Lanny Ripple la...@spotright.com wrote: A bump to say I found this http://stackoverflow.com/questions/15487540/pig-cassandra-message-length-exceeded so others are seeing similar behavior. From what I can see of org.apache.cassandra.hadoop nothing has changed since 1.1.5 when we didn't see such things but sure looks like there's a bug that's slipped in (or been uncovered) somewhere. I'll try to narrow down to a dataset and code that can reproduce. On Apr 10, 2013, at 6:29 PM, Lanny Ripple la...@spotright.com wrote: We are using Astyanax in production but I cut back to just Hadoop and Cassandra to confirm it's a Cassandra (or our use of Cassandra) problem. We do have some extremely large rows but we went from everything working with 1.1.5 to almost everything carping with 1.2.3. Something has changed. Perhaps we were doing something wrong earlier that 1.2.3 exposed but surprises are never welcome in production. On Apr 10, 2013, at 8:10 AM, moshe.kr...@barclays.com wrote: I also saw this when upgrading from C* 1.0 to 1.2.2, and from hector 0.6 to 0.8 Turns out the Thrift message really was too long. The mystery to me: Why no complaints in previous versions? Were some checks added in Thrift or Hector? -Original Message- From: Lanny Ripple [mailto:la...@spotright.com] Sent: Tuesday, April 09, 2013 6:17 PM To: user@cassandra.apache.org Subject: Thrift message length exceeded Hello, We have recently upgraded to Cass 1.2.3 from Cass 1.1.5. We ran sstableupgrades and got the ring on its feet and we are now seeing a new issue. When we run MapReduce jobs against practically any table we find the following errors: 2013-04-09 09:58:47,746 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2013-04-09 09:58:47,899 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId= 2013-04-09 09:58:48,021 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0 2013-04-09 09:58:48,024 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4a48edb5 2013-04-09 09:58:50,475 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2013-04-09 09:58:50,477 WARN org.apache.hadoop.mapred.Child: Error running child java.lang.RuntimeException: org.apache.thrift.TException: Message length exceeded: 106 at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:384) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:390) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:313) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFamilyRecordReader.java:103) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:444) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:460) at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278) at org.apache.hadoop.mapred.Child.main(Child.java:260) Caused by: org.apache.thrift.TException: Message length exceeded: 106 at org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393) at org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363) at org.apache.cassandra.thrift.Column.read(Column.java:528) at org.apache.cassandra.thrift.ColumnOrSuperColumn.read(ColumnOrSuperColumn.java:507) at org.apache.cassandra.thrift.KeySlice.read(KeySlice.java:408) at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12905) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at
Re: C* consumes all RAM
You are probably seeing this http://wiki.apache.org/cassandra/FAQ#mmap Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 16/04/2013, at 8:43 PM, Mikhail Mazursky ash...@gmail.com wrote: More details: USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND 219 3801 0.7 92.7 6561116 3567016 ? SLl Mar11 372:44 /usr/java/latest/bin/java Linux XXX@YYY 3.2.30-49.59.amzn1.x86_64 #1 SMP Wed Oct 3 19:54:33 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux total used free sharedbuffers cached Mem: 3754 3695 59 0 37 75 -/+ buffers/cache: 3582171 Swap: 2055165 1890 2013/4/16 Mikhail Mazursky ash...@gmail.com Hello. C* have been running without any problem for some weeks but now it started to consume all available ram. The cluster have very little data in it. There are no errors in logs, CPU is not loaded at all, jstack shows no deadlocks, there are 83 threads. Read/write latency is 1-4ms. The question is: how to find out where is the RAM used? # nodetool ring keyspace name Address DC RackStatus State Load Effective-Ownership Token 114967934986393518082067045023985443411 10.0.1.220 datacenter1 rack1 Up Normal 889.71 KB 100.00% 0 10.0.1.222 datacenter1 rack1 Up Normal 918.92 KB 100.00% 56662868904138943096229641828625627278 10.0.1.221 datacenter1 rack1 Up Normal 1.04 MB 100.00% 114967934986393518082067045023985443411 We use AWS m1.medium - 3.75 RAM, 1 Core java version 1.7.0_10 Java(TM) SE Runtime Environment (build 1.7.0_10-b18) Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode) C* is started with the following command /usr/java/latest/bin/java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1G -Xmx1G -Xmn512M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC Thanks in advance. Mikhail.
Re: Lost data after expanding cluster c* 1.2.3-1
Sorry can you repost the details of that issue including the CL you are using. Aaron - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 17/04/2013, at 12:57 AM, Kais Ahmed k...@neteck-fr.com wrote: Thanks aaron, I feel that rebuilding indexes went well, but the result of my query (SELECT * FROM userdata WHERE login='kais';) is still emty. INFO [Creating index: userdata.userdata_login_idx] 2013-03-30 01:16:33,110 SecondaryIndex.java (line 175) Submitting index build of userdata.userdata_login_idx INFO [Creating index: userdata.userdata_login_idx] 2013-03-30 01:34:11,667 SecondaryIndex.java (line 202) Index build of userdata.userdata_login_idx complete Thanks, 2013/4/9 aaron morton aa...@thelastpickle.com Look in the logs for messages from the SecondaryIndexManager starts with Submitting index build of end with Index build of Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 7/04/2013, at 12:55 AM, Kais Ahmed k...@neteck-fr.com wrote: hi aaron, nodetool compactionstats on all nodes return 1 pending task : ubuntu@app:~$ nodetool compactionstats host pending tasks: 1 Active compaction remaining time :n/a The command nodetool rebuild_index was launched several days ago. 2013/4/5 aaron morton aa...@thelastpickle.com but nothing's happening, how can i monitor the progress? and how can i know when it's finished? check nodetool compacitonstats Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 4/04/2013, at 2:51 PM, Kais Ahmed k...@neteck-fr.com wrote: Hi aaron, I ran the command nodetool rebuild_index host keyspace cf on all the nodes, in the log i see : INFO [RMI TCP Connection(5422)-10.34.139.xxx] 2013-04-04 08:31:53,641 ColumnFamilyStore.java (line 558) User Requested secondary index re-build for ... but nothing's happening, how can i monitor the progress? and how can i know when it's finished? Thanks, 2013/4/2 aaron morton aa...@thelastpickle.com The problem come from that i don't put auto_boostrap to true for the new nodes, not in this documentation (http://www.datastax.com/docs/1.2/install/expand_ami) auto_bootstrap defaults to True if not specified in the yaml. can i do that at any time, or when the cluster are not loaded Not sure what the question is. Both those operations are online operations you can do while the node is processing requests. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 1/04/2013, at 9:26 PM, Kais Ahmed k...@neteck-fr.com wrote: At this moment the errors started, we see that members and other data are gone, at this moment the nodetool status return (in red color the 3 new nodes) What errors? The errors was in my side in the application, not cassandra errors I put for each of them seeds = A ip, and start each with two minutes intervals. When I'm making changes I tend to change a single node first, confirm everything is OK and then do a bulk change. Thank you for that advice. I'm not sure what or why it went wrong, but that should get you to a stable place. If you have any problems keep an eye on the logs for errors or warnings. The problem come from that i don't put auto_boostrap to true for the new nodes, not in this documentation (http://www.datastax.com/docs/1.2/install/expand_ami) if you are using secondary indexes use nodetool rebuild_index to rebuild those. can i do that at any time, or when the cluster are not loaded Thanks aaron, 2013/4/1 aaron morton aa...@thelastpickle.com Please do not rely on colour in your emails, the best way to get your emails accepted by the Apache mail servers is to use plain text. At this moment the errors started, we see that members and other data are gone, at this moment the nodetool status return (in red color the 3 new nodes) What errors? I put for each of them seeds = A ip, and start each with two minutes intervals. When I'm making changes I tend to change a single node first, confirm everything is OK and then do a bulk change. Now the cluster seem to work normally, but i can use the secondary for the moment, the queryanswer are random run nodetool repair -pr on each node, let it finish before starting the next one. if you are using secondary indexes use nodetool rebuild_index to rebuild those. Add one node new node to the cluster and confirm everything is ok, then add the remaining ones. I'm not sure what or why it went wrong, but that should get you to a stable place. If you have any problems keep an eye on the logs for errors or warnings. Cheers - Aaron Morton Freelance
Re: Does Memtable resides in Heap?
Compression Meta data is also off heap http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 17/04/2013, at 3:02 AM, Jay Svc jaytechg...@gmail.com wrote: Thanks Edward.! On Fri, Apr 12, 2013 at 9:46 AM, Edward Capriolo edlinuxg...@gmail.com wrote: This issue describes the design of the arena allocation of memtabes. https://issues.apache.org/jira/browse/CASSANDRA-2252 On Fri, Apr 12, 2013 at 1:35 AM, Viktor Jevdokimov viktor.jevdoki...@adform.com wrote: Memtables resides in heap, write rate impacts GC, more writes - more frequent and longer ParNew GC pauses. From: Jay Svc [mailto:jaytechg...@gmail.com] Sent: Friday, April 12, 2013 01:03 To: user@cassandra.apache.org Subject: Does Memtable resides in Heap? Hi Team, I have got this 8GB of RAM out of that 4GB allocated to Java Heap. My question is the size of Memtable does it contribute to heap size? or they are part of off-heap? Does bigger Memtable would have impact on GC and overall memory management? I am using DSE 3.0 / Cassandra 1.1.9. Thanks, Jay Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063 Fax: +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
Repair Freeze / Gossip Invisibility / EC2 Public IP configuration
TL;DR; An EC2 Multi-Region Setup's Repair/Gossip Works with 1.1.10 but with 1.2.4, gossip does not see the nodes after restarting all nodes at once, and repair gets stuck. This is a working configuration: Cassandra 1.1.10 Cluster with 12 nodes in us-east-1 and 12 nodes in us-west-2 Using Ec2MultiRegionSnitch and SSL enabled for DC_ONLY and NetworkTopologyStrategy with strategy_options: us-east-1:3;us-west-2:3; C* instances have a security group called 'cluster1' security group 'cluster1' in each region is configured as such Allow TCP: 7199 from cluster1 (JMX) 1024 - 65535 from cluster1 (JMX Random Ports - This supersedes all specific ports, but I have the specific ports just for clarity ) 7100 from cluster1 (Configured Normal Storage) 7103 from cluster1 (Configured SSL Storage) 9160 from cluster1 (Configured Thrift RPC Port) 9160 from client_group foreach node's public IP we also have this rule set to enable cross region comminication: 7103 from public_ip (Open SSL storage) The above is a functioning and happy setup. You run repair, and it finishes successfully. Broken Setup: Upgrade to 1.2.4 without changing any of the above security group settings: Run repair. The repair will get stuck. Thus hanging. Now for each public_ip add a security group rule as such to cluster1 security group: Allow TCP: 7100 from public_ip Run repair. Things will work now. Also after restarting all nodes at the same time, gossip will see everyone again. I was told on https://issues.apache.org/jira/browse/CASSANDRA-5432 that nothing in terms of networking was changed. If nothing in terms of port and networking was changed in 1.2, then why the above is happening? I can constantly reproduce it. Please advice. -Arya
Re: Repair Freeze / Gossip Invisibility / EC2 Public IP configuration
So cassandra does inter node compression. I have not checked but this might be accidentally getting turned on by default. Because the storage port is typically 7000. Not sure why you are allowing 7100. In any case try allowing 7000 or with internode compression off. On Tue, Apr 16, 2013 at 6:42 PM, Arya Goudarzi gouda...@gmail.com wrote: TL;DR; An EC2 Multi-Region Setup's Repair/Gossip Works with 1.1.10 but with 1.2.4, gossip does not see the nodes after restarting all nodes at once, and repair gets stuck. This is a working configuration: Cassandra 1.1.10 Cluster with 12 nodes in us-east-1 and 12 nodes in us-west-2 Using Ec2MultiRegionSnitch and SSL enabled for DC_ONLY and NetworkTopologyStrategy with strategy_options: us-east-1:3;us-west-2:3; C* instances have a security group called 'cluster1' security group 'cluster1' in each region is configured as such Allow TCP: 7199 from cluster1 (JMX) 1024 - 65535 from cluster1 (JMX Random Ports - This supersedes all specific ports, but I have the specific ports just for clarity ) 7100 from cluster1 (Configured Normal Storage) 7103 from cluster1 (Configured SSL Storage) 9160 from cluster1 (Configured Thrift RPC Port) 9160 from client_group foreach node's public IP we also have this rule set to enable cross region comminication: 7103 from public_ip (Open SSL storage) The above is a functioning and happy setup. You run repair, and it finishes successfully. Broken Setup: Upgrade to 1.2.4 without changing any of the above security group settings: Run repair. The repair will get stuck. Thus hanging. Now for each public_ip add a security group rule as such to cluster1 security group: Allow TCP: 7100 from public_ip Run repair. Things will work now. Also after restarting all nodes at the same time, gossip will see everyone again. I was told on https://issues.apache.org/jira/browse/CASSANDRA-5432 that nothing in terms of networking was changed. If nothing in terms of port and networking was changed in 1.2, then why the above is happening? I can constantly reproduce it. Please advice. -Arya
Re: unexplained hinted handoff
On Sun, Apr 14, 2013 at 11:28 AM, aaron morton aa...@thelastpickle.com wrote: If hints are being stored, doesn't that imply DOWN nodes, and why don't I see that in the logs? Hints are stored for two reasons. First if the node is down when the write request starts, second if the node does not reply to the coordinator before rpc_timeout. If you are not seeing dropped write messages it may indicate network issues between the nodes. Very helpful! I increased the timeouts for *_request_timeout settings in cassandra.yaml and no longer see HH in the logs. Still not sure why I saw logs about dropped reads, and not dropped mutations. Anyhow, big timeouts helps in this case. Thanks :) Dane
How to stop Cassandra and then restart it in windows?
Hello, I installed single node cluster in my local dev box which is running Windows 7 and it was working fine. Due to some reason, I need to restart my desktop and then after that whenever I am doing like this on the command prompt, it always gives me the below exception- S:\Apache Cassandra\apache-cassandra-1.2.3\bincassandra -f Starting Cassandra Server Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7199; nested exception is: java.net.BindException: Address already in use: JVM_Bind Meaning port being used somewhere. I have made some changes in *cassandra.yaml *file so I need to shutdown the Cassandra server and then restart it again. Can anybody help me with this? Thanks for the help.
Cassandra Client Recommendation
Hello, I have recently started working with Cassandra Database. Now I am in the process of evaluating which Cassandra client I should go forward with. I am mainly interested in these three- --1) Astyanax client 2--) New Datastax client that uses Binary protocol. --3) Pelops client Can anyone provide some thoughts on this? Some advantages and disadvantages for these three will be great start for me. Keeping in mind, we are running Cassandra 1.2.2 in production environment. Thanks for the help.
Re: MySQL Cluster performing faster than Cassandra cluster on single table
MySQL cluster also has the index in ram. So with lots of rows the ram becomes a limiting factor. That's what my colleague found and hence why were sticking with Cassandra. On 16 Apr 2013 21:05, horschi hors...@gmail.com wrote: Ah, I see, that makes sense. Have you got a source for the storing of hundreds of gigabytes? And does Cassandra not store anything in memory? It stores bloom filters and index-samples in memory. But they are much smaller than the actual data and they can be configured. Yeah, my dataset is small at the moment - perhaps I should have chosen something larger for the work I'm doing (University dissertation), however, it is far too late to change now! On paper mysql-cluster looks great. But in daily use its not as nice as Cassandra (where you have machines dying, networks splitting, etc.). cheers, Christian
Re: C* consumes all RAM
Thank you, Aaron. p.s. we're on 1.1.9 - i forgot to mention that. 2013/4/17 aaron morton aa...@thelastpickle.com You are probably seeing this http://wiki.apache.org/cassandra/FAQ#mmap Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 16/04/2013, at 8:43 PM, Mikhail Mazursky ash...@gmail.com wrote: More details: USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND 219 3801 0.7 92.7 6561116 3567016 ? SLl Mar11 372:44 /usr/java/latest/bin/java Linux XXX@YYY 3.2.30-49.59.amzn1.x86_64 #1 SMP Wed Oct 3 19:54:33 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux total used free sharedbuffers cached Mem: 3754 3695 59 0 37 75 -/+ buffers/cache: 3582171 Swap: 2055165 1890 2013/4/16 Mikhail Mazursky ash...@gmail.com Hello. C* have been running without any problem for some weeks but now it started to consume all available ram. The cluster have very little data in it. There are no errors in logs, CPU is not loaded at all, jstack shows no deadlocks, there are 83 threads. Read/write latency is 1-4ms. The question is: how to find out where is the RAM used? # nodetool ring keyspace name Address DC RackStatus State Load Effective-Ownership Token 114967934986393518082067045023985443411 10.0.1.220 datacenter1 rack1 Up Normal 889.71 KB 100.00% 0 10.0.1.222 datacenter1 rack1 Up Normal 918.92 KB 100.00% 56662868904138943096229641828625627278 10.0.1.221 datacenter1 rack1 Up Normal 1.04 MB 100.00% 114967934986393518082067045023985443411 We use AWS m1.medium - 3.75 RAM, 1 Core java version 1.7.0_10 Java(TM) SE Runtime Environment (build 1.7.0_10-b18) Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode) C* is started with the following command /usr/java/latest/bin/java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1G -Xmx1G -Xmn512M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC Thanks in advance. Mikhail.
Commit Log question
I'm looking into a case where it appears that recycling a commit log segment and flushing the dirty CF's results in 46 CF's being flushed. Out of 47 in the keyspace. All this flush activity blocks writes. Before I dig further I wanted to confirm my understanding. At 10:46 the MeteredFlusher kicks and flushed the CF…. 2013-03-18T10:46:59-07:00 INFO [OptionalTasks:1] 2013-03-18 10:46:59,009 MeteredFlusher.java (line 62) flushing high-traffic column family CFS(Keyspace='KSNAME', ColumnFamily='CFNAME') (estimated 198959043 bytes) 2013-03-18T10:46:59-07:00 INFO [OptionalTasks:1] 2013-03-18 10:46:59,009 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-CFNAME@172178999(158553073/198959043 serialized/live bytes, 470065 ops) 2013-03-18T10:46:59-07:00 INFO [FlushWriter:482] 2013-03-18 10:46:59,010 Memtable.java (line 264) Writing Memtable-CFNAME@172178999(158553073/198959043 serialized/live bytes, 470065 ops) 2013-03-18T10:47:01-07:00 INFO [FlushWriter:482] 2013-03-18 10:47:01,503 Memtable.java (line 305) Completed flushing /mnt/cassandra/data/KSNAME/CFNAME/KSNAME-CFNAME-hf-590-Data.db (29212692 bytes) for commitlog position ReplayPosition(segmentId=1363374084494, position=982002808) I expect that the CF is marked as clean in all CL segments other than the active one (1363374084494), and in the active one if the last write location for the CF is before the current context position (982002808). There is nothing else in the log (other than liveRatio measuring) then a storm of flushing without the MeteredFlusher being involved. I attribute this to the CL growing over capacity and the CF's marked as dirty in the oldest segment being flushed. As part of that the CF from above is flushed. 2013-03-18T10:50:05-07:00 INFO [OptionalTasks:1] 2013-03-18 10:50:05,777 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-CFNAME@1841064520(93240219/117001736 serialized/live bytes, 71513 ops) 2013-03-18T10:50:05-07:00 INFO [FlushWriter:483] 2013-03-18 10:50:05,942 Memtable.java (line 264) Writing Memtable-CFNAME@1841064520(93240219/117001736 serialized/live bytes, 71513 ops) 2013-03-18T10:50:07-07:00 INFO [FlushWriter:483] 2013-03-18 10:50:07,277 Memtable.java (line 305) Completed flushing /mnt/cassandra/data/KSNAME/CFNAME/KSNAME-CFNAME-hf-591-Data.db (12050522 bytes) for commitlog position ReplayPosition(segmentId=1363374084495, position=8542666) Not that the segmentId has only increased by 1, so we have not put a huge amount of traffic through the commit log. My understanding is that after the first flush the CF would be clean in the oldest CL segment and I should not see the CF being flushed a second time. Is that correct? running on 1.1.6 Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com
Re: Added extra column as composite key while creation counter column family
cassandra 1.2.0
Is it a bug in 1.2.0 ?
Thanks
KK
On Wed, Apr 17, 2013 at 2:56 AM, aaron morton aa...@thelastpickle.comwrote:
What version are you using ?
WIth 1.2.4 …
cqlsh:dev CREATE TABLE counters (
... key text,
... value counter,
... PRIMARY KEY (key)
... ) WITH COMPACT STORAGE;
cqlsh:dev describe table counters;
CREATE TABLE counters (
key text PRIMARY KEY,
value counter
) WITH COMPACT STORAGE AND
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'SnappyCompressor'};
Cheers
-
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 15/04/2013, at 3:58 PM, Kuldeep Mishra kuld.cs.mis...@gmail.com
wrote:
Hi,
While I creating counter column family a extra column is being added
what I do ?
Table creation script
CREATE TABLE counters (
key text,
value counter,
PRIMARY KEY (key)
) WITH COMPACT STORAGE
after describing column family I am getting following
CREATE TABLE counters (
key text,
column1 text,
value counter,
PRIMARY KEY (key, column1)
) WITH COMPACT STORAGE AND
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'SnappyCompressor'};
extra column column1 is added
Please help
--
Thanks and Regards
Kuldeep Kumar Mishra
+919540965199
--
Thanks and Regards
Kuldeep Kumar Mishra
+919540965199
