OOMs during high (read?) load in Cassandra 1.2.11
We're getting fairly reproducible OOMs on a 2-node cluster using Cassandra 1.2.11, typically in situations with a heavy read load. A sample of some stack traces is at https://gist.github.com/KlausBrunner/7820902 - they're all failing somewhere down from table.getRow(), though I don't know if that's part of query processing, compaction, or something else. - The main CFs contain some 100k rows, none of them particularly wide. - Heap dumps invariably show a single huge byte array (~1.6 GiB associated with the OOM'ing thread) hogging 80% of the Java heap. The array seems to contain all/many rows of one CF. - We're moderately certain there's no killer query with a huge result set involved here, but we can't see exactly what triggers this. - We've tried to switch to LeveledCompaction, to no avail. - Xms/x is set to some 4 GB. - The logs show the usual signs of panic (flushing memtables) before actually OOMing. It seems that this scenario is often or even always after a compaction, but it's not quite conclusive. I'm somewhat worried that Cassandra would read so much data into a single contiguous byte[] at any point. Could this be related to compaction? Any ideas what we could do about this? Thanks Klaus
Try to configure commitlog_archiving.properties
Hello, I try to configure commitlog_archiving.properties to take advantage of backup and restore at a point of time, but there is no ressources on internet for that. So i need some help. If I understand I have 4 parameters: archive_command= restore_command= restore_directories= restore_point_in_time= Forget for the moment the restore_point_in_time, should i put one command only for the archive_command; my wish is to copy all the commitlogs on another directory: restore_directories=/produits/cassandra/cassandra_data/archived_logs archive_command=/bin/cp - f /produits/cassandra/cassandra_commit/*.log /produits/cassandra/cassandra_ data/archived_logs restore_command=/bin/cp - f /produits/cassandra/cassandra_data/archived_logs/*.log /produits/cassandr a/cassandra_commit But it doesn't work when i restart the node which interest me, it doesn't copy anything. Why, can you give me a good example and the good way to configure archive commit logs ? there's nothing on the net except Datastax Website. Thanks for your answear. Regards, Bonnet Jonathan.
cassandra backup
Hello everyone, I am trying to create backups of my data on AWS. My goal is to store the backups on S3 or glacier, as it's cheap to store this kind of data. So, if I have a cluster with N nodes, I would like to copy data from all N nodes to S3 and be able to restore later. I know Priam does that (we were using it), but I am using the latest cassandra version and we plan to use DSE some time, I am not sure Priam fits this case. I took a look at the docs: http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/operations/../../cassandra/operations/ops_backup_takes_snapshot_t.html And I am trying to understand if it's really needed to take a snapshot to create my backup. Suppose I do a flush and copy the sstables from each node, 1 by one, to s3. Not all at the same time, but one by one. When I try to restore my backup, data from node 1 will be older than data from node 2. Will this cause problems? AFAIK, if I am using a replication factor of 2, for instance, and Cassandra sees data from node X only, it will automatically copy it to other nodes, right? Is there any chance of cassandra nodes become corrupt somehow if I do my backups this way? Best regards, Marcelo Valle.
Re: cassandra backup
Hi Marcelo, Cassandra provides and eventually consistent model for backups. You can do staggered backups of data, with the idea that if you restore a node, and then do a repair, your data will be once again consistent. Cassandra will not automatically copy the data to other nodes (other than via hinted handoff). You should manually run repair after restoring a node. You should take snapshots when doing a backup, as it keeps the data you are backing up relevant to a single point in time, otherwise compaction could add/delete files one you mid-backup, or worse, I imagine attempt to access a SSTable mid-write. Snapshots work by using links, and don't take additional storage to perform. In our process we create the snapshot, perform the backup, and then clear the snapshot. One thing to keep in mind in your S3 cost analysis is that, even though storage is cheap, reads/writes to S3 are not (especially writes). If you are using LeveledCompaction, or otherwise have a ton of SSTables, some people have encountered increased costs moving the data to S3. Ourselves, we maintain backup EBS volumes that we regularly snaphot/rsync data too. Thus far this has worked very well for us. -Mike On Friday, December 6, 2013 8:14 AM, Marcelo Elias Del Valle marc...@s1mbi0se.com.br wrote: Hello everyone, I am trying to create backups of my data on AWS. My goal is to store the backups on S3 or glacier, as it's cheap to store this kind of data. So, if I have a cluster with N nodes, I would like to copy data from all N nodes to S3 and be able to restore later. I know Priam does that (we were using it), but I am using the latest cassandra version and we plan to use DSE some time, I am not sure Priam fits this case. I took a look at the docs: http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/operations/../../cassandra/operations/ops_backup_takes_snapshot_t.html And I am trying to understand if it's really needed to take a snapshot to create my backup. Suppose I do a flush and copy the sstables from each node, 1 by one, to s3. Not all at the same time, but one by one. When I try to restore my backup, data from node 1 will be older than data from node 2. Will this cause problems? AFAIK, if I am using a replication factor of 2, for instance, and Cassandra sees data from node X only, it will automatically copy it to other nodes, right? Is there any chance of cassandra nodes become corrupt somehow if I do my backups this way? Best regards, Marcelo Valle.
Write performance with 1.2.12
We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.0058.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.0058.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
help on backup muiltinode cluster
hey guys, I'm trying to take backups of a multi-node cassandra and save them on S3. My idea is simply doing ssh to each server and use nodetool to create the snapshots then push then to S3. So is this approach recommended? my concerns are about inconsistencies that this approach can lead, since the snapshots are taken one by one and not in parallel. Should i worry about it or cassandra finds a way to deal with inconsistencies when doing a restore? PS: I'm aware that datastax recommends to use pssh to take snapshots in parallel, but i couldn't use pssh because node tool requires you to specify the hostname. nodetool -h 10.10.10.1 snapshot thissnapshotname Any help would be appreciated. []'s
[JOB] - Full time opportunity in San Francisco bay area
We have a full time perm opportunity with a reputable client in the San Francisco bay area. Looking for good Cassandra and Java/ J2EE skills. Should you be interested, please reply with your contact number. Will call to discuss more. Thanks, * _* Gnani Balaraman | EthicalSoft, Inc. 2570 N 1st St, Ste 200, San Jose CA 95131 | ( (408) 329-0351 * gn...@ethicalsoft.com gn...@intelliswift.com
Re: OOMs during high (read?) load in Cassandra 1.2.11
Hi, Just taking a wild shot here, sorry if it does not help. Could it be thrown during reading the sstable? That is, try to find the configuration parameters for read operation, tune down a little for those settings. Also check on the the chunk_length_kb. http://www.datastax.com/documentation/cql/3.1/webhelp/cql/cql_reference/cql_storage_options_c.html /Jason On Fri, Dec 6, 2013 at 6:01 PM, Klaus Brunner klaus.brun...@gmail.comwrote: We're getting fairly reproducible OOMs on a 2-node cluster using Cassandra 1.2.11, typically in situations with a heavy read load. A sample of some stack traces is at https://gist.github.com/KlausBrunner/7820902 - they're all failing somewhere down from table.getRow(), though I don't know if that's part of query processing, compaction, or something else. - The main CFs contain some 100k rows, none of them particularly wide. - Heap dumps invariably show a single huge byte array (~1.6 GiB associated with the OOM'ing thread) hogging 80% of the Java heap. The array seems to contain all/many rows of one CF. - We're moderately certain there's no killer query with a huge result set involved here, but we can't see exactly what triggers this. - We've tried to switch to LeveledCompaction, to no avail. - Xms/x is set to some 4 GB. - The logs show the usual signs of panic (flushing memtables) before actually OOMing. It seems that this scenario is often or even always after a compaction, but it's not quite conclusive. I'm somewhat worried that Cassandra would read so much data into a single contiguous byte[] at any point. Could this be related to compaction? Any ideas what we could do about this? Thanks Klaus
Re: new project - Under Siege
Out of curiosity with quick look I found you have directory name as com.shift.undersiege https://github.com/StartTheShift/UnderSiege/blob/master/src/main/java/com.shift.undersiege/StatsdReporter.java You should have created the directory structure as src/main/java/com/shift/undersiege/StatsReporter.java. On Fri, Dec 6, 2013 at 12:14 AM, Jonathan Haddad j...@jonhaddad.com wrote: I've recently pushed up a new project to github, which we've named Under Siege. It's a java agent for reporting Cassandra metrics to statsd. We've in the process of deploying it to our production clusters. Tested against Cassandra 1.2.11. The metrics library seems to change on every release of C*, so I'm not sure what'll happen if you deploy against a different version. Might need to mvn package against the same version of metrics. https://github.com/StartTheShift/UnderSiege I'm not much of a Java programmer so there's probably about a hundred things I could have done better. Pull requests welcome. -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
Re: OOMs during high (read?) load in Cassandra 1.2.11
I am not sure if you had got a chance to take a look at this http://www.datastax.com/docs/1.1/troubleshooting/index#oom http://www.datastax.com/docs/1.1/install/recommended_settings Can you attach the cassandra logs and the cassandra.yaml, it should be able to give us more details about the issue? Thanks, -VK On Fri, Dec 6, 2013 at 3:31 PM, Klaus Brunner klaus.brun...@gmail.comwrote: We're getting fairly reproducible OOMs on a 2-node cluster using Cassandra 1.2.11, typically in situations with a heavy read load. A sample of some stack traces is at https://gist.github.com/KlausBrunner/7820902 - they're all failing somewhere down from table.getRow(), though I don't know if that's part of query processing, compaction, or something else. - The main CFs contain some 100k rows, none of them particularly wide. - Heap dumps invariably show a single huge byte array (~1.6 GiB associated with the OOM'ing thread) hogging 80% of the Java heap. The array seems to contain all/many rows of one CF. - We're moderately certain there's no killer query with a huge result set involved here, but we can't see exactly what triggers this. - We've tried to switch to LeveledCompaction, to no avail. - Xms/x is set to some 4 GB. - The logs show the usual signs of panic (flushing memtables) before actually OOMing. It seems that this scenario is often or even always after a compaction, but it's not quite conclusive. I'm somewhat worried that Cassandra would read so much data into a single contiguous byte[] at any point. Could this be related to compaction? Any ideas what we could do about this? Thanks Klaus
Re: how to find nodes by row key?
Thanks Rob, There is one thing bothers me. I have complex row key. $ create table b (x int, s text, ((x,s)) primary key); In cqlsh I cannot fill row key partially: $ insert into b (x) values(4); Bad Request: Missing mandatory PRIMARY KEY part s But nodetool can find hosts by incomplete key $ nodetool -h cas3 getendpoints anti_portal b 12 192.168.4.4 192.168.4.5 192.168.4.6 I found that columns are separated by :. And If I pass to many elements then the error happens. $ nodetool -h cas3 getendpoints anit_portal b 12:dd:dd Exception in thread main org.apache.cassandra.serializers.MarshalException: unable to make int from '12:dd:dd' at org.apache.cassandra.db.marshal.Int32Type.fromString(Int32Type.java:69) at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:2495) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487) at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at sun.rmi.transport.Transport$1.run(Transport.java:174) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.NumberFormatException: For input string: 12:dd:dd at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:492) at java.lang.Integer.parseInt(Integer.java:527) at org.apache.cassandra.db.marshal.Int32Type.fromString(Int32Type.java:65) ... 36 more On Thu, Dec 5, 2013 at 10:19 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Dec 5, 2013 at 9:58 AM, Daneel Yaitskov rtfm.rtfm.r...@gmail.comwrote: The best solution would be get IP address (node UUID) directly by a human readable structured row key. nodetool getendpoints exposes a JMX endpoint which does that. I'm not clear on whether that is sufficient for your needs. =Rob -- Daneel S. Yaitskov
Re: Write performance with 1.2.12
Hard to say much without knowing about the cassandra configurations. Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.0058.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.0058.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
Re: Try to configure commitlog_archiving.properties
Why, can you give me a good example and the good way to configure archive commit logs ? Take a look at the cassandra code ;) On Fri, Dec 6, 2013 at 3:34 PM, Bonnet Jonathan jonathan.bon...@externe.bnpparibas.com wrote: Hello, I try to configure commitlog_archiving.properties to take advantage of backup and restore at a point of time, but there is no ressources on internet for that. So i need some help. If I understand I have 4 parameters: archive_command= restore_command= restore_directories= restore_point_in_time= Forget for the moment the restore_point_in_time, should i put one command only for the archive_command; my wish is to copy all the commitlogs on another directory: restore_directories=/produits/cassandra/cassandra_data/archived_logs archive_command=/bin/cp - f /produits/cassandra/cassandra_commit/*.log /produits/cassandra/cassandra_ data/archived_logs restore_command=/bin/cp - f /produits/cassandra/cassandra_data/archived_logs/*.log /produits/cassandr a/cassandra_commit But it doesn't work when i restart the node which interest me, it doesn't copy anything. Why, can you give me a good example and the good way to configure archive commit logs ? there's nothing on the net except Datastax Website. Thanks for your answear. Regards, Bonnet Jonathan.
Re: cassandra backup
You should look at this - https://github.com/amorton/cassback i dont believe its setup to use 1.2.10 and above but i believe is just small tweeks to get it running. Thanks Rahul On Fri, Dec 6, 2013 at 7:09 PM, Michael Theroux mthero...@yahoo.com wrote: Hi Marcelo, Cassandra provides and eventually consistent model for backups. You can do staggered backups of data, with the idea that if you restore a node, and then do a repair, your data will be once again consistent. Cassandra will not automatically copy the data to other nodes (other than via hinted handoff). You should manually run repair after restoring a node. You should take snapshots when doing a backup, as it keeps the data you are backing up relevant to a single point in time, otherwise compaction could add/delete files one you mid-backup, or worse, I imagine attempt to access a SSTable mid-write. Snapshots work by using links, and don't take additional storage to perform. In our process we create the snapshot, perform the backup, and then clear the snapshot. One thing to keep in mind in your S3 cost analysis is that, even though storage is cheap, reads/writes to S3 are not (especially writes). If you are using LeveledCompaction, or otherwise have a ton of SSTables, some people have encountered increased costs moving the data to S3. Ourselves, we maintain backup EBS volumes that we regularly snaphot/rsync data too. Thus far this has worked very well for us. -Mike On Friday, December 6, 2013 8:14 AM, Marcelo Elias Del Valle marc...@s1mbi0se.com.br wrote: Hello everyone, I am trying to create backups of my data on AWS. My goal is to store the backups on S3 or glacier, as it's cheap to store this kind of data. So, if I have a cluster with N nodes, I would like to copy data from all N nodes to S3 and be able to restore later. I know Priam does that (we were using it), but I am using the latest cassandra version and we plan to use DSE some time, I am not sure Priam fits this case. I took a look at the docs: http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/operations/../../cassandra/operations/ops_backup_takes_snapshot_t.html And I am trying to understand if it's really needed to take a snapshot to create my backup. Suppose I do a flush and copy the sstables from each node, 1 by one, to s3. Not all at the same time, but one by one. When I try to restore my backup, data from node 1 will be older than data from node 2. Will this cause problems? AFAIK, if I am using a replication factor of 2, for instance, and Cassandra sees data from node X only, it will automatically copy it to other nodes, right? Is there any chance of cassandra nodes become corrupt somehow if I do my backups this way? Best regards, Marcelo Valle.
Re: Counters question - is there a better way to count
On Thu, Dec 5, 2013 at 7:44 AM, Christopher Wirt chris.w...@struq.comwrote: I want to build a really simple column family which counts the occurrence of a single event X. The guys from Disqus are big into counters: https://www.youtube.com/watch?v=A2WdS0YQADo http://www.slideshare.net/planetcassandra/cassandra-at-disqus (relevant slides start at 25) -- :- a) Alex Popescu @al3xandru
Re: How to monitor the progress of a HintedHandoff task?
Tom,
you should look at phi_convict_threshold and try and increase the value if
you have too much chatter on your network.
Also, rebuilding the entire node because of a OOM does not make sense,
could you please post the C* version that you are using the head size you
have configured?
Thanks
Rahul
On Tue, Dec 3, 2013 at 7:41 PM, Tom van den Berge t...@drillster.com wrote:
Rahul,
This problem occurs every now and then, and currently everything is ok, so
there are no hints. But whenever it happens, the hints are quickly piling
up. This results in heap problems on the node (Heap is 0.813462 full...
appears many times). This in turn results in the flushing of the 'hints'
column family, to relieve memory pressure. According to the log message,
the size varies between 50 and 60MB). But since the HintedHandoffManager is
reading from the hints CF, it will probably pull it back into a memtable
again -- that's at least my understanding of how it works.
So I guess that flushing the hints CF while the HintedHandoffManager is
working on it only makes things worse, and it could be the reason that the
process never ends.
What I typically see when this happens is that the hints keep piling up,
and eventually the node comes to a grinding halt (OOM). Then I have to
rebuild the node entirely (only removing the hints doesn't work).
The reason for hints to start accumulating in the first place might be a
spike in CF writes that must be replicated to a node in another data
center. The available bandwidth to that data center might not be able to
handle the data quickly enough, resulting in stored hints. The
HintedHandoff task that is started is targeting that remote node.
Thanks,
Tom
On Tue, Dec 3, 2013 at 2:22 PM, Rahul Menon ra...@apigee.com wrote:
Tom,
Do you know why these hints are piling up? What is the size of the hints
cf?
Thanks
Rahul
On Tue, Dec 3, 2013 at 6:41 PM, Tom van den Berge t...@drillster.comwrote:
Hi Rahul,
Thanks for your reply.
I have never seen message like Timed out replaying hints to..., which
is a good thing then, I suppose ;)
Normally, I do see the Finished hinted handoff... log message.
However, every now and then this message is not logged, not even after
several hours. This is the problem I'm trying to solve.
The log messages you describe are quite course-grained; they only tell
you that a task has started or finished, but not how this task is
progressing. And that's exactly what I would like to know if I see that a
task has started, but has not finished after a reasonable amount of time.
So I guess the only way to see learn the progress is to look inside the
'hints' column family then.I'll give that a try.
Thanks,
Tom
On Tue, Dec 3, 2013 at 1:43 PM, Rahul Menon ra...@apigee.com wrote:
Tom,
You should check the size of the hints column family to determine how
much are present. The hints are a super column family and its keys are
destination tokens. You could look at it if you would like.
Hints send and timedouts are logged, you should be seeing something
like
Timed out replaying hints to {}; aborting ({} delivered
OR
Finished hinted handoff of {} rows to endpoint {}
Thanks
Rahul
On Tue, Dec 3, 2013 at 2:36 PM, Tom van den Berge
t...@drillster.comwrote:
Hi,
Is there a way to monitor the progress of a hinted handoff task?
I found the following two mbeans providing some info:
org.apache.cassandra.internal:type=HintedHandoff, which tells me that
there is 1 active task, and
org.apache.cassandra.db:type=HintedHandoffManager#countPendingHints(),
which quite often gives a timeout when executed.
Ideally, I would like to see how many hints have been sent (e.g. over
the last minute or so), and how many hints are still to be sent (although
I
assume that's what countPendingHints normally does?)
I'm experiencing hinted handoff tasks that are started, but never
finish, so I would like to know what the task is doing.
My log shows this:
INFO [HintedHandoff:1] 2013-12-02
13:49:05,325 HintedHandOffManager.java (line 297) Started hinted handoff
for host: 6f80b942-5b6d-4233-9827-3727591abf55 with IP: /10.55.156.66
(nothing more for [HintedHandoff:1])
The node is up and running, the network connection is ok, no gossip
messages appear in the logs.
Any idea is welcome.
(Casandra 1.2.3)
--
Drillster BV
Middenburcht 136
3452MT Vleuten
Netherlands
+31 30 755 5330
Open your free account at www.drillster.com
--
Drillster BV
Middenburcht 136
3452MT Vleuten
Netherlands
+31 30 755 5330
Open your free account at www.drillster.com
--
Drillster BV
Middenburcht 136
3452MT Vleuten
Netherlands
+31 30 755 5330
Open your free account at www.drillster.com
Re: Write performance with 1.2.12
On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.com wrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
Re: Write performance with 1.2.12
Hi srmore, Perhaps if you use jconsole and connect to the jvm using jmx. Then uner MBeans tab, start inspecting the GC metrics. /Jason On Fri, Dec 6, 2013 at 11:40 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.com wrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
Re: Write performance with 1.2.12
You have passed the JVM configurations and not the cassandra configurations which is in cassandra.yaml. The spikes are not that significant in our case and we are running the cluster with 1.7 gb heap. Are these spikes causing any issue at your end? On Fri, Dec 6, 2013 at 9:10 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.com wrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
Re: cassandra backup
I believe SSTables are written to a temporary file then moved. If I remember correctly, tools like tablesnap listen for the inotify event IN_MOVED_TO. This should handle the try to back up sstable while in mid-write issue. On Fri, Dec 6, 2013 at 5:39 AM, Michael Theroux mthero...@yahoo.com wrote: Hi Marcelo, Cassandra provides and eventually consistent model for backups. You can do staggered backups of data, with the idea that if you restore a node, and then do a repair, your data will be once again consistent. Cassandra will not automatically copy the data to other nodes (other than via hinted handoff). You should manually run repair after restoring a node. You should take snapshots when doing a backup, as it keeps the data you are backing up relevant to a single point in time, otherwise compaction could add/delete files one you mid-backup, or worse, I imagine attempt to access a SSTable mid-write. Snapshots work by using links, and don't take additional storage to perform. In our process we create the snapshot, perform the backup, and then clear the snapshot. One thing to keep in mind in your S3 cost analysis is that, even though storage is cheap, reads/writes to S3 are not (especially writes). If you are using LeveledCompaction, or otherwise have a ton of SSTables, some people have encountered increased costs moving the data to S3. Ourselves, we maintain backup EBS volumes that we regularly snaphot/rsync data too. Thus far this has worked very well for us. -Mike On Friday, December 6, 2013 8:14 AM, Marcelo Elias Del Valle marc...@s1mbi0se.com.br wrote: Hello everyone, I am trying to create backups of my data on AWS. My goal is to store the backups on S3 or glacier, as it's cheap to store this kind of data. So, if I have a cluster with N nodes, I would like to copy data from all N nodes to S3 and be able to restore later. I know Priam does that (we were using it), but I am using the latest cassandra version and we plan to use DSE some time, I am not sure Priam fits this case. I took a look at the docs: http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/operations/../../cassandra/operations/ops_backup_takes_snapshot_t.html And I am trying to understand if it's really needed to take a snapshot to create my backup. Suppose I do a flush and copy the sstables from each node, 1 by one, to s3. Not all at the same time, but one by one. When I try to restore my backup, data from node 1 will be older than data from node 2. Will this cause problems? AFAIK, if I am using a replication factor of 2, for instance, and Cassandra sees data from node X only, it will automatically copy it to other nodes, right? Is there any chance of cassandra nodes become corrupt somehow if I do my backups this way? Best regards, Marcelo Valle. -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
Re: Write performance with 1.2.12
On Fri, Dec 6, 2013 at 9:59 AM, Vicky Kak vicky@gmail.com wrote: You have passed the JVM configurations and not the cassandra configurations which is in cassandra.yaml. Apologies, was tuning JVM and that's what was in my mind. Here are the cassandra settings http://pastebin.com/uN42GgYT The spikes are not that significant in our case and we are running the cluster with 1.7 gb heap. Are these spikes causing any issue at your end? There are no big spikes, the overall performance seems to be about 40% low. On Fri, Dec 6, 2013 at 9:10 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.com wrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
Re: Write performance with 1.2.12
Can you set the memtable_total_space_in_mb value, it is defaulting to 1/3 which is 8/3 ~ 2.6 gb in capacity http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-improved-memory-and-disk-space-management The flushing of 2.6 gb to the disk might slow the performance if frequently called, may be you have lots of write operations going on. On Fri, Dec 6, 2013 at 10:06 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:59 AM, Vicky Kak vicky@gmail.com wrote: You have passed the JVM configurations and not the cassandra configurations which is in cassandra.yaml. Apologies, was tuning JVM and that's what was in my mind. Here are the cassandra settings http://pastebin.com/uN42GgYT The spikes are not that significant in our case and we are running the cluster with 1.7 gb heap. Are these spikes causing any issue at your end? There are no big spikes, the overall performance seems to be about 40% low. On Fri, Dec 6, 2013 at 9:10 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.com wrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
Re: Write performance with 1.2.12
Looks like I am spending some time in GC. java.lang:type=GarbageCollector,name=ConcurrentMarkSweep CollectionTime = 51707; CollectionCount = 103; java.lang:type=GarbageCollector,name=ParNew CollectionTime = 466835; CollectionCount = 21315; On Fri, Dec 6, 2013 at 9:58 AM, Jason Wee peich...@gmail.com wrote: Hi srmore, Perhaps if you use jconsole and connect to the jvm using jmx. Then uner MBeans tab, start inspecting the GC metrics. /Jason On Fri, Dec 6, 2013 at 11:40 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.com wrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
Re: Write performance with 1.2.12
Since how long the server had been up, hours,days,months? On Fri, Dec 6, 2013 at 10:41 PM, srmore comom...@gmail.com wrote: Looks like I am spending some time in GC. java.lang:type=GarbageCollector,name=ConcurrentMarkSweep CollectionTime = 51707; CollectionCount = 103; java.lang:type=GarbageCollector,name=ParNew CollectionTime = 466835; CollectionCount = 21315; On Fri, Dec 6, 2013 at 9:58 AM, Jason Wee peich...@gmail.com wrote: Hi srmore, Perhaps if you use jconsole and connect to the jvm using jmx. Then uner MBeans tab, start inspecting the GC metrics. /Jason On Fri, Dec 6, 2013 at 11:40 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.com wrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
Re: Write performance with 1.2.12
Not long: Uptime (seconds) : 6828 Token: 56713727820156410577229101238628035242 ID : c796609a-a050-48df-bf56-bb09091376d9 Gossip active: true Thrift active: true Native Transport active: false Load : 49.71 GB Generation No: 1386344053 Uptime (seconds) : 6828 Heap Memory (MB) : 2409.71 / 8112.00 Data Center : DC Rack : RAC-1 Exceptions : 0 Key Cache: size 56154704 (bytes), capacity 104857600 (bytes), 27 hits, 155669426 requests, 0.000 recent hit rate, 14400 save period in seconds Row Cache: size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds On Fri, Dec 6, 2013 at 11:15 AM, Vicky Kak vicky@gmail.com wrote: Since how long the server had been up, hours,days,months? On Fri, Dec 6, 2013 at 10:41 PM, srmore comom...@gmail.com wrote: Looks like I am spending some time in GC. java.lang:type=GarbageCollector,name=ConcurrentMarkSweep CollectionTime = 51707; CollectionCount = 103; java.lang:type=GarbageCollector,name=ParNew CollectionTime = 466835; CollectionCount = 21315; On Fri, Dec 6, 2013 at 9:58 AM, Jason Wee peich...@gmail.com wrote: Hi srmore, Perhaps if you use jconsole and connect to the jvm using jmx. Then uner MBeans tab, start inspecting the GC metrics. /Jason On Fri, Dec 6, 2013 at 11:40 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.com wrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
AddContractPoint /VIP
Greetings, I have 4 node cassandra cluster that will grow upt to 10 nodes,we are using CQL Java client to access the data. What is the good practice to put in the code as addContactPoint ie.,how many servers ? 1) I am also thinking to put this way here I am not sure this good or bad if i conigure 4 serves into one VIP ( virtual IP/virtual DNS) and specifying that DSN in the code as ContactPoint, so that that VIP is smart enough to route to different nodes. 2) Is that problem if i use multiple Data centers in future ? thanks Chandra
Re: cassandra performance problems
On Thu, Dec 5, 2013 at 6:33 AM, Alexander Shutyaev shuty...@gmail.comwrote: We've plugged it into our production environment as a cache in front of postgres. Everything worked fine, we even stressed it by explicitly propagating about 30G (10G/node) data from postgres to cassandra. If you just want a caching layer, why wouldn't you use Memcached or Redis instead? Cassandra is designed to be a persist store and not so much designed as caching layer. If you were replacing your use of Postgres completely, that would be appropriate.
Re: Write performance with 1.2.12
Changed memtable_total_space_in_mb to 1024 still no luck. On Fri, Dec 6, 2013 at 11:05 AM, Vicky Kak vicky@gmail.com wrote: Can you set the memtable_total_space_in_mb value, it is defaulting to 1/3 which is 8/3 ~ 2.6 gb in capacity http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-improved-memory-and-disk-space-management The flushing of 2.6 gb to the disk might slow the performance if frequently called, may be you have lots of write operations going on. On Fri, Dec 6, 2013 at 10:06 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:59 AM, Vicky Kak vicky@gmail.com wrote: You have passed the JVM configurations and not the cassandra configurations which is in cassandra.yaml. Apologies, was tuning JVM and that's what was in my mind. Here are the cassandra settings http://pastebin.com/uN42GgYT The spikes are not that significant in our case and we are running the cluster with 1.7 gb heap. Are these spikes causing any issue at your end? There are no big spikes, the overall performance seems to be about 40% low. On Fri, Dec 6, 2013 at 9:10 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.com wrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
Re: cassandra backup
On Fri, Dec 6, 2013 at 5:13 AM, Marcelo Elias Del Valle marc...@s1mbi0se.com.br wrote: I am trying to create backups of my data on AWS. My goal is to store the backups on S3 or glacier, as it's cheap to store this kind of data. So, if I have a cluster with N nodes, I would like to copy data from all N nodes to S3 and be able to restore later. https://github.com/synack/tablesnap Automated backup, restore, purging, intended for use with Cassandra. =Rob
Re: help on backup muiltinode cluster
On Fri, Dec 6, 2013 at 6:41 AM, Amalrik Maia amal...@s1mbi0se.com.brwrote: hey guys, I'm trying to take backups of a multi-node cassandra and save them on S3. My idea is simply doing ssh to each server and use nodetool to create the snapshots then push then to S3. https://github.com/synack/tablesnap So is this approach recommended? my concerns are about inconsistencies that this approach can lead, since the snapshots are taken one by one and not in parallel. Should i worry about it or cassandra finds a way to deal with inconsistencies when doing a restore? The backup is as consistent as your cluster is at any given moment, which is not necessarily. Manual repair brings you closer to consistency, but only on data present when the repair started. =Rob
calculating sizes on disk
I am trying to do some disk capacity planning. I have been referring the datastax docs[1] and this older blog post[2]. I have a column family with the following, row key - 4 bytes column name - 8 bytes column value - 8 bytes max number of non-deleted columns per row - 20160 Is there an effective way to calculate the sizes (or at least a decent approximation) of the bloom filters and partition indexes on disk? [1] Calculating user data sizehttp://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docsversion=1.2file=index#cassandra/architecture/../../cassandra/architecture/architecturePlanningUserData_t.html [2] Cassandra Storage Sizing http://btoddb-cass-storage.blogspot.com/ -- - John
Re: calculating sizes on disk
I should have also mentioned that I have tried using the calculations from the storage sizing post. My lack of success may be due to the post basing things off of Cassandra 0.8 as well as a lack of understanding in how to do some of the calculations. On Fri, Dec 6, 2013 at 3:08 PM, John Sanda john.sa...@gmail.com wrote: I am trying to do some disk capacity planning. I have been referring the datastax docs[1] and this older blog post[2]. I have a column family with the following, row key - 4 bytes column name - 8 bytes column value - 8 bytes max number of non-deleted columns per row - 20160 Is there an effective way to calculate the sizes (or at least a decent approximation) of the bloom filters and partition indexes on disk? [1] Calculating user data sizehttp://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docsversion=1.2file=index#cassandra/architecture/../../cassandra/architecture/architecturePlanningUserData_t.html [2] Cassandra Storage Sizing http://btoddb-cass-storage.blogspot.com/ -- - John -- - John
Re: vnodes on aws
On Thu, Dec 5, 2013 at 6:58 PM, Andrey Ilinykh ailin...@gmail.com wrote: On Thu, Dec 5, 2013 at 3:31 PM, Jayadev Jayaraman jdisal...@gmail.comwrote: Availability zones are analogous to racks not data centres . EC2 regions are equivalent to data centres. Yes, this is what I meant. I guess my question is - is possible to put row in every rack using vnodes? https://issues.apache.org/jira/browse/CASSANDRA-4123 I should point out that the existing NTS (and also the modified implementation in CASSANDRA-3881) already prevent replicas from being placed on the same host. SimpleStrategy also is fixed in the CASSANDRA-4121 patch. The first couple of paragraphs in the ticket description could probably be removed. https://issues.apache.org/jira/browse/CASSANDRA-4121 https://issues.apache.org/jira/browse/CASSANDRA-3881 =Rob
Re: calculating sizes on disk
Not sure what your end setup will be, but I would probably just spin up a cluster and fill it with typical data to and measure the size on disk. __ Sent from iPhone On 7 Dec 2013, at 6:08 am, John Sanda john.sa...@gmail.com wrote: I am trying to do some disk capacity planning. I have been referring the datastax docs[1] and this older blog post[2]. I have a column family with the following, row key - 4 bytes column name - 8 bytes column value - 8 bytes max number of non-deleted columns per row - 20160 Is there an effective way to calculate the sizes (or at least a decent approximation) of the bloom filters and partition indexes on disk? [1] Calculating user data size [2] Cassandra Storage Sizing -- - John
Re: calculating sizes on disk
I have done that, but it only gets me so far because the cluster and app that manages it is run by 3rd parties. Ideally, I would like to provide my end users with a formula or heuristic for establishing some sort of baselines that at least gives them a general idea for planning. Generating data as you have suggested and as I have done is helpful, but it is hard for users to extrapolate out from that. On Fri, Dec 6, 2013 at 3:47 PM, Jacob Rhoden jacob.rho...@me.com wrote: Not sure what your end setup will be, but I would probably just spin up a cluster and fill it with typical data to and measure the size on disk. __ Sent from iPhone On 7 Dec 2013, at 6:08 am, John Sanda john.sa...@gmail.com wrote: I am trying to do some disk capacity planning. I have been referring the datastax docs[1] and this older blog post[2]. I have a column family with the following, row key - 4 bytes column name - 8 bytes column value - 8 bytes max number of non-deleted columns per row - 20160 Is there an effective way to calculate the sizes (or at least a decent approximation) of the bloom filters and partition indexes on disk? [1] Calculating user data sizehttp://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docsversion=1.2file=index#cassandra/architecture/../../cassandra/architecture/architecturePlanningUserData_t.html [2] Cassandra Storage Sizing http://btoddb-cass-storage.blogspot.com/ -- - John -- - John
Re: datastax community ami -- broken? --- datastax-agent conflicts with opscenter-agent
Hello again John, The AMI has been patched and tested for both DSE and C* and works for the standard 3 node test. The new code has been pushed to the 2.4 branch so launching a new set of instances will give you an updated AMI. You should now have the newest version of OpsCenter installed, along with the new DataStax Agents (that replace the OpsCenter Agents). Also, I've patched the two bugs for the motd and for allowing the other nodes to join. The issue came from a new release of nodetool that contained some unexpected text that choked up the AMI code as it waited for nodes to come online. Let me know if you see any further issues. Thanks, Joaquin Casares DataStax Software Engineer in Test http://www.datastax.com/what-we-offer/products-services/training/virtual-training On Fri, Dec 6, 2013 at 2:02 PM, Joaquin Casares joaq...@datastax.comwrote: Hey John, Thanks for letting us know. I'm also seeing that the motd gets stuck, but if I ctrl-c during the message and try a `nodetool status` there doesn't appear to be an issue. I'm currently investigating why it's getting stuck. Are you seeing something similar? What happens if you try to run a `sudo service cassandra restart`? Could you send me your /var/log/cassandra/system.log if this still fails? Also, I realize now that the package name for the newest version of OpsCenter changed from opscenter-free to opscenter. I committed that change to our dev AMI and am testing it now. Once this change is made you will no longer have to install agents via OpsCenter since they should already be on the system. That being said, you won't hit the current OpsCenter/DataStax Agent version mismatch you've been hitting. Also, we currently only have one AMI. Each time an instance is launched the newest version of the code is pulled down from https://github.com/riptano/ComboAMI to ensure the code never gets stale and can easily keep up with DSE/C* releases as well as AMI code fixes. I'll reply again as soon as I figure out and patch this motd issue. Thanks, Joaquin Casares DataStax Software Engineer in Test http://www.datastax.com/what-we-offer/products-services/training/virtual-training On Fri, Dec 6, 2013 at 7:16 AM, John R. Frank j...@mit.edu wrote: Hi C* experts, In the last 18hrs or so, I have been having trouble getting cassandra instances to launch using the datastax community AMI. Has anyone else seen this? The instance comes up but then cassandra fails to run. The most informative error message that I've seen so far is in the obscenter agent install log (below) --- see especially this line: datastax-agent conflicts with opscenter-agent I have also attached the ami.log I run into these issues when launching with either the browser page: https://aws.amazon.com/amis/datastax-auto-clustering-ami-2-2 Or command line, like this: ec2-run-instances ami-814ec2e8 -t m1.large --region us-east-1 --key buildbot -n $num_instances -d --clustername Foo --totalnodes $num_instances --version community --opscenter yes Is there a newer AMI? Any advice? jrf Some agent installations failed: - 10.6.133.241: Failure installing agent on 10.6.133.241. Error output: Unable to install the opscenter-agent package. Please check your apt-get configuration as well as the agent install log (/var/log/opscenter-agent/ installer.log). Standard output: Removing old opscenter-agent files. opscenter-agent: unrecognized service Reading package lists... Building dependency tree... Reading state information... 0 upgraded, 0 newly installed, 0 to remove and 171 not upgraded. Starting agent installation process for version 3.2.2 Reading package lists... Building dependency tree... Reading state information... sysstat is already the newest version. 0 upgraded, 0 newly installed, 0 to remove and 171 not upgraded. Selecting previously unselected package opscenter-agent. dpkg: regarding .../opscenter-agent.deb containing opscenter-agent: datastax-agent conflicts with opscenter-agent opscenter-agent (version 3.2.2) is to be installed. opscenter-agent provides opscenter-agent and is to be installed. dpkg: error processing opscenter_agent_setup.vYRzL0Tevn/opscenter-agent.deb (--install): conflicting packages - not installing opscenter-agent Errors were encountered while processing: opscenter_agent_setup.vYRzL0Tevn/opscenter-agent.deb FAILURE: Unable to install the opscenter-agent package. Please check your apt-get configuration as well as the agent install log (/var/log/opscenter-agent/installer.log). Exit code: 1
Re: datastax community ami -- broken? --- datastax-agent conflicts with opscenter-agent
Hi Joaquin, A quick word of praise - addressing the issue so quickly presents a really good view of Datastax cheers On Sat, Dec 7, 2013 at 8:14 AM, Joaquin Casares joaq...@datastax.comwrote: Hello again John, The AMI has been patched and tested for both DSE and C* and works for the standard 3 node test. The new code has been pushed to the 2.4 branch so launching a new set of instances will give you an updated AMI. You should now have the newest version of OpsCenter installed, along with the new DataStax Agents (that replace the OpsCenter Agents). Also, I've patched the two bugs for the motd and for allowing the other nodes to join. The issue came from a new release of nodetool that contained some unexpected text that choked up the AMI code as it waited for nodes to come online. Let me know if you see any further issues. Thanks, Joaquin Casares DataStax Software Engineer in Test http://www.datastax.com/what-we-offer/products-services/training/virtual-training On Fri, Dec 6, 2013 at 2:02 PM, Joaquin Casares joaq...@datastax.comwrote: Hey John, Thanks for letting us know. I'm also seeing that the motd gets stuck, but if I ctrl-c during the message and try a `nodetool status` there doesn't appear to be an issue. I'm currently investigating why it's getting stuck. Are you seeing something similar? What happens if you try to run a `sudo service cassandra restart`? Could you send me your /var/log/cassandra/system.log if this still fails? Also, I realize now that the package name for the newest version of OpsCenter changed from opscenter-free to opscenter. I committed that change to our dev AMI and am testing it now. Once this change is made you will no longer have to install agents via OpsCenter since they should already be on the system. That being said, you won't hit the current OpsCenter/DataStax Agent version mismatch you've been hitting. Also, we currently only have one AMI. Each time an instance is launched the newest version of the code is pulled down from https://github.com/riptano/ComboAMI to ensure the code never gets stale and can easily keep up with DSE/C* releases as well as AMI code fixes. I'll reply again as soon as I figure out and patch this motd issue. Thanks, Joaquin Casares DataStax Software Engineer in Test http://www.datastax.com/what-we-offer/products-services/training/virtual-training On Fri, Dec 6, 2013 at 7:16 AM, John R. Frank j...@mit.edu wrote: Hi C* experts, In the last 18hrs or so, I have been having trouble getting cassandra instances to launch using the datastax community AMI. Has anyone else seen this? The instance comes up but then cassandra fails to run. The most informative error message that I've seen so far is in the obscenter agent install log (below) --- see especially this line: datastax-agent conflicts with opscenter-agent I have also attached the ami.log I run into these issues when launching with either the browser page: https://aws.amazon.com/amis/datastax-auto-clustering-ami-2-2 Or command line, like this: ec2-run-instances ami-814ec2e8 -t m1.large --region us-east-1 --key buildbot -n $num_instances -d --clustername Foo --totalnodes $num_instances --version community --opscenter yes Is there a newer AMI? Any advice? jrf Some agent installations failed: - 10.6.133.241: Failure installing agent on 10.6.133.241. Error output: Unable to install the opscenter-agent package. Please check your apt-get configuration as well as the agent install log (/var/log/opscenter-agent/ installer.log). Standard output: Removing old opscenter-agent files. opscenter-agent: unrecognized service Reading package lists... Building dependency tree... Reading state information... 0 upgraded, 0 newly installed, 0 to remove and 171 not upgraded. Starting agent installation process for version 3.2.2 Reading package lists... Building dependency tree... Reading state information... sysstat is already the newest version. 0 upgraded, 0 newly installed, 0 to remove and 171 not upgraded. Selecting previously unselected package opscenter-agent. dpkg: regarding .../opscenter-agent.deb containing opscenter-agent: datastax-agent conflicts with opscenter-agent opscenter-agent (version 3.2.2) is to be installed. opscenter-agent provides opscenter-agent and is to be installed. dpkg: error processing opscenter_agent_setup.vYRzL0Tevn/opscenter-agent.deb (--install): conflicting packages - not installing opscenter-agent Errors were encountered while processing: opscenter_agent_setup.vYRzL0Tevn/opscenter-agent.deb FAILURE: Unable to install the opscenter-agent package. Please check your apt-get configuration as well as the agent install log (/var/log/opscenter-agent/installer.log). Exit code: 1 -- *Franc Carter* | Systems architect | Sirca Ltd marc.zianideferra...@sirca.org.au franc.car...@sirca.org.au | www.sirca.org.au Tel: +61 2 8355 2514 Level
