RE: Log Slow Queries
Hi Carlos, Different possibilities (to log slow queries). 1) A probabilistic way to catch slow queries (probabilistic, but with detailed info) = look for “nodetool settraceprobability” like in http://www.datastax.com/dev/blog/advanced-request-tracing-in-cassandra-1-2 2) Catch slow queries in the driver (it’s a recent feature, available for newest drivers only) = see http://datastax.github.io/java-driver/2.0.10/features/logging/#logging-query-latencies 3) Catch slow queries on server-side (but only with C* 2.1) = see slides 15-17 Lesser Known Features of Cassandra 2.1 http://fr.slideshare.net/planetcassandra/cassandra-summit-2014-lesser-known-features-of-cassandra-21 On our side, we are more keen to use (2) – which has the best ROI (IMHO). Regards, Dominique [@@ THALES GROUP INTERNAL @@] De : Carlos Alonso [mailto:i...@mrcalonso.com] Envoyé : jeudi 18 juin 2015 12:33 À : user@cassandra.apache.org Objet : Log Slow Queries Hi guys. I'm facing slow read requests from time to time, I've spotted the keyspace/cf where this is happening but I can't see anything obvious (single partition slice query, no tombstones, ...) anything else where to look at? I'd like to have the slow queries logged to either log or saved to a particular column family to analyse them later. I've googled about this and the only 'easy' solution available out there seems to be DataStax Enterprise. What are you guys using? Thanks, Carlos Alonso | Software Engineer | @calonsohttps://twitter.com/calonso
nodetool repair
Hi, I want to make on a regular base repairs on my cluster as suggested by the documentation. I want to do this in a way that the cluster is still responding to read requests. So I understand that I should not use the -par switch for that as it will do the repair in parallel and consume all available resources. If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? If we had down time periods I would issue a nodetool -par, but we don’t have down time periods. Sorry for the stupid questions. Thanks for your help.
Re: Deploying OpsCenter behind a HTTP(S) proxy
OpsCenter is a little bit tricky to simply just rewrite urls, the lhr requests and rest endpoints it hits are all specified a little differently in the javascript app it loads. We ended up monkey patching a buttload of the js files to get all the requests working properly with our proxy. Everytime a new release of OpsCenter comes out we have to rework it. If you are a DSE customer I would raise it as a support issue :) On 18 June 2015 at 02:29, Spencer Brown lilspe...@gmail.com wrote: First, your firewall should really be your frontend There operational frontend is apache, which is common. You want every url with opscenter in it handled elsewhere. You could also set up proxies for /. cluster-configs, etc... Then there is mod_rewrite, which provides a lot more granularity about when you want what gets handled where.I set up the architectural infrastructure for Orbitz and some major banks, and I'd be happpy to help you out on this. I charge $30/hr., but what you need isn't very complex so we're really just talking $100. On Thu, Jun 18, 2015 at 5:13 AM, Jonathan Ballet jbal...@gfproducts.ch wrote: Hi, I'm looking for information on how to correctly deploy an OpsCenter instance behind a HTTP(S) proxy. I have a running instance of OpsCenter 5.1 reachable at http://opscenter:/opscenter/ but I would like to be able to serve this kind of tool under a single hostname on HTTPS along with other tools of this kind, for easier convenience. I'm currently using Apache as my HTTP front-end and I tried this naive configuration: VirtualHost *:80 ServerName tools ... ProxyPreserveHost On # Proxy to OpsCenter # ProxyPass /opscenter/ http://opscenter:/opscenter/ ProxyPassReverse/opscenter/ http://opscenter:/opscenter/ /VirtualHost This doesn't quite work, as OpsCenter seem to also serve specific endpoints from / directly Of course, it doesn't correctly work, as OpsCenter seem to also serve specific data from / directly, such as: /cluster-configs /TestCluster /meta /rc /tcp Is there something I can configure in OpsCenter so that it serves these URLs from somewhere else, or a list of known URLs that I can remap on the proxy, or better yet, a known proxy configuration to put in front of OpsCenter? Regards, Jonathan -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: nodetool repair
If you're using DSE, you can schedule it automatically using the repair service. If you're open source, check out Spotify cassandra reaper, it'll manage it for you. https://github.com/spotify/cassandra-reaper On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay jean.tremb...@zen-innovations.com wrote: Hi, I want to make on a regular base repairs on my cluster as suggested by the documentation. I want to do this in a way that the cluster is still responding to read requests. So I understand that I should not use the -par switch for that as it will do the repair in parallel and consume all available resources. If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? If we had down time periods I would issue a nodetool -par, but we don’t have down time periods. Sorry for the stupid questions. Thanks for your help.
Re: nodetool repair
Thanks Jonathan. But I need to know the following: If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? Kind regards On 18 Jun 2015, at 19:19 , Jonathan Haddad j...@jonhaddad.commailto:j...@jonhaddad.com wrote: If you're using DSE, you can schedule it automatically using the repair service. If you're open source, check out Spotify cassandra reaper, it'll manage it for you. https://github.com/spotify/cassandra-reaper On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com wrote: Hi, I want to make on a regular base repairs on my cluster as suggested by the documentation. I want to do this in a way that the cluster is still responding to read requests. So I understand that I should not use the -par switch for that as it will do the repair in parallel and consume all available resources. If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? If we had down time periods I would issue a nodetool -par, but we don’t have down time periods. Sorry for the stupid questions. Thanks for your help.
sstableloader Could not retrieve endpoint ranges
I'm using sstableloader to bulk-load a table from one cluster to another. I can't just copy sstables because the clusters have different topologies. While we're looking to upgrade soon to Cassandra 2.0.x, we're on Cassandra 1.2.19. The source data comes from a nodetool snapshot. Here's the command I ran: sstableloader -d *IP_ADDRESSES_OF_SEED_NOTES* */SNAPSHOT_DIRECTORY/* Here's the result I got: Could not retrieve endpoint ranges: -pr,--principal kerberos principal -k,--keytab keytab location --ssl-keystoressl keystore location --ssl-keystore-password ssl keystore password --ssl-keystore-type ssl keystore type --ssl-truststore ssl truststore location --ssl-truststore-password ssl truststore password --ssl-truststore-type ssl truststore type Not sure what to make of this, what with the hints at security arguments that pop up. The source and destination clusters have no security. Hoping this might ring a bell with someone out there.
Re: nodetool repair
Hi Jean, Running nodetool repair on a node will repair only that node in the cluster. It is recommended to run nodetool repair on one node at a time. Few things to keep in mind while running repair 1. Running repair will trigger compactions 2. Increase in CPU utilization. Run node tool repair with -pr option, so that it will repair only the range that node is responsible for. On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay jean.tremb...@zen-innovations.com wrote: Thanks Jonathan. But I need to know the following: If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? Kind regards On 18 Jun 2015, at 19:19 , Jonathan Haddad j...@jonhaddad.com wrote: If you're using DSE, you can schedule it automatically using the repair service. If you're open source, check out Spotify cassandra reaper, it'll manage it for you. https://github.com/spotify/cassandra-reaper On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay jean.tremb...@zen-innovations.com wrote: Hi, I want to make on a regular base repairs on my cluster as suggested by the documentation. I want to do this in a way that the cluster is still responding to read requests. So I understand that I should not use the -par switch for that as it will do the repair in parallel and consume all available resources. If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? If we had down time periods I would issue a nodetool -par, but we don’t have down time periods. Sorry for the stupid questions. Thanks for your help. -- Arun Senior Hadoop/Cassandra Engineer Cloudwick 2014 Data Impact Award Winner (Cloudera) http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html
Re: Deploying OpsCenter behind a HTTP(S) proxy
In your examples, there are front pages then there are links to more info on different things. Do you want me to just write a front page or to provide content for the different links? On Thu, Jun 18, 2015 at 5:13 AM, Jonathan Ballet jbal...@gfproducts.ch wrote: Hi, I'm looking for information on how to correctly deploy an OpsCenter instance behind a HTTP(S) proxy. I have a running instance of OpsCenter 5.1 reachable at http://opscenter:/opscenter/ but I would like to be able to serve this kind of tool under a single hostname on HTTPS along with other tools of this kind, for easier convenience. I'm currently using Apache as my HTTP front-end and I tried this naive configuration: VirtualHost *:80 ServerName tools ... ProxyPreserveHost On # Proxy to OpsCenter # ProxyPass /opscenter/ http://opscenter:/opscenter/ ProxyPassReverse/opscenter/ http://opscenter:/opscenter/ /VirtualHost This doesn't quite work, as OpsCenter seem to also serve specific endpoints from / directly Of course, it doesn't correctly work, as OpsCenter seem to also serve specific data from / directly, such as: /cluster-configs /TestCluster /meta /rc /tcp Is there something I can configure in OpsCenter so that it serves these URLs from somewhere else, or a list of known URLs that I can remap on the proxy, or better yet, a known proxy configuration to put in front of OpsCenter? Regards, Jonathan
Re: Garbage collector launched on all nodes at once
How much memory do you have? Recently people have been seeing really great performance using G1GC with heaps 8GB and offheap memtable objects. On Thu, Jun 18, 2015 at 1:31 AM Jason Wee peich...@gmail.com wrote: okay, iirc memtable has been removed off heap, google and got this http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1 apparently, there are still some reference on heap. On Thu, Jun 18, 2015 at 1:11 PM, Marcus Eriksson krum...@gmail.com wrote: It is probably this: https://issues.apache.org/jira/browse/CASSANDRA-9549 On Wed, Jun 17, 2015 at 7:37 PM, Michał Łowicki mlowi...@gmail.com wrote: Looks that memtable heap size is growing on some nodes rapidly ( https://www.dropbox.com/s/3brloiy3fqang1r/Screenshot%202015-06-17%2019.21.49.png?dl=0). Drops are the places when nodes have been restarted. On Wed, Jun 17, 2015 at 6:53 PM, Michał Łowicki mlowi...@gmail.com wrote: Hi, Two datacenters with 6 nodes (2.1.6) each. In each DC garbage collection is launched at the same time on each node (See [1] for total GC duration per 5 seconds). RF is set to 3. Any ideas? [1] https://www.dropbox.com/s/bsbyew1jxbe3dgo/Screenshot%202015-06-17%2018.49.48.png?dl=0 -- BR, Michał Łowicki -- BR, Michał Łowicki
Re: Minor compaction not triggered
Hi, Curently the cluster runs 2.0.14 on all nodes, will try to upgrade to 2.0.15 The issue is in all the nodes.. simple restart as in a rolling restart of all the nodes? On 18 June 2015 at 15:18, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi, From what I read there are some issue using DTCS. You should probably go to the last minor version. You should also give more context so people can help. Have you tried a simple restart ? Does it happen on all your nodes ? C*heers, Alain 2015-06-18 11:42 GMT+02:00 Jayapandian Ponraj pandian...@gmail.com: Can anyone guide me with any pointers plz On 17 June 2015 at 17:05, Jayapandian Ponraj pandian...@gmail.com wrote: Hi I have a cassandra cluster of 6 nodes, with DateTiered compaction for the tables/CFs For some reason the minor compaction never happens. I have enabled debug logging and I don't see any debug logs related to compaction like the following https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L150 https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java#L127 As a result of no compactions, now the cluster has more than 50K SStables per node. How do i debug this issue further? Appreciate any help..
Re: compaction occurring very frequently
Hi, 2.1.x does have compaction issues. Upgrading to 2.1.6 should help you. You can also search this mailing list for some recommendations for that specific problem since it is a recurrent issue. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 www.pythian.com On Thu, Jun 18, 2015 at 9:27 AM, Rahul Bhardwaj rahul.bhard...@indiamart.com wrote: Hi All, We have a cluster (2.1.2) of 3 nodes , There is one table which contains huge amount of data and having sstable count more than 1800 on all 3 nodes. We have noticed that suddently compaction for that CF occurs very frequently right after completion of previous compaction and also taking long time in compacting. Par New collection time and CMS collection time have also increased on those node. Please suggest what could be the problem. Regards: Rahul Bhardwaj Follow IndiaMART.com http://www.indiamart.com for latest updates on this and more: https://plus.google.com/+indiamart https://www.facebook.com/IndiaMART https://twitter.com/IndiaMART Mobile Channel: https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641mt=8 https://play.google.com/store/apps/details?id=com.indiamart.m http://m.indiamart.com/ https://www.youtube.com/watch?v=DzORNbeSXN8list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1index=2 Check out some smart tips to generate more business - *IndiaMART Learning Centre* http://corporate.indiamart.com/quick-learn/ -- --
compaction occurring very frequently
Hi All, We have a cluster (2.1.2) of 3 nodes , There is one table which contains huge amount of data and having sstable count more than 1800 on all 3 nodes. We have noticed that suddently compaction for that CF occurs very frequently right after completion of previous compaction and also taking long time in compacting. Par New collection time and CMS collection time have also increased on those node. Please suggest what could be the problem. Regards: Rahul Bhardwaj -- Follow IndiaMART.com http://www.indiamart.com for latest updates on this and more: https://plus.google.com/+indiamart https://www.facebook.com/IndiaMART https://twitter.com/IndiaMART Mobile Channel: https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641mt=8 https://play.google.com/store/apps/details?id=com.indiamart.m http://m.indiamart.com/ https://www.youtube.com/watch?v=DzORNbeSXN8list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1index=2 Check out some smart tips to generate more business - *IndiaMART Learning Centre* http://corporate.indiamart.com/quick-learn/
Re: Deploying OpsCenter behind a HTTP(S) proxy
First, your firewall should really be your frontend There operational frontend is apache, which is common. You want every url with opscenter in it handled elsewhere. You could also set up proxies for /. cluster-configs, etc... Then there is mod_rewrite, which provides a lot more granularity about when you want what gets handled where.I set up the architectural infrastructure for Orbitz and some major banks, and I'd be happpy to help you out on this. I charge $30/hr., but what you need isn't very complex so we're really just talking $100. On Thu, Jun 18, 2015 at 5:13 AM, Jonathan Ballet jbal...@gfproducts.ch wrote: Hi, I'm looking for information on how to correctly deploy an OpsCenter instance behind a HTTP(S) proxy. I have a running instance of OpsCenter 5.1 reachable at http://opscenter:/opscenter/ but I would like to be able to serve this kind of tool under a single hostname on HTTPS along with other tools of this kind, for easier convenience. I'm currently using Apache as my HTTP front-end and I tried this naive configuration: VirtualHost *:80 ServerName tools ... ProxyPreserveHost On # Proxy to OpsCenter # ProxyPass /opscenter/ http://opscenter:/opscenter/ ProxyPassReverse/opscenter/ http://opscenter:/opscenter/ /VirtualHost This doesn't quite work, as OpsCenter seem to also serve specific endpoints from / directly Of course, it doesn't correctly work, as OpsCenter seem to also serve specific data from / directly, such as: /cluster-configs /TestCluster /meta /rc /tcp Is there something I can configure in OpsCenter so that it serves these URLs from somewhere else, or a list of known URLs that I can remap on the proxy, or better yet, a known proxy configuration to put in front of OpsCenter? Regards, Jonathan
Re: Minor compaction not triggered
Can anyone guide me with any pointers plz On 17 June 2015 at 17:05, Jayapandian Ponraj pandian...@gmail.com wrote: Hi I have a cassandra cluster of 6 nodes, with DateTiered compaction for the tables/CFs For some reason the minor compaction never happens. I have enabled debug logging and I don't see any debug logs related to compaction like the following https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L150 https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java#L127 As a result of no compactions, now the cluster has more than 50K SStables per node. How do i debug this issue further? Appreciate any help..
Deploying OpsCenter behind a HTTP(S) proxy
Hi, I'm looking for information on how to correctly deploy an OpsCenter instance behind a HTTP(S) proxy. I have a running instance of OpsCenter 5.1 reachable at http://opscenter:/opscenter/ but I would like to be able to serve this kind of tool under a single hostname on HTTPS along with other tools of this kind, for easier convenience. I'm currently using Apache as my HTTP front-end and I tried this naive configuration: VirtualHost *:80 ServerName tools ... ProxyPreserveHost On # Proxy to OpsCenter # ProxyPass /opscenter/ http://opscenter:/opscenter/ ProxyPassReverse/opscenter/ http://opscenter:/opscenter/ /VirtualHost This doesn't quite work, as OpsCenter seem to also serve specific endpoints from / directly Of course, it doesn't correctly work, as OpsCenter seem to also serve specific data from / directly, such as: /cluster-configs /TestCluster /meta /rc /tcp Is there something I can configure in OpsCenter so that it serves these URLs from somewhere else, or a list of known URLs that I can remap on the proxy, or better yet, a known proxy configuration to put in front of OpsCenter? Regards, Jonathan
Re: Minor compaction not triggered
Hi, From what I read there are some issue using DTCS. You should probably go to the last minor version. You should also give more context so people can help. Have you tried a simple restart ? Does it happen on all your nodes ? C*heers, Alain 2015-06-18 11:42 GMT+02:00 Jayapandian Ponraj pandian...@gmail.com: Can anyone guide me with any pointers plz On 17 June 2015 at 17:05, Jayapandian Ponraj pandian...@gmail.com wrote: Hi I have a cassandra cluster of 6 nodes, with DateTiered compaction for the tables/CFs For some reason the minor compaction never happens. I have enabled debug logging and I don't see any debug logs related to compaction like the following https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L150 https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java#L127 As a result of no compactions, now the cluster has more than 50K SStables per node. How do i debug this issue further? Appreciate any help..
Log Slow Queries
Hi guys. I'm facing slow read requests from time to time, I've spotted the keyspace/cf where this is happening but I can't see anything obvious (single partition slice query, no tombstones, ...) anything else where to look at? I'd like to have the slow queries logged to either log or saved to a particular column family to analyse them later. I've googled about this and the only 'easy' solution available out there seems to be DataStax Enterprise. What are you guys using? Thanks, Carlos Alonso | Software Engineer | @calonso https://twitter.com/calonso
