Re: Cassandra Metrics
Hi, SPM will monitor your Cassandra cluster - http://sematext.com/spm/integrations/cassandra-monitoring.html Otis On Jun 19, 2015, at 05:41, Saurabh Chandolia s.chando...@gmail.com wrote: Hi, I have recently started using Cassandra. As of now, I am only using cfstats and cfhistograms to monitor individual CF stats. What all cassandra metrics should I watch for stability and performance? Are there any tools to do the same? Is there any performance overhead if I start monitoring too many metrics? Thanks - Saurabh
Re: Hbase vs Cassandra
Hi Ajay, You won't be able to get unbiased opinion here easily. You'll need to try and see how each works for your use case. We use HBase for the SPM backend and it has worked well for us - it's stable, handles billions and billions of rows (I lost track of the actual number many moons ago) and fast, if you get your key design right. For our particular use case, HBase turned out to be a better choice, but we looked at Cassandra, too, back when we chose HBase. I'll answer your Q about monitoring: I'd say both are equally well monitorable. SPM http://sematext.com/spm can monitor both HBase and Cassandra equally well. Because Cassandra is a bit simpler (vs. HBase having multiple processes one needs to run), it's a bit simpler to add monitoring to Cassandra, but the difference is small. SPM is at http://sematext.com/spm if you want to have a look. We expose our own HBase clusters in the live demo, so you can see what metrics HBase exposes. We don't run Cassandra, so we can't show its graphs, but you can see some charts, metrics, and filters for Cassandra at http://blog.sematext.com/2014/06/02/announcement-cassandra-performance-monitoring-in-spm/ I hope this helps. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Fri, May 29, 2015 at 3:09 PM, Ajay ajay.ga...@gmail.com wrote: Hi, I need some info on Hbase vs Cassandra as a data store (in general plus specific to time series data). The comparison in the following helps: 1: features 2: deployment and monitoring 3: performance 4: anything else Thanks Ajay
Re: opscenter with community cassandra
Hi Tim, We have happy SPM for Cassandra users who are using SPM for monitoring, alerting, and anomaly detection for their Cassandra clusters. SPM agents phone home by definition if you are using the Cloud version. The On Premise / AWS AMI versions do not phone home. See: http://sematext.com/spm http://blog.sematext.com/tag/cassandra/ http://blog.sematext.com/2014/06/02/announcement-cassandra-performance-monitoring-in-spm/ Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Tue, Oct 28, 2014 at 12:03 PM, Tim Dunphy bluethu...@gmail.com wrote: Furthermore, people ask questions about monitoring and management utilities for Cassandra all the time--this is in the same vein. Speaking of which. Are there any viable alternatives to opscenter that people also like? On Tue, Oct 28, 2014 at 11:56 AM, Redmumba redmu...@gmail.com wrote: Furthermore, people ask questions about monitoring and management utilities for Cassandra all the time--this is in the same vein. On Tue, Oct 28, 2014 at 8:52 AM, Ken Hancock ken.hanc...@schange.com wrote: Your criteria for what is appropriate for production may differ from others, but it's equally incorrect of you to make a blanket statement that OpsCenter isn't suitable for production. A number of people use it in production. On Tue, Oct 28, 2014 at 11:48 AM, Colin co...@clark.ws wrote: No, actually, you cant Tyler. If you mean the useless information it provides outside of licence, fine, if you mean the components outside, then same argument. Last time i checked, this forumn was about apache and not about datastax. Maybe a separate group should be deducated to provider specific offerings. -- *Colin Clark* +1-320-221-9531 On Oct 28, 2014, at 10:41 AM, Tyler Hobbs ty...@datastax.com wrote: On Tue, Oct 28, 2014 at 10:08 AM, Colin colpcl...@gmail.com wrote: It is a mistake to call a proprietary piece of software community when you cant use it in production. You can use OpsCenter community in production (however you'd like). -- Tyler Hobbs DataStax http://datastax.com/ -- *Ken Hancock *| System Architect, Advanced Advertising SeaChange International 50 Nagog Park Acton, Massachusetts 01720 ken.hanc...@schange.com | www.schange.com | NASDAQ:SEAC http://www.schange.com/en-US/Company/InvestorRelations.aspx Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hanc...@schange.com | [image: Skype:]hancockks | [image: Yahoo IM:] hancockks[image: LinkedIn] http://www.linkedin.com/in/kenhancock [image: SeaChange International] http://www.schange.com/This e-mail and any attachments may contain information which is SeaChange International confidential. The information enclosed is intended only for the addressees herein and may not be copied or forwarded without permission from SeaChange International. -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
Re: High cpu usage segfaulting
Hi Stan,
Put some monitoring on this. The first thing I think of when I hear
chewing up CPU for Java apps is GC. In SPM http://sematext.com/spm/
you can easily see individual JVM memory pools and see if any of them are
at (close to) 100%. You can typically correlate that to increased GC times
and counts. I'd look at that before looking at strace and such.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr Elasticsearch Support * http://sematext.com/
On Tue, Nov 25, 2014 at 11:07 PM, Stan Lemon sle...@salesforce.com wrote:
We are using v2.0.11 and have seen several instances in our 24 node
cluster where the node becomes unresponsive, when we look into it we find
that there is a cassandra process chewing up a lot of CPU. There are no
other indications in logs or anything as to what might be happening,
however if we strace the process that is chewing up CPU we see a segmental
fault:
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigreturn(0x7fd61110f862)= 30618997712
futex(0x7fd614844054, FUTEX_WAIT_PRIVATE, 27333, NULL) = -1 EAGAIN
(Resource temporarily unavailable)
futex(0x7fd614844028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fd6148e2e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd6148e2e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7fd6148e2e28, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7fd614844054, FUTEX_WAIT_PRIVATE, 27335, NULL) = 0
futex(0x7fd614844028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fd6148e2e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd6148e2e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7fd6148e2e28, FUTEX_WAKE_PRIVATE, 1) = 1
And this happens over and over again while running strafe.
Has anyone seen this? Does anyone have any ideas what might be happening,
or how we could debug it further?
Thanks for your help,
Stan
Programmatic Cassandra version detection/extraction
Hi, Is there a way to detect which version of Cassandra one is running? Is there an API for that, or a constant with this value, or maybe an MBean or some other way to get to this info? Here's the use case: SPM monitors Cassandra http://sematext.com/spm/, but Cassandra MBeans and metrics have or may change over time. How will SPM agent know which MBeans to look for, which metrics to extract, and how to interpret values it extracts without knowing which version of Cassandra it's monitoring? It could try probing for some known MBeans and deduce Cassandra version from that, but that feels a little sloppy. Ideally, we'd be able to grab the version from some MBean and based on that extract metrics we know are exposed in that version of Cassandra. Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Operating on large cluster
Hi Alain, We use Puppet and introducing Ansible at Sematext. Not for Cassandra, but for other similar tech. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Thu, Oct 23, 2014 at 5:18 AM, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi, I was wondering about how do you guys handle a large cluster (50+ machines). I mean there is sometime you need to change configuration (cassandra.yaml) or send a command to one, some or all nodes (cleanup, upgradesstables, setstramthoughput or whatever). So far we have been using things like custom scripts for repairs or any routine maintenance and cssh for specific and one shot actions on the cluster. But I guess this doesn't really scale, I guess we coul use pssh instead. For configuration changes we use Capistrano that might scale properly. So I would like to known, what are the methods that operators use on large cluster out there ? Have some of you built some open sourced cluster management interfaces or scripts that could make things easier while operating on large Cassandra clusters ? Alain
Cassandra Wiki Immutable?
Hi, What is the state of Cassandra Wiki -- http://wiki.apache.org/cassandra ? I tried to update a few pages, but it looks like pages are immutable. Do I need to have my Wiki username (OtisGospodnetic) added to some ACL? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/
Re: Cassandra Wiki Immutable?
Ah, I missed that. Thanks. Email sent. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Mon, Aug 18, 2014 at 12:19 PM, Mark Reddy mark.re...@boxever.com wrote: Hi Otis, On the front page, https://wiki.apache.org/cassandra/FrontPage there are instructions on how to get edit permissions: If you would like to contribute to this wiki, please send an email to the mailing list dev.at.cassandra.apache-dot-org with your wiki username and we will be happy to add you. Contributions welcome! Mark On Mon, Aug 18, 2014 at 11:15 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, What is the state of Cassandra Wiki -- http://wiki.apache.org/cassandra ? I tried to update a few pages, but it looks like pages are immutable. Do I need to have my Wiki username (OtisGospodnetic) added to some ACL? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/
Re: Changes in the Metrics MBean
Hi, On Tue, Aug 5, 2014 at 1:37 AM, Patricia Gorla patri...@thelastpickle.com wrote: Bhavishya, All metrics that Cassandra emits are segmented by host name. For a cluster-wide view, you will need a separate metrics reporter such as OpsCenter or Riemann. Or SPM for Cassandra: http://blog.sematext.com/2014/06/02/announcement-cassandra-performance-monitoring-in-spm/ Otis See http://thelastpickle.com/blog/2013/11/22/4-hours-with-riemann.html for more information. Thanks, On Thu, Jul 31, 2014 at 9:39 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, Btw. what do you mean by overall metrics for a particular cluster? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Thu, Jul 10, 2014 at 9:13 AM, bhavishya.p bhavishy...@zohocorp.com wrote: Hi.. There seems to be a new feature added in the Metrics MBean which provides the overall metrics for a particular cluster. In which version was this feature added? Also, how can we determine in which version the changes are made in general? Thanks in advance. Regards, Bhavishya Palukurthi -- Patricia Gorla @patriciagorla Consultant Apache Cassandra Consulting http://www.thelastpickle.com http://thelastpickle.com
Re: Changes in the Metrics MBean
Hi, Btw. what do you mean by overall metrics for a particular cluster? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Thu, Jul 10, 2014 at 9:13 AM, bhavishya.p bhavishy...@zohocorp.com wrote: Hi.. There seems to be a new feature added in the Metrics MBean which provides the overall metrics for a particular cluster. In which version was this feature added? Also, how can we determine in which version the changes are made in general? Thanks in advance. Regards, Bhavishya Palukurthi
ANN: Cassandra monitoring anomaly detection
Hi, We've added Cassandra performance monitoring and anomaly detection to SPM http://sematext.com/spm/ recently. Here's the announcement with several screenshots: http://blog.sematext.com/2014/06/02/announcement-cassandra-performance-monitoring-in-spm/ Storm, Kafka, Elasticsearch, Hadoop... monitoring is in there, too, in case you use that with Cassandra. Actually, this may be of interest if you are running the Kafka+Storm+Cassandra (or Kafka+Storm+Elasticsearch) combo: http://blog.sematext.com/2014/06/11/monitoring-kafka-storm-and-cassandra-together-with-spm/ Otis
ANN: All Cassandra Resources Searchable
Hi, We did this a while back, but never notified Cassandra community - we're indexing all Apache Cassandra resources - user/dev lists, wiki, website, JIRA, source code, and javadoc over at http://search-hadoop.com/cassandra . Some Apache projects, like Hadoop - http://hadoop.apache.org/ and HBase - http://hbase.apache.org/ have added a search box to their site to search everything we index. Others have just linked to their indexed content, like Kafka - http://kafka.apache.org/contact.html Enjoy. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/
