RE: Cassandra monitoring

[Manu Chadha]

Just to try my luck, I added MCAC jar in cassandra.bat in bin directory

set JAVA_OPTS=-ea^
-javaagent:"%CASSANDRA_HOME%\lib\jamm-0.3.0.jar"^
-javaagent:"%CASSANDRA_HOME%\datastax-mcac-agent-0.1.13\lib\datastax-mcac-agent.jar"^
 <-HERE
-Xms2G^
-Xmx2G^
-XX:+HeapDumpOnOutOfMemoryError^
-XX:+UseParNewGC^
-XX:+UseConcMarkSweepGC^
-XX:+CMSParallelRemarkEnabled^
-XX:SurvivorRatio=8^
-XX:MaxTenuringThreshold=1^
-XX:CMSInitiatingOccupancyFraction=75^
-XX:+UseCMSInitiatingOccupancyOnly^
-Dlogback.configurationFile=logback.xml^
-Djava.library.path="%CASSANDRA_HOME%\lib\sigar-bin"^
-Dcassandra.jmx.local.port=7199

But this is now giving error

java.lang.RuntimeException: Cannot locate 
file:\\C:\Users\manuc\Documents\manu\apache-cassandra-3.11.6-for-scalardb\datastax-mcac-agent-0.1.13/config/metric-collector.yaml.

Note the change in path separator from backslash to frontslash at the end.

To be honest, still clueless on how to get metrics on my windows machine.

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Manu Chadha<mailto:manu.cha...@hotmail.com>
Sent: 18 December 2020 10:04
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: RE: Cassandra monitoring

Thanks. I tried it but I still don’t see anything on http://localhost:9103/.

My suspicion is that I am not putting the jars or am not setting the JVM_OPTS 
correctly or maybe the path is not being picked correctly considering that I am 
on windows machine and .sh is basically a unix script. I have tried three 
exporters and they all seem to have issues (except for Cassandra exporter 
standalone which works fine - https://github.com/criteo/cassandra_exporter).

So I am getting more certain that I am not putting javaagent correctly.

I downloaded the jar from 
https://github.com/datastax/metric-collector-for-apache-cassandra/releases/tag/v0.1.13
Then unzipped it

I added path like this

MCAC_ROOT="C:\\Users\\manuc\\Documents\\manu\\datastax-mcac-agent-0.1.13"
JVM_OPTS="$JVM_OPTS -javaagent:${MCAC_ROOT}\\lib\\datastax-mcac-agent.jar"

And then restarted Cassandra but no love ☹

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Ahmed Eljami<mailto:ahmed.elj...@gmail.com>
Sent: 18 December 2020 09:21
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Cassandra monitoring

Hi,

I suggest you try MCAC for DataStax, available here => 
https://github.com/datastax/metric-collector-for-apache-cassandra

Cheers,

Le ven. 18 déc. 2020 à 10:01, Manu Chadha 
mailto:manu.cha...@hotmail.com>> a écrit :
Sorry, clicked send too soon.

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

I thought I’ll test it on my Windows laptop first but I can’t connect with 
using localhost:7199.


As I am on Windows, instead of echo 'JVM_OPTS="$JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"' 
>> conf/cassandra-env.sh, I have added the following line in the 
cassandra-env.ps1 (I don’t know to be honest if I should have)

# Default JMX setup, bound to local loopback address only

$env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"

$env:JVM_OPTS="$env:JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"' 
<-- HERE



$env:JVM_OPTS="$env:JVM_OPTS $env:JVM_EXTRA_OPTS"

}



but http://localhost:7070/metrics doesn't work

I have also tried to do the original change in cassandar.yaml i.e. 
JVM_OPTS="$JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml

But I still can’t access the metrics.

Any help or other suggestions about monitoring would be welcome.

Thanks
Manu

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Manu Chadha<mailto:manu.cha...@hotmail.com>
Sent: 18 December 2020 08:57
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Cassandra monitoring

Hi

Could the group please suggest options for monitoring Cassandra in production 
(GCP)?

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10




--
Cordialement;
Ahmed ELJAMI

RE: Cassandra monitoring

[Manu Chadha]

Thanks. I tried it but I still don’t see anything on http://localhost:9103/.

My suspicion is that I am not putting the jars or am not setting the JVM_OPTS 
correctly or maybe the path is not being picked correctly considering that I am 
on windows machine and .sh is basically a unix script. I have tried three 
exporters and they all seem to have issues (except for Cassandra exporter 
standalone which works fine - https://github.com/criteo/cassandra_exporter).

So I am getting more certain that I am not putting javaagent correctly.

I downloaded the jar from 
https://github.com/datastax/metric-collector-for-apache-cassandra/releases/tag/v0.1.13
Then unzipped it

I added path like this

MCAC_ROOT="C:\\Users\\manuc\\Documents\\manu\\datastax-mcac-agent-0.1.13"
JVM_OPTS="$JVM_OPTS -javaagent:${MCAC_ROOT}\\lib\\datastax-mcac-agent.jar"

And then restarted Cassandra but no love ☹

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Ahmed Eljami<mailto:ahmed.elj...@gmail.com>
Sent: 18 December 2020 09:21
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Cassandra monitoring

Hi,

I suggest you try MCAC for DataStax, available here => 
https://github.com/datastax/metric-collector-for-apache-cassandra

Cheers,

Le ven. 18 déc. 2020 à 10:01, Manu Chadha 
mailto:manu.cha...@hotmail.com>> a écrit :
Sorry, clicked send too soon.

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

I thought I’ll test it on my Windows laptop first but I can’t connect with 
using localhost:7199.


As I am on Windows, instead of echo 'JVM_OPTS="$JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"' 
>> conf/cassandra-env.sh, I have added the following line in the 
cassandra-env.ps1 (I don’t know to be honest if I should have)

# Default JMX setup, bound to local loopback address only

$env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"

$env:JVM_OPTS="$env:JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"' 
<-- HERE



$env:JVM_OPTS="$env:JVM_OPTS $env:JVM_EXTRA_OPTS"

}



but http://localhost:7070/metrics doesn't work

I have also tried to do the original change in cassandar.yaml i.e. 
JVM_OPTS="$JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml

But I still can’t access the metrics.

Any help or other suggestions about monitoring would be welcome.

Thanks
Manu

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Manu Chadha<mailto:manu.cha...@hotmail.com>
Sent: 18 December 2020 08:57
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Cassandra monitoring

Hi

Could the group please suggest options for monitoring Cassandra in production 
(GCP)?

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10




--
Cordialement;
Ahmed ELJAMI

Re: Cassandra monitoring

[Ahmed Eljami]

Hi,

I suggest you try MCAC for DataStax, available here =>
https://github.com/datastax/metric-collector-for-apache-cassandra

Cheers,

Le ven. 18 déc. 2020 à 10:01, Manu Chadha  a
écrit :

> Sorry, clicked send too soon.
>
>
>
> I am trying to use Prometheus and followed this tutorial  -
> https://www.robustperception.io/monitoring-cassandra-with-prometheus
>
>
>
> I thought I’ll test it on my Windows laptop first but I can’t connect with
> using localhost:7199.
>
>
>
> As I am on Windows, instead of echo 'JVM_OPTS="$JVM_OPTS
> -javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"'
> >> conf/cassandra-env.sh, I have added the following line in the 
> >> cassandra-env.ps1
> (I don’t know to be honest if I should have)
>
> # Default JMX setup, bound to local loopback address only
>
> $env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"
>
> $env:JVM_OPTS="$env:JVM_OPTS 
> -javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"'
>  <-- HERE
>
>
>
> $env:JVM_OPTS="$env:JVM_OPTS $env:JVM_EXTRA_OPTS"
>
> }
>
>
>
> but http://localhost:7070/metrics doesn't work
>
>
>
> I have also tried to do the original change in cassandar.yaml i.e. 
> JVM_OPTS="$JVM_OPTS
> -javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml
>
>
>
> But I still can’t access the metrics.
>
>
>
> Any help or other suggestions about monitoring would be welcome.
>
>
>
> Thanks
>
> Manu
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> *From: *Manu Chadha 
> *Sent: *18 December 2020 08:57
> *To: *user@cassandra.apache.org
> *Subject: *Cassandra monitoring
>
>
>
> Hi
>
>
>
> Could the group please suggest options for monitoring Cassandra in
> production (GCP)?
>
>
>
> I am trying to use Prometheus and followed this tutorial  -
> https://www.robustperception.io/monitoring-cassandra-with-prometheus
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
>
>


-- 
Cordialement;

Ahmed ELJAMI

RE: Cassandra monitoring

[Manu Chadha]

Sorry, clicked send too soon.

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

I thought I’ll test it on my Windows laptop first but I can’t connect with 
using localhost:7199.


As I am on Windows, instead of echo 'JVM_OPTS="$JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"' 
>> conf/cassandra-env.sh, I have added the following line in the 
cassandra-env.ps1 (I don’t know to be honest if I should have)

# Default JMX setup, bound to local loopback address only

$env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"

$env:JVM_OPTS="$env:JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"' 
<-- HERE



$env:JVM_OPTS="$env:JVM_OPTS $env:JVM_EXTRA_OPTS"

}



but http://localhost:7070/metrics doesn't work

I have also tried to do the original change in cassandar.yaml i.e. 
JVM_OPTS="$JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml

But I still can’t access the metrics.

Any help or other suggestions about monitoring would be welcome.

Thanks
Manu

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Manu Chadha<mailto:manu.cha...@hotmail.com>
Sent: 18 December 2020 08:57
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Cassandra monitoring

Hi

Could the group please suggest options for monitoring Cassandra in production 
(GCP)?

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

Cassandra monitoring

[Manu Chadha]

Hi

Could the group please suggest options for monitoring Cassandra in production 
(GCP)?

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

Sent from Mail for Windows 10

Re: EXT: Cassandra Monitoring tool

[Rahul Singh]

Good article about it on LI

https://www.linkedin.com/pulse/snap-cassandra-s3-tablesnap-vijaya-kumar-hosamani/

On May 25, 2018, 2:52 PM -0500, Joaquin Casares <joaq...@thelastpickle.com>, 
wrote:
> Hello Aneesh,
>
> While this doesn't provide a GUI, tablesnap is a community tool that does a 
> great job at handling backups:
>
> > https://github.com/JeremyGrosser/tablesnap
>
> Cheers,
>
> Joaquin
>
> Joaquin Casares
> Consultant
> Austin, TX
>
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> > On Fri, May 25, 2018 at 11:36 AM, ANEESH KUMAR K.M <anee...@gmail.com> 
> > wrote:
> > > Thank you Hari for the details.
> > >
> > > One more question, please suggest me some cluster managing too for 
> > > Cassandra cluster. Looking for opensource tools that will support to take 
> > > snapshot and restore via GUI.
> > >
> > > Regards,
> > > Aneesh
> > >
> > > > On Fri, May 25, 2018 at 10:00 PM, Harikrishnan Pillai 
> > > > <hpil...@walmartlabs.com> wrote:
> > > > > I assume you are using open source cassandra and you can look at 
> > > > > Prometheus grafana for cassandra monitoring and lot information 
> > > > > available in internet regarding how to setup the Prometheus 
> > > > > monitoring for cassandra .
> > > > >
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On May 25, 2018, at 9:23 AM, ANEESH KUMAR K.M <anee...@gmail.com> 
> > > > > > wrote:
> > > > > >
> > > > > > Please suggest me some good cluster monitoring tool for cassandra 
> > > > > > multi region cluster.
> > > > > >
> > > > >
> > > > > -
> > > > > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > > > > For additional commands, e-mail: user-h...@cassandra.apache.org
> > > > >
> > >
>

Re: EXT: Cassandra Monitoring tool

[Joaquin Casares]

Hello Aneesh,

While this doesn't provide a GUI, tablesnap is a community tool that does a
great job at handling backups:

https://github.com/JeremyGrosser/tablesnap


Cheers,

Joaquin

Joaquin Casares
Consultant
Austin, TX

Apache Cassandra Consulting
http://www.thelastpickle.com

On Fri, May 25, 2018 at 11:36 AM, ANEESH KUMAR K.M <anee...@gmail.com>
wrote:

> Thank you Hari for the details.
>
> One more question, please suggest me some cluster managing too for
> Cassandra cluster. Looking for opensource tools that will support to take
> snapshot and restore via GUI.
>
> Regards,
> Aneesh
>
> On Fri, May 25, 2018 at 10:00 PM, Harikrishnan Pillai <
> hpil...@walmartlabs.com> wrote:
>
>> I assume you are using open source cassandra and you can look at
>> Prometheus grafana for cassandra monitoring and lot information available
>> in internet regarding how to setup the Prometheus monitoring for cassandra .
>>
>>
>> Sent from my iPhone
>>
>> > On May 25, 2018, at 9:23 AM, ANEESH KUMAR K.M <anee...@gmail.com>
>> wrote:
>> >
>> > Please suggest me some good cluster monitoring tool for cassandra multi
>> region cluster.
>> >
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>

Re: Cassandra Monitoring tool

[Joaquin Casares]

Hello Aneesh,

Alain and I also worked on this project that you might find useful:

https://github.com/thelastpickle/docker-cassandra-bootstrap


It is a Docker Compose environment to showcase the Prometheus graphs that
we worked on. There's still a missing update, so do please check back soon,
but the majority of what we've worked on is housed in that repo. If you
decide to move forward with Prometheus, you can use the curl scripts
provided in the repo to upload the charts and all preset alerts into your
Grafana environment. If you use the GUI to upload the graphs, you'll lose
all preset visual thresholds and alerts, but still keep all completed
tooltips/descriptions.

Now that I remembered it was missing an update, we should push the new
features out sooner rather than later. The new update includes better
descriptive text, alert text, and alert fixes.

Hope it helps!

Joaquin

Joaquin Casares
Consultant
Austin, TX

Apache Cassandra Consulting
http://www.thelastpickle.com

On Fri, May 25, 2018 at 12:54 PM, Alain RODRIGUEZ 
wrote:

> Hello,
>
> With more details on your criteria, it would be easier.
>
> I used:
>
> - SPM from Sematext (Commercial - dashboards out of the box)
> - Datastax OpsCenter (Commercial, With DSE only nowadays)
> - Grafana/Graphite or Prometheus (Open source - 'do it yourself' - but
> some templates exist)
> - Datadog (Commercial - dashboards out of the box)
>
> It depends what you want really. All those worked for me at some point in
> time :).
>
> Note: We built the Dashboards that are out of the box for Datadog. I
> believe it should put anyone starting with Cassandra in good conditions to
> operate and troubleshot Apache Cassandra. My opinion here is biased most
> definitely :). I liked the interfaces of Datadog (using regularly) and
> Sematext (old experience) the most.
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2018-05-25 17:23 GMT+01:00 ANEESH KUMAR K.M :
>
>> Please suggest me some good cluster monitoring tool for cassandra multi
>> region cluster.
>>
>>
>

Re: Cassandra Monitoring tool

[Alain RODRIGUEZ]

Hello,

With more details on your criteria, it would be easier.

I used:

- SPM from Sematext (Commercial - dashboards out of the box)
- Datastax OpsCenter (Commercial, With DSE only nowadays)
- Grafana/Graphite or Prometheus (Open source - 'do it yourself' - but some
templates exist)
- Datadog (Commercial - dashboards out of the box)

It depends what you want really. All those worked for me at some point in
time :).

Note: We built the Dashboards that are out of the box for Datadog. I
believe it should put anyone starting with Cassandra in good conditions to
operate and troubleshot Apache Cassandra. My opinion here is biased most
definitely :). I liked the interfaces of Datadog (using regularly) and
Sematext (old experience) the most.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-05-25 17:23 GMT+01:00 ANEESH KUMAR K.M :

> Please suggest me some good cluster monitoring tool for cassandra multi
> region cluster.
>
>

Re: EXT: Cassandra Monitoring tool

[ANEESH KUMAR K.M]

Thank you Hari for the details.

One more question, please suggest me some cluster managing too for
Cassandra cluster. Looking for opensource tools that will support to take
snapshot and restore via GUI.

Regards,
Aneesh

On Fri, May 25, 2018 at 10:00 PM, Harikrishnan Pillai <
hpil...@walmartlabs.com> wrote:

> I assume you are using open source cassandra and you can look at
> Prometheus grafana for cassandra monitoring and lot information available
> in internet regarding how to setup the Prometheus monitoring for cassandra .
>
>
> Sent from my iPhone
>
> > On May 25, 2018, at 9:23 AM, ANEESH KUMAR K.M <anee...@gmail.com> wrote:
> >
> > Please suggest me some good cluster monitoring tool for cassandra multi
> region cluster.
> >
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: EXT: Cassandra Monitoring tool

[Harikrishnan Pillai]

I assume you are using open source cassandra and you can look at Prometheus 
grafana for cassandra monitoring and lot information available in internet 
regarding how to setup the Prometheus monitoring for cassandra .


Sent from my iPhone

> On May 25, 2018, at 9:23 AM, ANEESH KUMAR K.M <anee...@gmail.com> wrote:
> 
> Please suggest me some good cluster monitoring tool for cassandra multi 
> region cluster.
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Cassandra Monitoring tool

[ANEESH KUMAR K.M]

Please suggest me some good cluster monitoring tool for cassandra multi
region cluster.

Re: Cassandra monitoring

[Julien Anguenot]

Hey Otis, 

hehe :-) 

I do have the latest agents running but these metrics are still empty on my 
side. Will take that issue on Sematext side then.

Thanks.

   J.

> On Jun 15, 2016, at 8:26 AM, Otis Gospodnetić  
> wrote:
> 
> Hi,
> 
> On Wed, Jun 15, 2016 at 8:58 AM, Julien Anguenot  > wrote:
> 
>> On Jun 14, 2016, at 2:10 PM, Arun Ramakrishnan > > wrote:
> 
> […]
> 
>> 
>> Sematext has a integrations for monitoring cassandra. Does anyone have good 
>> experience with it ?
> 
> We are using Sematext with Cassandra 3.0.x and it is mostly working fine for 
> us expect couple of metrics (pending writes and pending cluster ops) that 
> never got supported after Casandra 2.2.x. Did report the issue but it never 
> got fixed. Otis if you are listening? :-)
> 
> I am - almost always ;)
> We've added those metrics you mentioned a while back - see the various tabs 
> on the left in the attached screenshots.  Make sure you have the latest SPM 
> agent for Cassandra.
> 
> Otis
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ 
> 
>  
> 
> --
> Julien Anguenot (@anguenot)
> 
> 
>  AM.png>

--
Julien Anguenot (@anguenot)

Re: Cassandra monitoring

[Otis Gospodnetić]

Hi,

On Tue, Jun 14, 2016 at 4:20 PM, Arun Ramakrishnan <
sinchronized.a...@gmail.com> wrote:

> Thanks Jonathan.
>
> Out of curiosity, does opscenter support some later version of cassandra
> that is not OSS ?
>
> Well, the most minimal requirement is that, I want to be able to monitor
> for cluster health and hook this info to some alerting platform. We are AWS
> heavy. We just really heavily on AWS cloud watch for our metrics as of now.
> We prefer to not spend our time setting up additional tools if we can help
> it. So, if we needed a 3rd party service we would consider an APM or
> monitoring service that is on the cheaper side.
>

Right.  Using N open-source tools can lead to the image like the one on
https://sematext.com/blog/2015/04/22/monitoring-stream-processing-tools-cassandra-kafka-and-spark/

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/




>
>
>
>
> On Tue, Jun 14, 2016 at 12:20 PM, Jonathan Haddad 
> wrote:
>
>> Depends what you want to monitor.  I wouldn't use a lesser version of
>> Cassandra for OpsCenter, it doesn't give you a ton you can't get elsewhere
>> and it's not ever going to support OSS > 2.1, so you kind of limit yourself
>> to a pretty old version of Cassandra for a non-good reason.
>>
>> What else do you use for monitoring in your infra?  I've used a mix of
>> OSS tools (nagios, statsd, graphite, ELK), and hosted solutions. The nice
>> part about them is that you can monitor your whole stack in a single UI not
>> just your database.
>>
>> On Tue, Jun 14, 2016 at 12:10 PM Arun Ramakrishnan <
>> sinchronized.a...@gmail.com> wrote:
>>
>>> What are the options for a very small and nimble startup to do keep a
>>> cassandra cluster running well oiled. We are on AWS. We are interested in a
>>> monitoring tool and potentially also cluster management tools.
>>>
>>> We are currently on apache cassandra 3.7. We were hoping the datastax
>>> opscenter would be it (It is free for startups our size). But, looks like
>>> it does not support cassandra versions greater than v2.1. It is pretty
>>> surprising considering cassandra v2.1  came out in 2014.
>>>
>>> We would consider downgrading to datastax cassandra 2.1 just to have
>>> robust monitoring tools. But, I am not sure if having opscenter offsets all
>>> the improvements that have been added to cassandra since 2.1.
>>>
>>> Sematext has a integrations for monitoring cassandra. Does anyone have
>>> good experience with it ?
>>>
>>> How much work would be involved to setup Ganglia or some such option for
>>> cassandra ?
>>>
>>> Thanks,
>>> Arun
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>

Re: Cassandra monitoring

[Julien Anguenot]

> On Jun 14, 2016, at 2:10 PM, Arun Ramakrishnan  
> wrote:

[…]

> 
> Sematext has a integrations for monitoring cassandra. Does anyone have good 
> experience with it ?

We are using Sematext with Cassandra 3.0.x and it is mostly working fine for us 
expect couple of metrics (pending writes and pending cluster ops) that never 
got supported after Casandra 2.2.x. Did report the issue but it never got 
fixed. Otis if you are listening? :-)

[…]

   J.

--
Julien Anguenot (@anguenot)

Re: Cassandra monitoring

[Kai Wang]

I use graphite/jmxtrans/collectd to monitor not just Cassandra but also
other jvm applications as well as OS. I found it's more useful and flexible
than opscenter in terms of monitoring.
On Jun 14, 2016 3:10 PM, "Arun Ramakrishnan" 
wrote:

What are the options for a very small and nimble startup to do keep a
cassandra cluster running well oiled. We are on AWS. We are interested in a
monitoring tool and potentially also cluster management tools.

We are currently on apache cassandra 3.7. We were hoping the datastax
opscenter would be it (It is free for startups our size). But, looks like
it does not support cassandra versions greater than v2.1. It is pretty
surprising considering cassandra v2.1  came out in 2014.

We would consider downgrading to datastax cassandra 2.1 just to have robust
monitoring tools. But, I am not sure if having opscenter offsets all the
improvements that have been added to cassandra since 2.1.

Sematext has a integrations for monitoring cassandra. Does anyone have good
experience with it ?

How much work would be involved to setup Ganglia or some such option for
cassandra ?

Thanks,
Arun

Re: Cassandra monitoring

[Michał Łowicki]

My team ended up with Diamond / StatsD / Graphite / Grafana (more
background in
medium.com/@mlowicki/alternatives-to-datastax-opscenter-8ad893efe063).
We're relying on such stack heavily in other projects and our infra in
general.

On Tue, Jun 14, 2016 at 10:29 PM, Jonathan Haddad  wrote:

> OpsCenter going forward is limited to datastax enterprise versions.
>
> I know a lot of people like DataDog, but I haven't used it.  Maybe other
> people on the list can speak from recent first hand experience on it's pros
> and cons.
>
>
> On Tue, Jun 14, 2016 at 1:20 PM Arun Ramakrishnan <
> sinchronized.a...@gmail.com> wrote:
>
>> Thanks Jonathan.
>>
>> Out of curiosity, does opscenter support some later version of cassandra
>> that is not OSS ?
>>
>> Well, the most minimal requirement is that, I want to be able to monitor
>> for cluster health and hook this info to some alerting platform. We are AWS
>> heavy. We just really heavily on AWS cloud watch for our metrics as of now.
>> We prefer to not spend our time setting up additional tools if we can help
>> it. So, if we needed a 3rd party service we would consider an APM or
>> monitoring service that is on the cheaper side.
>>
>>
>>
>>
>> On Tue, Jun 14, 2016 at 12:20 PM, Jonathan Haddad 
>> wrote:
>>
>>> Depends what you want to monitor.  I wouldn't use a lesser version of
>>> Cassandra for OpsCenter, it doesn't give you a ton you can't get elsewhere
>>> and it's not ever going to support OSS > 2.1, so you kind of limit yourself
>>> to a pretty old version of Cassandra for a non-good reason.
>>>
>>> What else do you use for monitoring in your infra?  I've used a mix of
>>> OSS tools (nagios, statsd, graphite, ELK), and hosted solutions. The nice
>>> part about them is that you can monitor your whole stack in a single UI not
>>> just your database.
>>>
>>> On Tue, Jun 14, 2016 at 12:10 PM Arun Ramakrishnan <
>>> sinchronized.a...@gmail.com> wrote:
>>>
 What are the options for a very small and nimble startup to do keep a
 cassandra cluster running well oiled. We are on AWS. We are interested in a
 monitoring tool and potentially also cluster management tools.

 We are currently on apache cassandra 3.7. We were hoping the datastax
 opscenter would be it (It is free for startups our size). But, looks like
 it does not support cassandra versions greater than v2.1. It is pretty
 surprising considering cassandra v2.1  came out in 2014.

 We would consider downgrading to datastax cassandra 2.1 just to have
 robust monitoring tools. But, I am not sure if having opscenter offsets all
 the improvements that have been added to cassandra since 2.1.

 Sematext has a integrations for monitoring cassandra. Does anyone have
 good experience with it ?

 How much work would be involved to setup Ganglia or some such option
 for cassandra ?

 Thanks,
 Arun







>>


-- 
BR,
Michał Łowicki

Re: Cassandra monitoring

[Jonathan Haddad]

OpsCenter going forward is limited to datastax enterprise versions.

I know a lot of people like DataDog, but I haven't used it.  Maybe other
people on the list can speak from recent first hand experience on it's pros
and cons.


On Tue, Jun 14, 2016 at 1:20 PM Arun Ramakrishnan <
sinchronized.a...@gmail.com> wrote:

> Thanks Jonathan.
>
> Out of curiosity, does opscenter support some later version of cassandra
> that is not OSS ?
>
> Well, the most minimal requirement is that, I want to be able to monitor
> for cluster health and hook this info to some alerting platform. We are AWS
> heavy. We just really heavily on AWS cloud watch for our metrics as of now.
> We prefer to not spend our time setting up additional tools if we can help
> it. So, if we needed a 3rd party service we would consider an APM or
> monitoring service that is on the cheaper side.
>
>
>
>
> On Tue, Jun 14, 2016 at 12:20 PM, Jonathan Haddad 
> wrote:
>
>> Depends what you want to monitor.  I wouldn't use a lesser version of
>> Cassandra for OpsCenter, it doesn't give you a ton you can't get elsewhere
>> and it's not ever going to support OSS > 2.1, so you kind of limit yourself
>> to a pretty old version of Cassandra for a non-good reason.
>>
>> What else do you use for monitoring in your infra?  I've used a mix of
>> OSS tools (nagios, statsd, graphite, ELK), and hosted solutions. The nice
>> part about them is that you can monitor your whole stack in a single UI not
>> just your database.
>>
>> On Tue, Jun 14, 2016 at 12:10 PM Arun Ramakrishnan <
>> sinchronized.a...@gmail.com> wrote:
>>
>>> What are the options for a very small and nimble startup to do keep a
>>> cassandra cluster running well oiled. We are on AWS. We are interested in a
>>> monitoring tool and potentially also cluster management tools.
>>>
>>> We are currently on apache cassandra 3.7. We were hoping the datastax
>>> opscenter would be it (It is free for startups our size). But, looks like
>>> it does not support cassandra versions greater than v2.1. It is pretty
>>> surprising considering cassandra v2.1  came out in 2014.
>>>
>>> We would consider downgrading to datastax cassandra 2.1 just to have
>>> robust monitoring tools. But, I am not sure if having opscenter offsets all
>>> the improvements that have been added to cassandra since 2.1.
>>>
>>> Sematext has a integrations for monitoring cassandra. Does anyone have
>>> good experience with it ?
>>>
>>> How much work would be involved to setup Ganglia or some such option for
>>> cassandra ?
>>>
>>> Thanks,
>>> Arun
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>

Re: Cassandra monitoring

[Arun Ramakrishnan]

Thanks Jonathan.

Out of curiosity, does opscenter support some later version of cassandra
that is not OSS ?

Well, the most minimal requirement is that, I want to be able to monitor
for cluster health and hook this info to some alerting platform. We are AWS
heavy. We just really heavily on AWS cloud watch for our metrics as of now.
We prefer to not spend our time setting up additional tools if we can help
it. So, if we needed a 3rd party service we would consider an APM or
monitoring service that is on the cheaper side.




On Tue, Jun 14, 2016 at 12:20 PM, Jonathan Haddad  wrote:

> Depends what you want to monitor.  I wouldn't use a lesser version of
> Cassandra for OpsCenter, it doesn't give you a ton you can't get elsewhere
> and it's not ever going to support OSS > 2.1, so you kind of limit yourself
> to a pretty old version of Cassandra for a non-good reason.
>
> What else do you use for monitoring in your infra?  I've used a mix of OSS
> tools (nagios, statsd, graphite, ELK), and hosted solutions. The nice part
> about them is that you can monitor your whole stack in a single UI not just
> your database.
>
> On Tue, Jun 14, 2016 at 12:10 PM Arun Ramakrishnan <
> sinchronized.a...@gmail.com> wrote:
>
>> What are the options for a very small and nimble startup to do keep a
>> cassandra cluster running well oiled. We are on AWS. We are interested in a
>> monitoring tool and potentially also cluster management tools.
>>
>> We are currently on apache cassandra 3.7. We were hoping the datastax
>> opscenter would be it (It is free for startups our size). But, looks like
>> it does not support cassandra versions greater than v2.1. It is pretty
>> surprising considering cassandra v2.1  came out in 2014.
>>
>> We would consider downgrading to datastax cassandra 2.1 just to have
>> robust monitoring tools. But, I am not sure if having opscenter offsets all
>> the improvements that have been added to cassandra since 2.1.
>>
>> Sematext has a integrations for monitoring cassandra. Does anyone have
>> good experience with it ?
>>
>> How much work would be involved to setup Ganglia or some such option for
>> cassandra ?
>>
>> Thanks,
>> Arun
>>
>>
>>
>>
>>
>>
>>

Re: Cassandra monitoring

[Jonathan Haddad]

Depends what you want to monitor.  I wouldn't use a lesser version of
Cassandra for OpsCenter, it doesn't give you a ton you can't get elsewhere
and it's not ever going to support OSS > 2.1, so you kind of limit yourself
to a pretty old version of Cassandra for a non-good reason.

What else do you use for monitoring in your infra?  I've used a mix of OSS
tools (nagios, statsd, graphite, ELK), and hosted solutions. The nice part
about them is that you can monitor your whole stack in a single UI not just
your database.

On Tue, Jun 14, 2016 at 12:10 PM Arun Ramakrishnan <
sinchronized.a...@gmail.com> wrote:

> What are the options for a very small and nimble startup to do keep a
> cassandra cluster running well oiled. We are on AWS. We are interested in a
> monitoring tool and potentially also cluster management tools.
>
> We are currently on apache cassandra 3.7. We were hoping the datastax
> opscenter would be it (It is free for startups our size). But, looks like
> it does not support cassandra versions greater than v2.1. It is pretty
> surprising considering cassandra v2.1  came out in 2014.
>
> We would consider downgrading to datastax cassandra 2.1 just to have
> robust monitoring tools. But, I am not sure if having opscenter offsets all
> the improvements that have been added to cassandra since 2.1.
>
> Sematext has a integrations for monitoring cassandra. Does anyone have
> good experience with it ?
>
> How much work would be involved to setup Ganglia or some such option for
> cassandra ?
>
> Thanks,
> Arun
>
>
>
>
>
>
>

Cassandra monitoring

[Arun Ramakrishnan]

What are the options for a very small and nimble startup to do keep a
cassandra cluster running well oiled. We are on AWS. We are interested in a
monitoring tool and potentially also cluster management tools.

We are currently on apache cassandra 3.7. We were hoping the datastax
opscenter would be it (It is free for startups our size). But, looks like
it does not support cassandra versions greater than v2.1. It is pretty
surprising considering cassandra v2.1  came out in 2014.

We would consider downgrading to datastax cassandra 2.1 just to have robust
monitoring tools. But, I am not sure if having opscenter offsets all the
improvements that have been added to cassandra since 2.1.

Sematext has a integrations for monitoring cassandra. Does anyone have good
experience with it ?

How much work would be involved to setup Ganglia or some such option for
cassandra ?

Thanks,
Arun

ANN: Cassandra monitoring anomaly detection

[Otis Gospodnetic]

Hi,

We've added Cassandra performance monitoring and anomaly detection to SPM
http://sematext.com/spm/ recently.  Here's the announcement with several
screenshots:

http://blog.sematext.com/2014/06/02/announcement-cassandra-performance-monitoring-in-spm/

Storm, Kafka, Elasticsearch, Hadoop... monitoring is in there, too, in case
you use that with Cassandra.

Actually, this may be of interest if you are running the
Kafka+Storm+Cassandra (or Kafka+Storm+Elasticsearch) combo:
http://blog.sematext.com/2014/06/11/monitoring-kafka-storm-and-cassandra-together-with-spm/

Otis

Re: cassandra monitoring

[Nick Bailey]

 Install Errored: Failure installing agent on beta.jokefire.com. Error
 output: /var/lib/opscenter/ssl/agentKeyStore.pem: No such file or directory
 Exit code: 1


This indicates that there was a problem generating ssl files when OpsCenter
first started up. I would check the log around the first time you started
opscenter for errors. Another option would be to disable ssl communication
between OpsCenter and the agents.

http://www.datastax.com/documentation/opscenter/4.0/webhelp/index.html#opsc/configure/opscConfigSSL_g.html



 I was wondering where I could go from here. Also I would like to password
 protect my OpsCenter installation (assuming I can ever get any useful data
 into it). Are there any docs on how I can do that?


http://www.datastax.com/documentation/opscenter/4.0/webhelp/index.html#opsc/configure/opscConfigureUserAccess_c.html#opscConfigureUserAccess

Re: cassandra monitoring

[Aaron Morton]

  JMX is doing it's thing on the cassandra node and is running on port 8081
Have you set the JMX port for the cluster in Ops Centre ? The default JMX port 
has been 7199 for a while.

Off the top of the my head it’s in the same area where you specify the initial 
nodes in the cluster, maybe behind an “Advanced” button. 

The Ops Centre agent talks to the server to find out what JMX port it should 
use to talk to the local Cassandra install. 

Also check the logs in /var/log/datastax 

Cheers

-
Aaron Morton
New Zealand
@aaronmorton

Co-Founder  Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 30/12/2013, at 2:21 am, Tim Dunphy bluethu...@gmail.com wrote:

 Hi all,
 
 I'm attempting to configure datastax agent so that opscenter can monitor 
 cassandra. I am running cassandra 2.0.3 and opscenter-4.0.1-2.noarch running. 
 Cassandra is running on a centos 5.9 host and the opscenter host is running 
 on centos 6.5
 
 A ps shows the agent running
 
 [root@beta:~] #ps -ef | grep datastax | grep -v grep 
 root  2166 1  0 03:31 ?00:00:00 /bin/bash 
 /usr/share/datastax-agent/bin/datastax_agent_monitor
 106   2187 1  0 03:31 ?00:01:37 
 /etc/alternatives/javahome/bin/java -Xmx40M -Xms40M 
 -Djavax.net.ssl.trustStore=/var/lib/datastax-agent/ssl/agentKeyStore 
 -Djavax.net.ssl.keyStore=/var/lib/datastax-agent/ssl/agentKeyStore 
 -Djavax.net.ssl.keyStorePassword=opscenter 
 -Dagent-pidfile=/var/run/datastax-agent/datastax-agent.pid 
 -Dlog4j.configuration=/etc/datastax-agent/log4j.properties -jar 
 datastax-agent-4.0.2-standalone.jar /var/lib/datastax-agent/conf/address.yaml
 
 And the service itself claims that it is running:
 
 [root@beta:~] #service datastax-agent status 
 datastax-agent (pid  2187) is running...
 
 On the cassandra node I have ports 61620 and 61621 open on the firewall.
 
 But if I do an lsof and look for those ports I see no activity there.
 
 [root@beta:~] #lsof -i :61620 
 [root@beta:~] #lsof -i :61621
 
 And a netstat turns up nothing either:
 [root@beta:~] #netstat -tapn | egrep (datastax|ops)
 
 
 So I guess it should come as no surprise that the opscenter interface reports 
 the node as down.
 
 And trying to reinstall the agent remotely by clicking the 'fix' link errors 
 out:
 
 g is null
 
 If you need to make changes, you can press Retry and the installations will 
 be retried.
 
 And also I got on another attempt:
 
 Cannot call method 'getRequstStatus' of null. 
 
 I'm really wondering why I'm doing wrong here, and how I can work my way out 
 of this quagmire. It would be beyond awesome to actually get this working!
 
 I've also attempted to get Cassandra Cluster Admin working. JMX is doing it's 
 thing on the cassandra node and is running on port 8081. CCA is running on 
 the same host as the opscenter.
 
 But cca gives me this error once I log in:
 
 Cassandra Cluster Admin
 
 Logout
 
 Fatal error: Uncaught exception 'TTransportException' with message 'TSocket: 
 timed out reading 4 bytes from beta.jokefire.com:9160' in 
 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php:268 
 Stack trace: #0 
 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87): 
 TSocket-read(4) #1 
 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(135):
  TTransport-readAll(4) #2 
 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(102):
  TFramedTransport-readFrame() #3 
 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87): 
 TFramedTransport-read(4) #4 
 /var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(300):
  TTransport-readAll(4) #5 
 /var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(192):
  TBinaryProtocol-readI32(NULL) #6 
 /var/www/Cassandra-Cluster-Admin/include/thrift/packages/cassandra/cassandra.Cassandra.client.php(1017):
  TBinaryProtocol-readMessageBegin(NULL, 0, 0) # in 
 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php on line 
 268
 
 Any advice I could get on my CCA problem and /or my Opcenter problem would be 
 great and appreciated.
 
 Thanks
 Tim
 
 -- 
 GPG me!!
 
 gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
 

Re: cassandra monitoring

[Timothy P. Dunphy]

Hi Aaron, 

You were right. JMX is running on port 7199, it's just the web interface that's 
on 8081. My mistake. But what I did was to delete my existing cluster and try 
to build a new cluster within opscenter and try pointing it at my existing 
cassandra node. Just one node for now, but when we go to production we plan to 
scale out. 

When I tried to install the agent with opscetner, the installation begins but 
fails with this message a few moments later: 

Install Errored: Failure installing agent on beta.jokefire.com. Error output: 
/var/lib/opscenter/ssl/agentKeyStore.pem: No such file or directory Exit code: 
1 

I was wondering where I could go from here. Also I would like to password 
protect my OpsCenter installation (assuming I can ever get any useful data into 
it). Are there any docs on how I can do that? 

Thanks 
Tim 


- Original Message -

From: Aaron Morton aa...@thelastpickle.com 
To: Cassandra User user@cassandra.apache.org 
Sent: Monday, December 30, 2013 9:19:05 PM 
Subject: Re: cassandra monitoring 




JMX is doing it's thing on the cassandra node and is running on port 8081 


Have you set the JMX port for the cluster in Ops Centre ? The default JMX port 
has been 7199 for a while. 

Off the top of the my head it’s in the same area where you specify the initial 
nodes in the cluster, maybe behind an “Advanced” button. 

The Ops Centre agent talks to the server to find out what JMX port it should 
use to talk to the local Cassandra install. 

Also check the logs in /var/log/datastax 

Cheers 

- 
Aaron Morton 
New Zealand 
@aaronmorton 

Co-Founder  Principal Consultant 
Apache Cassandra Consulting 
http://www.thelastpickle.com 

On 30/12/2013, at 2:21 am, Tim Dunphy  bluethu...@gmail.com  wrote: 


blockquote

Hi all, 

I'm attempting to configure datastax agent so that opscenter can monitor 
cassandra. I am running cassandra 2.0.3 and opscenter-4.0.1-2.noarch running. 
Cassandra is running on a centos 5.9 host and the opscenter host is running on 
centos 6.5 

A ps shows the agent running 

[root@beta:~] #ps -ef | grep datastax | grep -v grep 
root 2166 1 0 03:31 ? 00:00:00 /bin/bash 
/usr/share/datastax-agent/bin/datastax_agent_monitor 
106 2187 1 0 03:31 ? 00:01:37 /etc/alternatives/javahome/bin/java -Xmx40M 
-Xms40M -Djavax.net.ssl.trustStore=/var/lib/datastax-agent/ssl/agentKeyStore 
-Djavax.net.ssl.keyStore=/var/lib/datastax-agent/ssl/agentKeyStore 
-Djavax.net.ssl.keyStorePassword=opscenter 
-Dagent-pidfile=/var/run/datastax-agent/datastax-agent.pid 
-Dlog4j.configuration=/etc/datastax-agent/log4j.properties -jar 
datastax-agent-4.0.2-standalone.jar /var/lib/datastax-agent/conf/address.yaml 

And the service itself claims that it is running: 

[root@beta:~] #service datastax-agent status 
datastax-agent (pid 2187) is running... 

On the cassandra node I have ports 61620 and 61621 open on the firewall. 

But if I do an lsof and look for those ports I see no activity there. 

[root@beta:~] #lsof -i :61620 
[root@beta:~] #lsof -i :61621 

And a netstat turns up nothing either: 
[root@beta:~] #netstat -tapn | egrep (datastax|ops) 


So I guess it should come as no surprise that the opscenter interface reports 
the node as down. 

And trying to reinstall the agent remotely by clicking the 'fix' link errors 
out: 

g is null 

If you need to make changes, you can press Retry and the installations will 
be retried. 

And also I got on another attempt: 

Cannot call method 'getRequstStatus' of null. 

I'm really wondering why I'm doing wrong here, and how I can work my way out of 
this quagmire. It would be beyond awesome to actually get this working! 

I've also attempted to get Cassandra Cluster Admin working. JMX is doing it's 
thing on the cassandra node and is running on port 8081. CCA is running on the 
same host as the opscenter. 

But cca gives me this error once I log in: 

Cassandra Cluster Admin 
Logout 

Fatal error : Uncaught exception 'TTransportException' with message 'TSocket: 
timed out reading 4 bytes from beta.jokefire.com:9160 ' in 
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php:268 Stack 
trace: #0 
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87): 
TSocket-read(4) #1 
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(135):
 TTransport-readAll(4) #2 
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(102):
 TFramedTransport-readFrame() #3 
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87): 
TFramedTransport-read(4) #4 
/var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(300):
 TTransport-readAll(4) #5 
/var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(192):
 TBinaryProtocol-readI32(NULL) #6 
/var/www/Cassandra-Cluster-Admin/include/thrift/packages/cassandra/cassandra.Cassandra.client.php(1017):
 TBinaryProtocol-readMessageBegin(NULL, 0, 0) # in 
/var

cassandra monitoring

[Tim Dunphy]

Hi all,

I'm attempting to configure datastax agent so that opscenter can monitor
cassandra. I am running cassandra 2.0.3 and opscenter-4.0.1-2.noarch
running. Cassandra is running on a centos 5.9 host and the opscenter host
is running on centos 6.5

A ps shows the agent running

[root@beta:~] #ps -ef | grep datastax | grep -v grep
root  2166 1  0 03:31 ?00:00:00 /bin/bash
/usr/share/datastax-agent/bin/datastax_agent_monitor
106   2187 1  0 03:31 ?00:01:37
/etc/alternatives/javahome/bin/java -Xmx40M -Xms40M
-Djavax.net.ssl.trustStore=/var/lib/datastax-agent/ssl/agentKeyStore
-Djavax.net.ssl.keyStore=/var/lib/datastax-agent/ssl/agentKeyStore
-Djavax.net.ssl.keyStorePassword=opscenter
-Dagent-pidfile=/var/run/datastax-agent/datastax-agent.pid
-Dlog4j.configuration=/etc/datastax-agent/log4j.properties -jar
datastax-agent-4.0.2-standalone.jar
/var/lib/datastax-agent/conf/address.yaml

And the service itself claims that it is running:

[root@beta:~] #service datastax-agent status
datastax-agent (pid  2187) is running...

On the cassandra node I have ports 61620 and 61621 open on the firewall.

But if I do an lsof and look for those ports I see no activity there.

[root@beta:~] #lsof -i :61620
[root@beta:~] #lsof -i :61621

And a netstat turns up nothing either:
[root@beta:~] #netstat -tapn | egrep (datastax|ops)


So I guess it should come as no surprise that the opscenter interface
reports the node as down.

And trying to reinstall the agent remotely by clicking the 'fix' link
errors out:

g is null

If you need to make changes, you can press Retry and the installations
will be retried.

And also I got on another attempt:

Cannot call method 'getRequstStatus' of null.

I'm really wondering why I'm doing wrong here, and how I can work my way
out of this quagmire. It would be beyond awesome to actually get this
working!

I've also attempted to get Cassandra Cluster Admin working. JMX is doing
it's thing on the cassandra node and is running on port 8081. CCA is
running on the same host as the opscenter.

But cca gives me this error once I log in:

Cassandra Cluster Admin http://cca.jokefire.com/
Logout http://cca.jokefire.com/logout.php

*Fatal error*: Uncaught exception 'TTransportException' with message
'TSocket: timed out reading 4 bytes from beta.jokefire.com:9160' in
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php:268
Stack trace: #0
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87):
TSocket-read(4) #1
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(135):
TTransport-readAll(4) #2
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(102):
TFramedTransport-readFrame() #3
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87):
TFramedTransport-read(4) #4
/var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(300):
TTransport-readAll(4) #5
/var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(192):
TBinaryProtocol-readI32(NULL) #6
/var/www/Cassandra-Cluster-Admin/include/thrift/packages/cassandra/cassandra.Cassandra.client.php(1017):
TBinaryProtocol-readMessageBegin(NULL, 0, 0) # in
*/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php* on
line *268*

Any advice I could get on my CCA problem and /or my Opcenter problem would
be great and appreciated.

Thanks
Tim

-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Re: Cassandra Monitoring

[Edward Capriolo]

On Thu, Jul 14, 2011 at 8:58 AM, Albert Vila a...@imente.com wrote:

 Anyone has Cassandra's cacti templates for server 0.7.4+?

 On 20 December 2010 17:40, Edward Capriolo edlinuxg...@gmail.com wrote:
  On Sun, Dec 19, 2010 at 10:37 PM, Dave Viner davevi...@gmail.com
 wrote:
  Can you share the code for run_column_family_stores.sh ?
 
  On Sun, Dec 19, 2010 at 6:14 PM, Edward Capriolo edlinuxg...@gmail.com
 
  wrote:
 
  On Sun, Dec 19, 2010 at 2:01 PM, Ran Tavory ran...@gmail.com wrote:
   Mx4j is in process, same jvm, you just need to throw mx4j-tools.jar
 in
   the lib before you start Cassandra jmx-to-rest runs in a separate
 jvm.
It also has a nice useful HTML interface that you can look into any
   running host.
  
   On Sunday, December 19, 2010, Dave Viner davevi...@gmail.com
 wrote:
   How does mx4j compare with the earlier jmx-to-rest bridge listed in
 the
   operations page:
   JMX-to-REST bridge available
   at http://code.google.com/p/polarrose-jmx-rest-bridge;
  
   ThanksDave Viner
  
  
   On Sun, Dec 19, 2010 at 7:01 AM, Ran Tavory ran...@gmail.com
 wrote:
   FYI, I just added an mx4j section to the bottom of this
   page http://wiki.apache.org/cassandra/Operations
  
  
   On Sun, Dec 19, 2010 at 4:30 PM, Jonathan Ellis jbel...@gmail.com
   wrote:
   mx4j? https://issues.apache.org/jira/browse/CASSANDRA-1068
  
  
  
  
   On Sun, Dec 19, 2010 at 8:36 AM, Peter Schuller
   peter.schul...@infidyne.com wrote:
   How / what are you monitoring? Best practices someone?
  
   I recently set up monitoring using the cassandra-munin-plugins
   (https://github.com/jamesgolick/cassandra-munin-plugins). However,
 due
   to various little details that wasn't too fun to integrate properly
   with munin-node-configure and automated configuration management. A
   problem is also the starting of a JVM for each use of jmxquery,
 which
   can become a problem with many column families.
  
   I like your web server idea. Something persistent that can sit there
   and do the JMX acrobatics, and expose something more easily consumed
   for stuff like munin/zabbix/etc. It would be pretty nice to have
 that
   out of the box with Cassandra, though I expect that would be
   considered bloat. :)
  
   --
   / Peter Schuller
  
  
   --
   Jonathan Ellis
   Project Chair, Apache Cassandra
   co-founder of Riptano, the source for professional Cassandra support
   http://riptano.com
  
  
   --
   /Ran
  
  
  
  
  
   --
   /Ran
  
 
  There is a lot of overhead on your monitoring station to kick up so
  many JMX connections. There can also be nat/hostname problems for
  remote JMX.
 
  My solution is to execute JMX over nagios remote plugin executor
 (NRPE).
 
 
 command[run_column_family_stores]=/usr/lib64/nagios/plugins/run_column_family_stores.sh
  $ARG1$ $ARG2$ $ARG3$ $ARG4$ $ARG5$ $ARG6$
 
  Maybe not as fancy as a rest-jmx bridge, but solves most of the RMI
  issues involved in pulling stats over JMX,
 
 
 
  That script is just a wrapper:
 
  For example we can have our NMS directly call the JMX fetch code like
 this:
  java -cp /usr/lib64/nagios/plugins/cassandra-cacti-m6.jar
  com.jointhegrid.m6.cassandra.CFStores
  service:jmx:rmi:///jndi/rmi://host:port/jmxrmi user pass
 
 org.apache.cassandra.db:columnfamily=columnfamily,keyspace=keyspace,type=ColumnFamilyStores
 
  But as mentioned this puts a lot of pressure on the monitoring node to
  open up all these JMX connections. With NRPE I can farm the requests
  out over NRPE. Nodes end up executing their checks locally.
 
  # cat /usr/lib64/nagios/plugins/run_column_family_stores.sh
  java -cp /usr/lib64/nagios/plugins/cassandra-cacti-m6.jar
  com.jointhegrid.m6.cassandra.CFStores
  service:jmx:rmi:///jndi/rmi://${1}:${2}/jmxrmi ${3} ${4}
 
 org.apache.cassandra.db:columnfamily=${5},keyspace=${6},type=ColumnFamilyStores
 
  All the code is up here:
  http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
 
 http://www.jointhegrid.com/svn/cassandra-cacti-m6/trunk/src/com/jointhegrid/m6/cassandra/CFStores.java
 
  My main goal was to point out that you do not need REST bridges and
  embedded web servers to run JMX checks remotely.
 



 --
 Albert Vila Puig
 a...@imente.com
 iMente.com http://www.imente.com



http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp

There is some preliminary support for 0.7.X but I have not ported over all
the graphs yet. Look over the next couple of days.

Edward

[INFO] Apache Cassandra monitoring through Hyperic HQ

[Sanjay Sharma]

Sharing an useful article on Cassandra Monitoring through Hyperic HQ- 
http://www.theserverside.com/news/thread.tss?thread_id=62185

Regards,
Sanjay Sharma
Impetus



Are you exploring a Big Data Strategy ? Listen to this recorded webinar on 
Planning your Hadoop/ NoSQL projects for 2011 at 
www.impetus.com/featured_webinar?eventid=37

Follow us on www.twitter.com/impetuscalling or visit www.impetus.com to know 
more.


NOTE: This message may contain information that is confidential, proprietary, 
privileged or otherwise protected by law. The message is intended solely for 
the named addressee. If received in error, please destroy and notify the 
sender. Any use of this email is prohibited when received in error. Impetus 
does not represent, warrant and/or guarantee, that the integrity of this 
communication has been maintained nor that the communication is free of errors, 
virus, interception or interference.

Cassandra monitoring tool

[Héctor Izquierdo Seliva]

Hi everyone.

Looking for ways to monitor cassandra with zabbix I could not found
anything that was really usable, till I found mention of a nice class by
smeet. I have based my modification upon his work and now I give it back
to the community.

Here's the project url:

http://code.google.com/p/simple-cassandra-monitoring/

It allows to get statistics for any Keyspace/ColumnFamily you want. To
start it just build the jar, and launch it using as classpath your
cassandra installation lib folder.

The first parameter is the node host name. The second parameter is a
comma separated list of KS:CF values. For example:

java -cp blablabla localhost ks1:cf1,ks1:cf2.

Then point curl to http://localhost:9090/ks1/cf1 and some basic stats
will be displayed.

You can also point to http://localhost:9090/nodeinfo to get some info
about the server.

If you have any suggestion or improvement you would like to see, please
contact me and I will be glad to work on it. Right now it's a bit rough,
but it gets the job done.

Thanks for your time!

Re: Cassandra monitoring tool

[Ali Ahsan]

Thanks for sharing this info,I am getting following error,Can please be 
more specific how can i run this



java -cp 
/home/ali/apache-cassandra-0.6.3/lib/simple-cassandra-monitoring-1.0.jar 
127.0.0.1 ks1:cf1,ks1:cf2

Exception in thread main java.lang.NoClassDefFoundError: 127/0/0/1
Caused by: java.lang.ClassNotFoundException: 127.0.0.1
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: 127.0.0.1. Program will exit.



OR


java -jar 
/home/ali/apache-cassandra-0.6.3/lib/simple-cassandra-monitoring-1.0.jar  localhost 
ks1:cf1,ks1:cf2


Failed to load Main-Class manifest attribute from
/home/ali/apache-cassandra-0.6.3/lib/simple-cassandra-monitoring-1.0.jar



On 04/12/2011 07:26 PM, Héctor Izquierdo Seliva wrote:

Hi everyone.

Looking for ways to monitor cassandra with zabbix I could not found
anything that was really usable, till I found mention of a nice class by
smeet. I have based my modification upon his work and now I give it back
to the community.

Here's the project url:

http://code.google.com/p/simple-cassandra-monitoring/

It allows to get statistics for any Keyspace/ColumnFamily you want. To
start it just build the jar, and launch it using as classpath your
cassandra installation lib folder.

The first parameter is the node host name. The second parameter is a
comma separated list of KS:CF values. For example:

java -cp blablabla localhost ks1:cf1,ks1:cf2.

Then point curl to http://localhost:9090/ks1/cf1 and some basic stats
will be displayed.

You can also point to http://localhost:9090/nodeinfo to get some info
about the server.

If you have any suggestion or improvement you would like to see, please
contact me and I will be glad to work on it. Right now it's a bit rough,
but it gets the job done.

Thanks for your time!






--
S.Ali Ahsan

Senior System Engineer

e-Business (Pvt) Ltd

49-C Jail Road, Lahore, P.O. Box 676
Lahore 54000, Pakistan

Tel: +92 (0)42 3758 7140 Ext. 128

Mobile: +92 (0)345 831 8769

Fax: +92 (0)42 3758 0027

Email: ali.ah...@panasiangroup.com



www.ebusiness-pg.com

www.panasiangroup.com

Confidentiality: This e-mail and any attachments may be confidential
and/or privileged. If you are not a named recipient, please notify the
sender immediately and do not disclose the contents to another person
use it for any purpose or store or copy the information in any medium.
Internet communications cannot be guaranteed to be timely, secure, error
or virus-free. We do not accept liability for any errors or omissions.

Re: Cassandra monitoring tool

[Héctor Izquierdo Seliva]

El mar, 12-04-2011 a las 21:24 +0500, Ali Ahsan escribió:
 Thanks for sharing this info,I am getting following error,Can please be 
 more specific how can i run this
 
 
 java -cp 
 /home/ali/apache-cassandra-0.6.3/lib/simple-cassandra-monitoring-1.0.jar 
 127.0.0.1 ks1:cf1,ks1:cf2
 Exception in thread main java.lang.NoClassDefFoundError: 127/0/0/1
 Caused by: java.lang.ClassNotFoundException: 127.0.0.1
  at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
 Could not find the main class: 127.0.0.1. Program will exit.
 
 
  
  OR
 
 java -jar 
 /home/ali/apache-cassandra-0.6.3/lib/simple-cassandra-monitoring-1.0.jar  
 localhost 
 ks1:cf1,ks1:cf2
 
 Failed to load Main-Class manifest attribute from
 /home/ali/apache-cassandra-0.6.3/lib/simple-cassandra-monitoring-1.0.jar
 
 

Hi Ali. You should run it like this

java -cp /home/ali/apache-cassandra-0.6.3/lib/*
com.google.code.scm.CassandraMonitoring localhost ks1:cf1,ks2:cf2,etc

I forgot to mention it has been coded against 0.7.x, and I'm not sure it
will work on 0.6.x. I'll try to add support for both 0.6.x and the new
0.8.x version as soon as possible.

 
 On 04/12/2011 07:26 PM, Héctor Izquierdo Seliva wrote:
  Hi everyone.
 
  Looking for ways to monitor cassandra with zabbix I could not found
  anything that was really usable, till I found mention of a nice class by
  smeet. I have based my modification upon his work and now I give it back
  to the community.
 
  Here's the project url:
 
  http://code.google.com/p/simple-cassandra-monitoring/
 
  It allows to get statistics for any Keyspace/ColumnFamily you want. To
  start it just build the jar, and launch it using as classpath your
  cassandra installation lib folder.
 
  The first parameter is the node host name. The second parameter is a
  comma separated list of KS:CF values. For example:
 
  java -cp blablabla localhost ks1:cf1,ks1:cf2.
 
  Then point curl to http://localhost:9090/ks1/cf1 and some basic stats
  will be displayed.
 
  You can also point to http://localhost:9090/nodeinfo to get some info
  about the server.
 
  If you have any suggestion or improvement you would like to see, please
  contact me and I will be glad to work on it. Right now it's a bit rough,
  but it gets the job done.
 
  Thanks for your time!
 
 
 
 
 

Re: Cassandra monitoring tool

[Ali Ahsan]

On 04/12/2011 10:42 PM, Héctor Izquierdo Seliva wrote:


I forgot to mention it has been coded against 0.7.x, and I'm not sure it
will work on 0.6.x. I'll try to add support for both 0.6.x and the new
0.8.x version as soon as possible.




I think these error  is because of 0.6.3 ?



xception in thread main java.io.IOException: Failed to retrieve 
RMIServer stub: javax.naming.CommunicationException [Root exception is 
java.rmi.ConnectIOException: error during JRMP connection establishment; 
nested exception is:

java.io.EOFException]
at 
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
at 
javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
at 
com.google.code.scm.CassandraMonitoring.start(CassandraMonitoring.java:58)
at 
com.google.code.scm.CassandraMonitoring.main(CassandraMonitoring.java:190)
Caused by: javax.naming.CommunicationException [Root exception is 
java.rmi.ConnectIOException: error during JRMP connection establishment; 
nested exception is:

java.io.EOFException]
at 
com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
at 
com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)

at javax.naming.InitialContext.lookup(InitialContext.java:409)
at 
javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1902)
at 
javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1871)
at 
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:276)

... 3 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection 
establishment; nested exception is:

java.io.EOFException
at 
sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304)

at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at 
com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)

... 8 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267)
at 
sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:246)

... 12 more


--
S.Ali Ahsan

Senior System Engineer

e-Business (Pvt) Ltd

49-C Jail Road, Lahore, P.O. Box 676
Lahore 54000, Pakistan

Tel: +92 (0)42 3758 7140 Ext. 128

Mobile: +92 (0)345 831 8769

Fax: +92 (0)42 3758 0027

Email: ali.ah...@panasiangroup.com



www.ebusiness-pg.com

www.panasiangroup.com

Confidentiality: This e-mail and any attachments may be confidential
and/or privileged. If you are not a named recipient, please notify the
sender immediately and do not disclose the contents to another person
use it for any purpose or store or copy the information in any medium.
Internet communications cannot be guaranteed to be timely, secure, error
or virus-free. We do not accept liability for any errors or omissions.

Re: Cassandra monitoring tool

[Héctor Izquierdo Seliva]

I'm not sure. Are you runing it in the same host as the cassandra node?

El mar, 12-04-2011 a las 22:54 +0500, Ali Ahsan escribió:
 On 04/12/2011 10:42 PM, Héctor Izquierdo Seliva wrote:
 
  I forgot to mention it has been coded against 0.7.x, and I'm not sure it
  will work on 0.6.x. I'll try to add support for both 0.6.x and the new
  0.8.x version as soon as possible.
 
 
 
 I think these error  is because of 0.6.3 ?
 
 
 
 xception in thread main java.io.IOException: Failed to retrieve 
 RMIServer stub: javax.naming.CommunicationException [Root exception is 
 java.rmi.ConnectIOException: error during JRMP connection establishment; 
 nested exception is:
  java.io.EOFException]
  at 
 javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
  at 
 javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
  at 
 com.google.code.scm.CassandraMonitoring.start(CassandraMonitoring.java:58)
  at 
 com.google.code.scm.CassandraMonitoring.main(CassandraMonitoring.java:190)
 Caused by: javax.naming.CommunicationException [Root exception is 
 java.rmi.ConnectIOException: error during JRMP connection establishment; 
 nested exception is:
  java.io.EOFException]
  at 
 com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
  at 
 com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
  at javax.naming.InitialContext.lookup(InitialContext.java:409)
  at 
 javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1902)
  at 
 javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1871)
  at 
 javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:276)
  ... 3 more
 Caused by: java.rmi.ConnectIOException: error during JRMP connection 
 establishment; nested exception is:
  java.io.EOFException
  at 
 sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304)
  at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
  at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
  at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
  at 
 com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
  ... 8 more
 Caused by: java.io.EOFException
  at java.io.DataInputStream.readByte(DataInputStream.java:267)
  at 
 sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:246)
  ... 12 more
 
 

Re: Cassandra monitoring tool

[Ali Ahsan]

Yes same host,I will test this with my developer team  and let you know 
more on it.


On 04/12/2011 11:14 PM, Héctor Izquierdo Seliva wrote:

I'm not sure. Are you runing it in the same host as the cassandra node?



--
S.Ali Ahsan

Senior System Engineer

e-Business (Pvt) Ltd

49-C Jail Road, Lahore, P.O. Box 676
Lahore 54000, Pakistan

Tel: +92 (0)42 3758 7140 Ext. 128

Mobile: +92 (0)345 831 8769

Fax: +92 (0)42 3758 0027

Email: ali.ah...@panasiangroup.com



www.ebusiness-pg.com

www.panasiangroup.com

Confidentiality: This e-mail and any attachments may be confidential
and/or privileged. If you are not a named recipient, please notify the
sender immediately and do not disclose the contents to another person
use it for any purpose or store or copy the information in any medium.
Internet communications cannot be guaranteed to be timely, secure, error
or virus-free. We do not accept liability for any errors or omissions.

Re: Cassandra Monitoring

[Peter Schuller]

 How / what are you monitoring? Best practices someone?

I recently set up monitoring using the cassandra-munin-plugins
(https://github.com/jamesgolick/cassandra-munin-plugins). However, due
to various little details that wasn't too fun to integrate properly
with munin-node-configure and automated configuration management. A
problem is also the starting of a JVM for each use of jmxquery, which
can become a problem with many column families.

I like your web server idea. Something persistent that can sit there
and do the JMX acrobatics, and expose something more easily consumed
for stuff like munin/zabbix/etc. It would be pretty nice to have that
out of the box with Cassandra, though I expect that would be
considered bloat. :)

-- 
/ Peter Schuller

Re: Cassandra Monitoring

[Jonathan Ellis]

mx4j? https://issues.apache.org/jira/browse/CASSANDRA-1068

On Sun, Dec 19, 2010 at 8:36 AM, Peter Schuller peter.schul...@infidyne.com
 wrote:

  How / what are you monitoring? Best practices someone?

 I recently set up monitoring using the cassandra-munin-plugins
 (https://github.com/jamesgolick/cassandra-munin-plugins). However, due
 to various little details that wasn't too fun to integrate properly
 with munin-node-configure and automated configuration management. A
 problem is also the starting of a JVM for each use of jmxquery, which
 can become a problem with many column families.

 I like your web server idea. Something persistent that can sit there
 and do the JMX acrobatics, and expose something more easily consumed
 for stuff like munin/zabbix/etc. It would be pretty nice to have that
 out of the box with Cassandra, though I expect that would be
 considered bloat. :)

 --
 / Peter Schuller




-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: Cassandra Monitoring

[Ran Tavory]

FYI, I just added an mx4j section to the bottom of this page
http://wiki.apache.org/cassandra/Operations


On Sun, Dec 19, 2010 at 4:30 PM, Jonathan Ellis jbel...@gmail.com wrote:

 mx4j? https://issues.apache.org/jira/browse/CASSANDRA-1068


 On Sun, Dec 19, 2010 at 8:36 AM, Peter Schuller 
 peter.schul...@infidyne.com wrote:

  How / what are you monitoring? Best practices someone?

 I recently set up monitoring using the cassandra-munin-plugins
 (https://github.com/jamesgolick/cassandra-munin-plugins). However, due
 to various little details that wasn't too fun to integrate properly
 with munin-node-configure and automated configuration management. A
 problem is also the starting of a JVM for each use of jmxquery, which
 can become a problem with many column families.

 I like your web server idea. Something persistent that can sit there
 and do the JMX acrobatics, and expose something more easily consumed
 for stuff like munin/zabbix/etc. It would be pretty nice to have that
 out of the box with Cassandra, though I expect that would be
 considered bloat. :)

 --
 / Peter Schuller




 --
 Jonathan Ellis
 Project Chair, Apache Cassandra
 co-founder of Riptano, the source for professional Cassandra support
 http://riptano.com




-- 
/Ran

Re: Cassandra Monitoring

[Dave Viner]

How does mx4j compare with the earlier jmx-to-rest bridge listed in the
operations page:

JMX-to-REST bridge available at
http://code.google.com/p/polarrose-jmx-rest-bridge;

Thanks
Dave Viner


On Sun, Dec 19, 2010 at 7:01 AM, Ran Tavory ran...@gmail.com wrote:

 FYI, I just added an mx4j section to the bottom of this page
 http://wiki.apache.org/cassandra/Operations


 On Sun, Dec 19, 2010 at 4:30 PM, Jonathan Ellis jbel...@gmail.com wrote:

 mx4j? https://issues.apache.org/jira/browse/CASSANDRA-1068


 On Sun, Dec 19, 2010 at 8:36 AM, Peter Schuller 
 peter.schul...@infidyne.com wrote:

  How / what are you monitoring? Best practices someone?

 I recently set up monitoring using the cassandra-munin-plugins
 (https://github.com/jamesgolick/cassandra-munin-plugins). However, due
 to various little details that wasn't too fun to integrate properly
 with munin-node-configure and automated configuration management. A
 problem is also the starting of a JVM for each use of jmxquery, which
 can become a problem with many column families.

 I like your web server idea. Something persistent that can sit there
 and do the JMX acrobatics, and expose something more easily consumed
 for stuff like munin/zabbix/etc. It would be pretty nice to have that
 out of the box with Cassandra, though I expect that would be
 considered bloat. :)

 --
 / Peter Schuller




 --
 Jonathan Ellis
 Project Chair, Apache Cassandra
 co-founder of Riptano, the source for professional Cassandra support
 http://riptano.com




 --
 /Ran

Re: Cassandra Monitoring

[Adrian Cockcroft]

I'm currently working to configure AppDynamics to monitor cassandra. It
does byte-code instrumentation, so there is an agent added to the
cassandra JVM, which gives the ability to capture latency for requests and
see where the bottleneck is coming from. We have been using it on our
other Java apps. They have a free version to try it out. It doesn't track
thrift calls out of the box, but I'm encouraging AD to figure out a way to
do that, and working on a config for capturing the entry points in the
meantime.

The way the page cache works is that pages stay in memory linked to a
specific file. If you delete that file, the pages are all considered
invalid at that point, so get zero'ed out and go to the start of the free
list. So compaction creates a new file first (which is competing with
existing read traffic to try and keep its pages in memory) then removes
the old files that were being merged, so at that point there is a supply
of blank pages, but disk reads will be needed to warm up the cache again.
The use case that I'm working with is more like a persistent memcached
replacement, so we are trying to have more RAM than data on m2.4xl EC2
instances (~70GB) and keep all reads in memory all the time.

Adrian

On 12/19/10 5:36 AM, Peter Schuller peter.schul...@infidyne.com wrote:

 How / what are you monitoring? Best practices someone?

I recently set up monitoring using the cassandra-munin-plugins
(https://github.com/jamesgolick/cassandra-munin-plugins). However, due
to various little details that wasn't too fun to integrate properly
with munin-node-configure and automated configuration management. A
problem is also the starting of a JVM for each use of jmxquery, which
can become a problem with many column families.

I like your web server idea. Something persistent that can sit there
and do the JMX acrobatics, and expose something more easily consumed
for stuff like munin/zabbix/etc. It would be pretty nice to have that
out of the box with Cassandra, though I expect that would be
considered bloat. :)

-- 
/ Peter Schuller

Re: Cassandra Monitoring

[Dave Viner]

Can you share the code for run_column_family_stores.sh ?

On Sun, Dec 19, 2010 at 6:14 PM, Edward Capriolo edlinuxg...@gmail.comwrote:

 On Sun, Dec 19, 2010 at 2:01 PM, Ran Tavory ran...@gmail.com wrote:
  Mx4j is in process, same jvm, you just need to throw mx4j-tools.jar in
  the lib before you start Cassandra jmx-to-rest runs in a separate jvm.
   It also has a nice useful HTML interface that you can look into any
  running host.
 
  On Sunday, December 19, 2010, Dave Viner davevi...@gmail.com wrote:
  How does mx4j compare with the earlier jmx-to-rest bridge listed in the
 operations page:
  JMX-to-REST bridge available at
 http://code.google.com/p/polarrose-jmx-rest-bridge;
 
  ThanksDave Viner
 
 
  On Sun, Dec 19, 2010 at 7:01 AM, Ran Tavory ran...@gmail.com wrote:
  FYI, I just added an mx4j section to the bottom of this page
 http://wiki.apache.org/cassandra/Operations
 
 
  On Sun, Dec 19, 2010 at 4:30 PM, Jonathan Ellis jbel...@gmail.com
 wrote:
  mx4j? https://issues.apache.org/jira/browse/CASSANDRA-1068
 
 
 
 
  On Sun, Dec 19, 2010 at 8:36 AM, Peter Schuller 
 peter.schul...@infidyne.com wrote:
  How / what are you monitoring? Best practices someone?
 
  I recently set up monitoring using the cassandra-munin-plugins
  (https://github.com/jamesgolick/cassandra-munin-plugins). However, due
  to various little details that wasn't too fun to integrate properly
  with munin-node-configure and automated configuration management. A
  problem is also the starting of a JVM for each use of jmxquery, which
  can become a problem with many column families.
 
  I like your web server idea. Something persistent that can sit there
  and do the JMX acrobatics, and expose something more easily consumed
  for stuff like munin/zabbix/etc. It would be pretty nice to have that
  out of the box with Cassandra, though I expect that would be
  considered bloat. :)
 
  --
  / Peter Schuller
 
 
  --
  Jonathan Ellis
  Project Chair, Apache Cassandra
  co-founder of Riptano, the source for professional Cassandra support
  http://riptano.com
 
 
  --
  /Ran
 
 
 
 
 
  --
  /Ran
 

 There is a lot of overhead on your monitoring station to kick up so
 many JMX connections. There can also be nat/hostname problems for
 remote JMX.

 My solution is to execute JMX over nagios remote plugin executor (NRPE).

 command[run_column_family_stores]=/usr/lib64/nagios/plugins/run_column_family_stores.sh
 $ARG1$ $ARG2$ $ARG3$ $ARG4$ $ARG5$ $ARG6$

 Maybe not as fancy as a rest-jmx bridge, but solves most of the RMI
 issues involved in pulling stats over JMX,

Re: Cassandra Monitoring

[Edward Capriolo]

On Fri, Dec 17, 2010 at 5:48 AM, Daniel Doubleday
daniel.double...@gmx.net wrote:
 Hi all
 just wanted to share a simple way we use to monitor cassandra internals with
 zabbix.
 We use a minimal http server which reads jmx and shows returns them in a
 property form. Thats read by zabbix every 30secs.
 That's started together with cassandra:
 https://gist.github.com/744761
 Output looks something like:
 d...@caladan[~]$ curl http://b22:9090/jmxexport
 OperationMode=Normal
 Load=151.379
 ReadOperations=506334
 WriteOperations=865867
 TotalReadLatencyMicros=6663882635
 TotalWriteLatencyMicros=352292885
 BytesCompacted=0
 BytesTotalInProgress=0
 PendingTasks=0
 HeapUsed=1153810280
 How / what are you monitoring? Best practices someone?
 Cheers,
 Daniel Doubleday,
 smeet.com, Berlin

Using cacti and -  http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
Many people are using munin good support there.

Best Bractices:
Monitor SSTable sizes and growth.
Monitor Reads/Write sec
Monitor Cache hit rate
Monitor Compactions (what % of the day and average node is compacting)
Monitor SSTable count (make sure you do not have to many)
Monitor IO wait. (make sure you are not disk bound)
Monitor JVM memory (make sure you have some overhead for bursts of traffic)

Re: Cassandra Monitoring

[Dan Kuebrich]

Is anyone using cassandra with monit?  All I have is this embarrassing bit
of monit config:

check process cassandra with pidfile /var/run/cassandra.pid
  start program = /etc/init.d/cassandra start with timeout 60 seconds
  stop program  = /etc/init.d/cassandra stop
  if failed port 9160 type tcp
 with timeout 15 seconds
 then restart
  if 3 restarts within 5 cycles then timeout
  group server

I'm sure there's some good numbers available via JMX to alert on as well but
I'm not sure best way to poll it from monit.  Comments/contributions
appreciated.

dan

On Fri, Dec 17, 2010 at 11:03 AM, Edward Capriolo edlinuxg...@gmail.comwrote:

 On Fri, Dec 17, 2010 at 5:48 AM, Daniel Doubleday
 daniel.double...@gmx.net wrote:
  Hi all
  just wanted to share a simple way we use to monitor cassandra internals
 with
  zabbix.
  We use a minimal http server which reads jmx and shows returns them in a
  property form. Thats read by zabbix every 30secs.
  That's started together with cassandra:
  https://gist.github.com/744761
  Output looks something like:
  d...@caladan[~]$ curl http://b22:9090/jmxexport
  OperationMode=Normal
  Load=151.379
  ReadOperations=506334
  WriteOperations=865867
  TotalReadLatencyMicros=6663882635
  TotalWriteLatencyMicros=352292885
  BytesCompacted=0
  BytesTotalInProgress=0
  PendingTasks=0
  HeapUsed=1153810280
  How / what are you monitoring? Best practices someone?
  Cheers,
  Daniel Doubleday,
  smeet.com, Berlin

 Using cacti and - 
 http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
 Many people are using munin good support there.

 Best Bractices:
 Monitor SSTable sizes and growth.
 Monitor Reads/Write sec
 Monitor Cache hit rate
 Monitor Compactions (what % of the day and average node is compacting)
 Monitor SSTable count (make sure you do not have to many)
 Monitor IO wait. (make sure you are not disk bound)
 Monitor JVM memory (make sure you have some overhead for bursts of traffic)

Re: cassandra monitoring

[hoivan]

Hello Daniel,

Just as an FYI, Evident ClearStone supports monitoring of Cassandra
clusters. Our product supports various NoSQL products (including Cassandra)
and other distributed caching technologies. 

The product aggregates metrics (from JMX) across all the nodes in the
cluster. We also monitor JVM stats like CPU, heap, and GC. The user
interface is an Adobe Flex application. We have also include some of the
nodetool operations within the product to allow you to point an click on a
node to run some of the nodetool functions. There's many more features in
the product such as alerting and historical reporting.

You can find more information about Evident ClearStone here:
http://www.evidentsoftware.com/products/clearstone-for-cassandra

Free downloads are available here: http://www.evidentsoftware.com/download

-- 
View this message in context: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/cassandra-monitoring-tp4924497p5698743.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
Nabble.com.

cassandra monitoring

[Simeonov, Daniel]

Hi,
   What is the preferred way of monitoring Cassandra clusters? Is Cassandra 
integrated with Ganglia? Thank you very much!
Best regards, Daniel.

Re: cassandra monitoring

[Jonathan Ellis]

Anything that can consume JMX.

On Mon, Apr 19, 2010 at 5:34 AM, Simeonov, Daniel
daniel.simeo...@sap.com wrote:
 Hi,
    What is the preferred way of monitoring Cassandra clusters? Is Cassandra
 integrated with Ganglia? Thank you very much!
 Best regards, Daniel.