cassandra monitoring

2013-12-29 Thread Tim Dunphy
Hi all,

I'm attempting to configure datastax agent so that opscenter can monitor
cassandra. I am running cassandra 2.0.3 and opscenter-4.0.1-2.noarch
running. Cassandra is running on a centos 5.9 host and the opscenter host
is running on centos 6.5

A ps shows the agent running

[root@beta:~] #ps -ef | grep datastax | grep -v grep
root  2166 1  0 03:31 ?00:00:00 /bin/bash
/usr/share/datastax-agent/bin/datastax_agent_monitor
106   2187 1  0 03:31 ?00:01:37
/etc/alternatives/javahome/bin/java -Xmx40M -Xms40M
-Djavax.net.ssl.trustStore=/var/lib/datastax-agent/ssl/agentKeyStore
-Djavax.net.ssl.keyStore=/var/lib/datastax-agent/ssl/agentKeyStore
-Djavax.net.ssl.keyStorePassword=opscenter
-Dagent-pidfile=/var/run/datastax-agent/datastax-agent.pid
-Dlog4j.configuration=/etc/datastax-agent/log4j.properties -jar
datastax-agent-4.0.2-standalone.jar
/var/lib/datastax-agent/conf/address.yaml

And the service itself claims that it is running:

[root@beta:~] #service datastax-agent status
datastax-agent (pid  2187) is running...

On the cassandra node I have ports 61620 and 61621 open on the firewall.

But if I do an lsof and look for those ports I see no activity there.

[root@beta:~] #lsof -i :61620
[root@beta:~] #lsof -i :61621

And a netstat turns up nothing either:
[root@beta:~] #netstat -tapn | egrep (datastax|ops)


So I guess it should come as no surprise that the opscenter interface
reports the node as down.

And trying to reinstall the agent remotely by clicking the 'fix' link
errors out:

g is null

If you need to make changes, you can press Retry and the installations
will be retried.

And also I got on another attempt:

Cannot call method 'getRequstStatus' of null.

I'm really wondering why I'm doing wrong here, and how I can work my way
out of this quagmire. It would be beyond awesome to actually get this
working!

I've also attempted to get Cassandra Cluster Admin working. JMX is doing
it's thing on the cassandra node and is running on port 8081. CCA is
running on the same host as the opscenter.

But cca gives me this error once I log in:

Cassandra Cluster Admin http://cca.jokefire.com/
Logout http://cca.jokefire.com/logout.php

*Fatal error*: Uncaught exception 'TTransportException' with message
'TSocket: timed out reading 4 bytes from beta.jokefire.com:9160' in
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php:268
Stack trace: #0
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87):
TSocket-read(4) #1
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(135):
TTransport-readAll(4) #2
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(102):
TFramedTransport-readFrame() #3
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87):
TFramedTransport-read(4) #4
/var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(300):
TTransport-readAll(4) #5
/var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(192):
TBinaryProtocol-readI32(NULL) #6
/var/www/Cassandra-Cluster-Admin/include/thrift/packages/cassandra/cassandra.Cassandra.client.php(1017):
TBinaryProtocol-readMessageBegin(NULL, 0, 0) # in
*/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php* on
line *268*

Any advice I could get on my CCA problem and /or my Opcenter problem would
be great and appreciated.

Thanks
Tim

-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B


Re: Cassandra unit testing becoming nearly impossible: suggesting alternative.

2013-12-29 Thread Joe Stein
I updated my repo with Vagrant and bash scripts to install Cassandra 2.0.3
https://github.com/stealthly/scala-cassandra/

0) git clone https://github.com/stealthly/scala-cassandra
1) cd scala-cassandra
2) vagrant up

Cassandra will be running in the virtual machine on 172.16.7.2 and is
accessible from your host machine (cqlsh, your app, whatever).

To verify step 3 would be ./sbt test just to make sure everything is
running right.

Everyone time you rebuild the VM (takes a minute or two) it is a whole new
instance.  If you fork foreground you have to worry about data and that not
isolated and other stuff.

On Fri, Dec 27, 2013 at 10:48 PM, Edward Capriolo edlinuxg...@gmail.comwrote:

 I think i will invest the time launching cassandra in a forked forground
 process, maybe building the yaml dynamically.

 On Friday, December 27, 2013, Nate McCall n...@thelastpickle.com wrote:
  I've also moved on to container-based (using Vagrant+docker) setup for
 doing automated integration stuff. This is more difficult to configure for
 build systems like Jenkins, but it can be done and once completed the
 benefits are substantial - as Joe notes, the most immediate is the removal
 of variance between different environments.
  However, for in process testing with Maven or similar, the Usergrid
 project [0] probably has the most functionally advanced test architecture
 [1]. Do understand that it took us a very long time to get there and
 involves some fairly tight integration with JUnit and (to a lesser degree)
 maven.
  The UG plumbing is purpose built towards a specific data model so it's
 not something that can be just dropped in, but it can be pulled apart in a
 straight forward way (provided you understand JUnit - which is not really
 trivial) and generalized pretty easily. It's all ASF-licensed, so take what
 you need if you find it useful.
  [0] https://usergrid.incubator.apache.org/
  [1]
 https://github.com/usergrid/usergrid/blob/master/stack/test-utils/src/main/java/org/usergrid/cassandra/CassandraResource.java
 
  On Wed, Dec 25, 2013 at 2:42 PM, Joe Stein crypt...@gmail.com wrote:
 
  I have been using vagrant (e.g.
 https://github.com/stealthly/scala-cassandra/ ) which is 100%
 reproducible across devs and test systems (prod in some cases).  Also have
 a Docker setup too https://github.com/pegasussolutions/docker-cassandra .
  I have been doing this more and more with clients to better mimic
 production before production and smoothing the release process from
 development.  I also use packer (scripts released soon) to build images too
 (http://packer.io)
  Love vagrant, packer and docker!!!  Apache Mesos too :)
 
 
  /***
   Joe Stein
   Founder, Principal Consultant
   Big Data Open Source Security LLC
   http://www.stealth.ly
   Twitter: @allthingshadoop
  /
 
  On Dec 25, 2013, at 3:28 PM, horschi hors...@gmail.com wrote:
 
  Hi Ed,
 
  my opinion on unit testing with C* is: Use the real database, not any
 embedded crap :-)
 
  All you need are fast truncates, by which I mean:
  JVM_OPTS=$JVM_OPTS -Dcassandra.unsafesystem=true
  and
  auto_snapshot: false
 
  This setup works really nice for me (C* 1.1 and 1.2, have not tested 2.0
 yet).
 
  Imho this setup is better for multiple reasons:
  - No extra classpath issues
  - Faster: Running JUnits and C* in one JVM would require a really large
 heap (for me at least).
  - Faster: No Cassandra startup everytime I run my tests.
 
  The only downside is that developers must change the properties in their
 configs.
 
  cheers,
  Christian
 
 
 
  On Tue, Dec 24, 2013 at 9:31 PM, Edward Capriolo edlinuxg...@gmail.com
 wrote:
 
  I am not sure there how many people have been around developing
 Cassandra for as long as I have, but the state of all the client libraries
 and the cassandra server is WORD_I_DONT_WANT_TO_SAY.
  Here is an example of something I am seeing:
  ERROR 14:59:45,845 Exception in thread Thread[Thrift:5,5,main]
  java.lang.AbstractMethodError:
 org.apache.thrift.ProcessFunction.isOneway()Z
  at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:51)
  at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
  at
 org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:194)
  at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
  at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
  at java.lang.Thread.run(Thread.java:722)
  DEBUG 14:59:51,654 retryPolicy for schema_triggers is 0.99
  In short: If you are new to cassandra and only using the newest client I
 am sure everything is peachy for you.
  For people that have been using Cassandra for a while it is harder to
 jump ship when something better comes along. You need sometimes to
 support both hector and astyanax, it happens.
  For a while I have been using hector. Even not to use hector 

Cleanup and old files

2013-12-29 Thread David McNelis
I am currently running a cluster with 1.2.8.  One of my larger column
families on one of my nodes has keyspace-tablename-ic--Data.db with a
modify date in August.

Since august we have added several nodes (with vnodes), with the same
number of vnodes as all the existing nodes.

As a result, (we've since gone from 15 to 21 nodes), then ~32% of my data
of the original 15 nodes should have been essentially balanced out to the 6
new nodes.  (1/15 + 1/16 +  1/21).

When I run a cleanup, however, the old data files never get updated, and I
can't believe that they all should have remained the same.

The only recently updated files in that data directory are secondary index
sstable files.  Am I doing something wrong here?  Am I thinking about this
wrong?

David