Re: backup data with Cass-operator and Medusa

[John Sanda]

Hi Manu

I have started doing the preliminary work. I have a PR[1] for Medusa that
needs to be reviewed/merged. I have done some work[2] for initial,
low-level integration directly against statefulsets. I hope to have medusa
integration land within the next couple releases of cass-operator.

[1] https://github.com/thelastpickle/cassandra-medusa/pull/141
[2] https://github.com/thelastpickle/medusa-operator

Cheers

John

On Fri, Oct 9, 2020 at 12:04 PM Manu Chadha  wrote:

> Hi
>
>
>
> By when the Medusa integration for backing up C* using Cass-Operator would
> be available? Found this discussion -
> https://community.datastax.com/questions/6364/triggering-consistent-backups.html
>
>
>
> Would this be the recommended way? Till the time it becomes available what
> other options could I consider?
>
>
>
> Thanks
>
> Manu
>
>
>
> Sent from Mail  for
> Windows 10
>
>
>


-- 

- John

Re: Connect java application to Cassandra in Kubernetes

[John Sanda]

It is worth mentioning that depending on how you configure your client
application deployment, you should not have to worry about bouncing the
driver. If you add a liveness probe for your client deployment that relies
on the driver being able to connect to the cluster, then kubernetes will
restart the client container when the liveness probe fails. And if you
configure the driver to connect via the headless service, you will get the
update endpoints.

On Thu, Aug 6, 2020 at 11:00 PM John Sanda  wrote:

> Hi Pushpendra
>
> You should use the headless service, e.g.,
>
> // Note that this code snippet is using v3.x of the driver.
> // Assume the service is deployed in namespace dev and is
> // named cassandra-service. The FQDN of the service would then
> // be cassandra-service.dev.svc.cluster.local. If your client
> // is deployed in the same namespace, you can reach it with by
> // cassandra-service without the rest.
>
> cluster = Cluster.builder
> .addContactPoint(headlessService)
> .build();
> Session session = cluster.connect();
>
> A headless service will resolve to multiple endpoints. The exact endpoints
> to which it maps will be determined by the label selector you use for your
> service. You can check this with:
>
> $ kubectl get endpoints
>
> The service will update the endpoints with any IP address changes. The
> addContactPoint method calls the following:
>
> addContactPoints(*IntetAddress.getAllByName*(headlessService));
>
> getAllByName will return all of the endpoints. If the entire C* cluster
> goes down, you will need to bounce the driver.
>
> Cheers
>
> John
>
> On Thu, Aug 6, 2020 at 4:47 AM Pushpendra Rajpoot <
> pushpendra.nh.rajp...@gmail.com> wrote:
>
>>
>> We have created a statefulset & headless service to deploy Cassandra in
>> Kubernetes. Our client is also in the same Kubernetes cluster. We have
>> identified two ways by which we can find contact point for driver in client
>> application:
>>
>>1. Use 'cassandra-headless-service-name' as contactPoints
>>2. Fetch the IPs of pods from headless-service & externalize the
>> IPs. Read these IP as contact points when initializing the connection.
>>
>>
>> So far so good. Above will work if one/some pods are restarted and their
>> IP changes. In this case, the driver will update the new IP automatically.
>>
>> *How will this work in case of complete outage (all Cassandra pods down)
>> ? If all the pods are down and they come back online with different IPs (IP
>> can change in Kubernetes), how will the application connect to Cassandra?*
>>
>> *What is the best way to connect Cassandra cluster in Kubernetes with a
>> client running in the same cluster.*
>>
>>
>> Regards,
>>
>> Pushpendra
>>
>>
>
> --
>
> - John
>


-- 

- John

Re: Connect java application to Cassandra in Kubernetes

[John Sanda]

Hi Pushpendra

You should use the headless service, e.g.,

// Note that this code snippet is using v3.x of the driver.
// Assume the service is deployed in namespace dev and is
// named cassandra-service. The FQDN of the service would then
// be cassandra-service.dev.svc.cluster.local. If your client
// is deployed in the same namespace, you can reach it with by
// cassandra-service without the rest.

cluster = Cluster.builder
.addContactPoint(headlessService)
.build();
Session session = cluster.connect();

A headless service will resolve to multiple endpoints. The exact endpoints
to which it maps will be determined by the label selector you use for your
service. You can check this with:

$ kubectl get endpoints

The service will update the endpoints with any IP address changes. The
addContactPoint method calls the following:

addContactPoints(*IntetAddress.getAllByName*(headlessService));

getAllByName will return all of the endpoints. If the entire C* cluster
goes down, you will need to bounce the driver.

Cheers

John

On Thu, Aug 6, 2020 at 4:47 AM Pushpendra Rajpoot <
pushpendra.nh.rajp...@gmail.com> wrote:

>
> We have created a statefulset & headless service to deploy Cassandra in
> Kubernetes. Our client is also in the same Kubernetes cluster. We have
> identified two ways by which we can find contact point for driver in client
> application:
>
>1. Use 'cassandra-headless-service-name' as contactPoints
>2. Fetch the IPs of pods from headless-service & externalize the  IPs.
>Read these IP as contact points when initializing the connection.
>
>
> So far so good. Above will work if one/some pods are restarted and their
> IP changes. In this case, the driver will update the new IP automatically.
>
> *How will this work in case of complete outage (all Cassandra pods down) ?
> If all the pods are down and they come back online with different IPs (IP
> can change in Kubernetes), how will the application connect to Cassandra?*
>
> *What is the best way to connect Cassandra cluster in Kubernetes with a
> client running in the same cluster.*
>
>
> Regards,
>
> Pushpendra
>
>

-- 

- John

Re: Replacing a Cassandra node in K8S

[John Sanda]

The Cassandra pod will get scheduled to run on a different worker node,
provided there is an available node that satisfies affinity rules, resource
requirements, etc. And you are correct that the volume will get remounted.
If however you are using a local or hostPath volume, then it will be lost
and you will want to restart Cassandra with the -Dcassandra.replace_address
option.

On Mon, Jul 27, 2020 at 5:17 AM manish khandelwal <
manishkhandelwa...@gmail.com> wrote:

> Hi Team
>
> I was wondering how Cassandra node is replaced if one of the worker node
> fails in k8s.  My understanding is that since PVCs are remounted to their
> volume mounts, no matter where the pods are rescheduled (any node), so
> replacing a node will not be a issue only ip will get changed.
>
>
> Regards
> Manish
>


-- 

- John

Re: What is the way to scale down Cassandra/Kubernetes cluster from 3 to 1 nodes using cass-operator

[John Sanda]

Cass Operator currently does not support scaling down.

Thanks

John

On Thu, Jul 2, 2020 at 1:02 PM Manu Chadha  wrote:

> Hi
>
>
>
> I changed the file and applied it but the new configuration hasn’t got
> applied.
>
>
>
>
>
> metadata:
>
>   name: dc1
>
> spec:
>
>   clusterName: cluster1
>
>   serverType: cassandra
>
>   serverVersion: "3.11.6"
>
>   managementApiAuth:
>
> insecure: {}
>
>   size: 1 ß made change here
>
>   storageConfig:
>
> ...
>
>
>
> kubectl apply -n cass-operator -f ./cass-dc-2-nodes.yaml
>
>
>
> manuchadha25@cloudshell:~ (copper-frame-262317)$ kubectl get all -n 
> cass-operator
>
> NAME READY   STATUSRESTARTS   AGE
>
> pod/cass-operator-5f8cdf99fc-9c5g4   1/1 Running   0  2d20h
>
> pod/cluster1-dc1-default-sts-0   2/2 Running   0  2d20h
>
> pod/cluster1-dc1-default-sts-1   2/2 Running   0  9h
>
> pod/cluster1-dc1-default-sts-2   2/2 Running   0  9h
>
>
>
> NAME  TYPE   CLUSTER-IP  
> EXTERNAL-IP PORT(S) AGE
>
> service/cass-operator-metrics ClusterIP  10.51.243.147   
>   8383/TCP,8686/TCP   2d20h
>
> service/cassandra-loadbalancerLoadBalancer   10.51.240.24
> 34.91.214.233   9042:30870/TCP  2d
>
> service/cassandradatacenter-webhook-service   ClusterIP  10.51.243.86
>   443/TCP 2d20h
>
> service/cluster1-dc1-all-pods-service ClusterIP  None
> 2d20h
>
> service/cluster1-dc1-service  ClusterIP  None
>   9042/TCP,8080/TCP   2d20h
>
> service/cluster1-seed-service ClusterIP  None
> 2d20h
>
>
>
> NAMEREADY   UP-TO-DATE   AVAILABLE   AGE
>
> deployment.apps/cass-operator   1/1 11   2d20h
>
>
>
> NAME   DESIRED   CURRENT   READY   AGE
>
> replicaset.apps/cass-operator-5f8cdf99fc   1 1 1   2d20h
>
>
>
> NAMEREADY   AGE
>
> statefulset.apps/cluster1-dc1-default-sts   3/3 2d20h ß still 3/3
>
> manuchadha25@cloudshell:~ (copper-frame-262317)$
>
>
>
> thanks
>
> Manu
>
> Sent from Mail  for
> Windows 10
>
>
>
> *From: *vishal kharjul 
> *Sent: *02 July 2020 12:46
> *To: *user@cassandra.apache.org
> *Subject: *Re: What is the way to scale down Cassandra/Kubernetes cluster
> from 3 to 1 nodes using cass-operator
>
>
>
> Hello Many,
>
>
>
> I tried scale up and it's just need size parameter change . So try same
> for scale down. Just change the size parameter of CassandraDatacenter CRD
> and apply it again. Basically same step which you took to spinoff 3 node
> with just the size parameter changed. Operator will bring down Cassandra
> nodes accordingly.  No need to shut down or restart.
>
>
>
> Thanks and Regards,
>
> Vishal
>
> On Thu, Jul 2, 2020, 3:41 AM Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
>
> On Thu, Jul 2, 2020 at 9:29 AM Manu Chadha 
> wrote:
>
> Thanks Alex. Will give this a try. So I just change the yaml file and
> hot-patch it or would I need to stop the cluster, delete it and make a new
> one?
>
>
>
> I've no experience with this specific operator, but I expect that editing
> the file and applying it using kubectl is the way to go, especially if you
> don't want to lose your data.
>
>
>
> --
>
> Alex
>
>
>
>
>


-- 

- John

Re: Question on cass-operator

[John Sanda]

Hi Manu,

The 2/2 indicates that there are two containers and each is in the ready
state. As Vishal suggested, run kubectl describe pod  to get more
details. You also use kubectl get pod  -o yaml. The former will
include events in the output. You can run nodetool commands like this:

$ kubectl -n cass-operator exec -it  -c cassandra -- nodetool status

Lastly, there is the #cassandra-kubernetes channel on ASF slack. Feel free
to drop in there with questions.

Thanks

John

On Fri, Jul 3, 2020 at 7:49 AM vishal kharjul 
wrote:

> Hello Manu,
>
> It's actually a K8 query and not Cassnadra. AFIK READY= 2/2 could
> represent a status of individual. Container in each pod. 2/2 suggests Pod
> consists of two containers and both ready. Try "kubectl describe" on each
> pod and you can see container spec. Also I will recommend getting started
> kubernetes tutorial on kubernetes.io to refresh kubernetes concepts.
>
>
> On Fri, Jul 3, 2020, 5:16 AM Manu Chadha  wrote:
>
>> Hi
>>
>>
>>
>> I have a 3 node Kubernetes cluster and I have set up Cassandra on it
>> using Cass-Operator.
>>
>>
>>
>> What does the 2/2 mean in the output of the following command
>>
>>
>>
>> kubectl get all -n cass-operator
>>
>> NAMEREADY   STATUSRESTARTS   AGE
>>
>> pod/cass-operator-78c6469c6-6qhsb   1/1 Running   0  139m
>>
>> pod/cluster1-dc1-default-sts-0  2/2 Running   0  138m
>>
>> pod/cluster1-dc1-default-sts-1  2/2 Running   0  138m
>>
>> pod/cluster1-dc1-default-sts-2  2/2 Running   0  138m
>>
>>
>>
>> Does it mean that there are 3 data centres each running 2 cassandra
>> nodes? It should be because my K8S cluster has only 3 nodes.
>>
>>
>>
>> manuchadha25@cloudshell:~ (copper-frame-262317)$ gcloud compute instances 
>> list
>>
>> NAME  ZONE
>> MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IPSTATUS
>>
>> gke-cassandra-cluster-default-pool-92d544da-6fq8  europe-west4-a  
>> n1-standard-1   10.164.0.26  34.91.214.233  RUNNING
>>
>> gke-cassandra-cluster-default-pool-92d544da-g0b5  europe-west4-a  
>> n1-standard-1   10.164.0.25  34.91.101.218  RUNNING
>>
>> gke-cassandra-cluster-default-pool-92d544da-l87v  europe-west4-a  
>> n1-standard-1   10.164.0.27  34.91.86.10RUNNING
>>
>>
>>
>> Or is Cassandra-operator running two containers per K8S Node?
>>
>>
>>
>> thanks
>>
>> Manu
>>
>>
>>
>

-- 

- John

Re: Minimum System Requirements

[John Sanda]

I recently had to set up an integration testing environment that involves
running multiple C* instances in containers using docker-compose. I am able
to do so with a total memory for the container set at 512 MB and a 256 MB
heap for C*. This is with C* 3.11.4. Going below 512 MB causes the
container to get OOM killed consistently. I do not have any CPU limits or
constraints currently set up. I am using out of the box settings for C*
configuration. I am not doing much with the nodes other than starting them
and allowing them to reach the UN state. I want to reiterate that this is
for an automated testing environment that can be memory constrained.

On Sat, Mar 28, 2020 at 4:43 AM A  wrote:

> Thank you.
>
>
> Sent from Yahoo Mail for iPhone
> 
>
> On Friday, March 27, 2020, 9:02 PM, Erick Ramirez <
> erick.rami...@datastax.com> wrote:
>
> It really depends on your definition of "stable" but you can run C* on as
> little as a single-core machine with 4-6GB of RAM. It will be stable enough
> to do 1 or 2 queries per second (or maybe a bit more) if you were just
> doing some development and wanted to do minimal testing. There are even
> examples out there where users have clusters deployed on Raspberry Pis as a
> fun project.
>
> But if you're talking about production deployments, the general
> recommendation is to run it on at least 4 core machines + 16-20GB RAM if
> you're just testing it out and only expect low traffic volumes. Otherwise,
> it is recommended to deploy it on 8-core machines with at least 32-40GB RAM
> so you have enough memory to allocate to the heap. Cheers!
>
> GOT QUESTIONS? Apache Cassandra experts from the community and DataStax
> have answers! Share your expertise on https://community.datastax.com/.
>
>

-- 

- John

Re: slurm for cluster job scheduling and coordination

[John Sanda]

>
> I've been working towards organizing an effort around using Kubernetes for
> cluster management. There is a lot of work to do but this could be
> something really important to tackle as a community if you(or anyone else)
> are interested in getting involved.
>

This is a big area of interest for me and would be more than happy to be
involved.

On Tue, Mar 10, 2020 at 12:41 PM Patrick McFadin  wrote:

> Carl,
>
> Slurm might be a nice way to keep things you already have built in some
> sort of control plane. Things already built meaning terraform, ansible,
> salt, chef,  Cassandra cluster, but that doesn't mean it's not used.
>
> I've been working towards organizing an effort around using Kubernetes for
> cluster management. There is a lot of work to do but this could be
> something really important to tackle as a community if you(or anyone else)
> are interested in getting involved.
>
> Patrick
>
> On Mon, Mar 9, 2020 at 9:34 AM Carl Mueller
>  wrote:
>
>> Between repairs, rolling restarts, scheduled maintenance bounces,
>> backups, upgrades, etc there are lots of cluster-wide tasks that would be
>> nice to be scheduled and viewed.
>>
>> Slurm appears to have some features that support this but might be
>> heavyweight considering its primary application is supercomputer job
>> scheduling.
>>
>> I'd like something kinda independent of cloud vendor, container/VM/metal
>> strategies, or specific "cloud os".
>>
>> Anyone tried slurm or have an alternative? I really don't want to write
>> yet-another-scheduler.
>>
>

-- 

- John

Re: Cassandra on Kubernetes

[John Sanda]

One of the problems I have experienced in the past has more to do with Java
than Cassandra in particular, and that is the JVM ignoring cgroups. With
Cassandra in particular I would often see memory usage go higher than what
was desired. This would lead to pods getting oom killed. This was fixed in
Java 10 though, and I believe even back ported to Java 8.

I think another issue is the lack of options for backup/restore.

I worked with Cassandra in Kubernetes pre-StatefulSets. That was a bit
rough :)

Local volumes were promoted to GA in Kubernetes 1.14. That is certainly a
good thing for stateful applications like Cassandra.

It is also important to have a sufficiently large value for the termination
grace period on pods to allow drain operation to complete, assuming you
perform a drain on shutdown.

On Wed, Oct 30, 2019 at 1:46 PM Akshit Jain  wrote:

> Hi Jean
> Thanks for replying. I had seen CassKop and the amount functionality it
> provides is quite awesome as compared to other operators.
>
> I would like to know how stable is kubernetes for stateful/database
> applications right now?
>
> I haven't read/heard any major production stateful application running on
> k8s.
>
>
> -Akshit
>
>
>
>
> On Wed, 30 Oct, 2019, 8:12 PM Jean-Armel Luce,  wrote:
>
>> Hi,
>>
>> We are currently developping CassKop, a Cassandra operator for K8S.
>> This operator is developped in Go, based on the operator-sdk framework.
>>
>> At this time of the project, the goal is to deploy a Cassandra cluster in
>> 1 Kubernetes datacenter, but this will change in next versions to deal with
>> Kubernetes in multi-datacenters.
>>
>> The following features are already supported by CassKop:
>> - Deployment of a C* cluster (rack or AZ aware)
>> - Scaling up the cluster (with cleanup)
>> - Scaling down the cluster (with decommission prior to Kubernetes scale
>> down)
>> - Pods operations (removenode, upgradesstable, cleanup, rebuild..)
>> - Adding a Cassandra DC
>> - Removing a Cassandra DC
>> - Setting and modifying configuration files
>> - Setting and modifying configuration parameters
>> - Update of the Cassandra docker image
>> - Rolling update of a Cassandra cluster
>> - Update of Cassandra version (including upgradesstable in case of major
>> upgrade)
>> - Update of JVM
>> - Update of configuration
>> - Stopping a Kubernetes node for maintenance
>> - Process a remove node (and create new Cassandra node on another
>> Kubernetes node)
>> - Process a replace address (of the old Cassandra node on another
>> Kubernetes node)
>> - Manage operations on pods through CassKop plugin (cleanup, rebuild,
>> upgradesstable, removenode..)
>> - Monitoring (using Instaclustr Prometheus exporter to Prometheus/Grafana)
>> - Pause/Restart & rolling restart operations through CassKoP plugin.
>>
>> We use also Cassandra reaper for scheduling repair sessions.
>>
>>
>> If you would like more informations about this operator, you may have a
>> look here : https://github.com/Orange-OpenSource/cassandra-k8s-operator
>>
>> Please, feel free to download it and try it. We would be more than happy
>> to receive your feedback
>>
>>
>> If you have any question about this operator, feel free to contact us via
>> our mailing-list: prj.casskop.supp...@list.orangeportails.net or on our
>> slack https://casskop.slack.com
>>
>> Note : this operator is still in alpha version and works only in a mono
>> region architecture for now. We are currently working hard for adding new
>> features in order to run it in multi-regions architecture.
>>
>>
>> Thanks.
>>
>>
>>
>> Le mer. 30 oct. 2019 à 13:56, Akshit Jain  a
>> écrit :
>>
>>> Hi everyone,
>>>
>>> Is there anyone who is running Cassandra on K8s clusters. It would be
>>> great if you can share your experience , the operator you are using and the
>>> overall stability of stateful sets in Kubernetes
>>>
>>> -Akshit
>>>
>>

-- 

- John

Re: Rebuilding a node without clients hitting it

[John Sanda]

Assuming the rebuild is happening on a node in another DC, then there
should not be an issue if you are using LOCAL_ONE. If the node is in the
local DC (i.e., same DC as the client), I am inclined to think repair would
be more appropriate than rebuild but I am not 100% certain.

On Mon, Aug 5, 2019 at 11:23 PM Jeff Jirsa  wrote:

> No, not strictly sufficient - makes it much less likely though
>
> A client may connect to another node and still send the request to that
> host if the snitch picks it. You can make THAT less likely with some snitch
> trickery (setting the badness for the rebuilding host) via jmx
>
> On Aug 5, 2019, at 8:17 PM, Cyril Scetbon  wrote:
>
> Hey guys,
>
> Can you confirm that disabling the native transport (nodetool
> disablebinary) is enough with Cassandra 3.11+ to avoid clients hitting
> inconsistent data on that node when they use LOCAL_ONE  consistency ?
> (Particularly when the node is rebuilding …)
> I'd like to avoid any fancy client configuration like blacklisting nodes.
>
> Thanks
> —
> Cyril Scetbon
>
>

-- 

- John

Re: Cassandra DataStax Java Driver in combination with Java EE / EJBs

[John Sanda]

Hi Ralph,

A session is intended to be a long-lived, i.e., application-scoped object.
You only need one session per cluster. I think what you are doing with
the @Singleton is fine. In my opinion though, EJB really does not offer
much value when working with Cassandra. I would be inclined to just use CDI.

Cheers

John

On Tue, Jun 11, 2019 at 5:38 PM Ralph Soika  wrote:

> Hi,
>
> I have a question concerning the Cassandra DataStax Java Driver in
> combination with Java EE and EJBs.
>
> I have implemented a Rest Service API based on Java EE8. In my application
> I have for example a jax-rs rest resource to write data into cassandra
> cluster. My first approach was to create in each method call
>
>1.  a new Casssandra Cluster and Session object,
>2.  write my data into cassandra
>3.  and finally close the session and the cluster object.
>
> This works but it takes a lot of time (2-3 seconds) until the cluster
> object / session is opened for each request.
>
>  So my second approach is now a @Singleton EJB providing the session
> object for my jax-rs resources. My service implementation to hold the
> Session object looks something like this:
>
>
> *@Singleton*
> *public class* ClusterService {
> private Cluster cluster;
> private Session session;
>
> @PostConstruct
> *private void* init() throws ArchiveException {
> cluster=initCluster();
> session = initArchiveSession();
> }
>
> @PreDestroy
> *private* void tearDown() throws ArchiveException {
> // close session and cluster object
> if (session != null) {
> session.close();
> }
> if (cluster != null) {
> cluster.close();
> }
> }
>
> *public* Session getSession() {
> if (session==null) {
> try {
> init();
> } catch (ArchiveException e) {
> logger.warning("unable to get falid session: " +
> e.getMessage());
> e.printStackTrace();
> }
> }
> *return* session;
> }
>
>.
>
> }
>
>
> And my rest service calls now looking like this:
>
>
> @Path("/archive")
> @Stateless
> *public class* ArchiveRestService {
>
> @EJB
> ClusterService clusterService;
>
> @POST
> @Consumes({ MediaType.APPLICATION_XML, MediaType.TEXT_XML })
> *public* Response postData(XMLDocument xmlDocument) {
> Session session = clusterService.getSession();
> session.execute();
> ...
> }
> ...
> }
>
>
> The result is now a super-fast behavior! Seems to be clear because my
> rest service no longer need to open a new session for each request.
>
> My question is: Is this approach with a @Singleton ClusterService EJB
> valid or is there something I should avoid?
> As far as I can see this works pretty fine and is really fast. I am
> running the application on a Wildfly 15 server which is Java EE8.
>
> Thanks for your comments
>
> Ralph
>
>
>
>
> --
>
> *Imixs Software Solutions GmbH*
> *Web:* www.imixs.com *Phone:* +49 (0)89-452136 16
> *Office:* Agnes-Pockels-Bogen 1, 80992 München
> Registergericht: Amtsgericht Muenchen, HRB 136045
> Geschaeftsführer: Gaby Heinle u. Ralph Soika
>
> *Imixs* is an open source company, read more: www.imixs.org
>


-- 

- John

Re: CassKop : a Cassandra operator for Kubernetes developped by Orange

[John Sanda]

There is also
https://github.com/sky-uk/cassandra-operator

On Fri, May 24, 2019 at 2:34 PM Rahul Singh 
wrote:

> Fantastic! Now there are three teams making k8s operators for C*:
> Datastax, Instaclustr, and now Orange.
>
> rahul.xavier.si...@gmail.com
>
> http://cassandra.link
>
> I'm speaking at #DataStaxAccelerate, the world’s premiere #ApacheCassandra
> conference, and I want to see you there! Use my code Singh50 for 50% off
> your registration. www.datastax.com/accelerate
>
>
> On Fri, May 24, 2019 at 9:07 AM Jean-Armel Luce 
> wrote:
>
>> Hi folks,
>>
>> We are excited to announce that CassKop, a Cassandra operator for
>> Kubernetes developped by Orange teams, is now ready for Beta testing.
>>
>> CassKop works as a usual K8S controller (reconcile the real state with a
>> desired state) and automates the Cassandra operations through JMX. All the
>> operations are launched by calling standard K8S APIs (kubectl apply …) or
>> by using a K8S plugin (kubectl casskop …).
>>
>> CassKop is developed in GO, based on CoreOS operator-sdk framework.
>> Main features already available :
>> - deploying a rack aware cluster (or AZ aware cluster)
>> - scaling up & down (including cleanups)
>> - setting and modifying configuration parameters (C* and JVM parameters)
>> - adding / removing a datacenter in Cassandra (all datacenters must be in
>> the same region)
>> - rebuilding nodes
>> - removing node or replacing node (in case of hardware failure)
>> - upgrading C* or Java versions (including upgradesstables)
>> - monitoring (using Prometheus/Grafana)
>> - ...
>>
>> By using local and persistent volumes, it is possible to handle failures
>> or stop/start nodes for maintenance operations with no transfer of data
>> between nodes.
>> Moreover, we can deploy cassandra-reaper in K8S and use it for scheduling
>> repair sessions.
>> For now, we can deploy a C* cluster only as a mono-region cluster. We
>> will work during the next weeks to be able to deploy a C* cluster as a
>> multi regions cluster.
>>
>> Still in the roadmap :
>> - Network encryption
>> - Monitoring (exporting logs and metrics)
>> - backup & restore
>> - multi-regions support
>>
>> We'd be interested to hear you try this and let us know what you think!
>>
>> Please read the description and installation instructions on
>> https://github.com/Orange-OpenSource/cassandra-k8s-operator.
>> For a quick start, you can also follow this step by step guide :
>> https://orange-opensource.github.io/cassandra-k8s-operator/index.html?slides=Slides-CassKop-demo.md#1
>>
>>
>> The CassKop Team
>>
> --

- John

Re: Commit Log sync problems

[John Sanda]

Hi Meg,

I believe that the average duration reported is the total amount of time
that exceeded the interval divided by the number of syncs that exceeded the
interval. Cassandra is not complaining because commit log syncs took 0.66
ms but rather on average 4 commit log syncs 10060 ms.

Cheers,

John

On Thu, Mar 7, 2019 at 11:33 AM Meg Mara  wrote:

> Hello all,
>
>
>
> I recently upgraded from C* 3.0.10 to 3.0.16 and have been receiving these
> warnings about Commit-Log durations being longer than the configured
> interval. I don’t understand what the problem is, why is the system
> complaining about such small sync durations? Please advice.
>
>
>
> Here are some warnings:
>
>
>
> 3:WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2019-03-07 13:32:23,785 
> NoSpamLogger.java:94 - Out of 2750 commit log syncs over the past 274s with 
> average duration of 0.66ms, 4 have exceeded the configured commit interval by 
> an average of 22.50ms
>
>
>
> 2:WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2019-03-07 13:27:01,214 
> NoSpamLogger.java:94 - Out of 1 commit log syncs over the past 0s with 
> average duration of 113.00ms, 1 have exceeded the configured commit interval 
> by an average of 13.00ms
>
>
>
>
>
> Node’s cassandra.yaml setting related to Commit-Log:
>
>
>
> commit_failure_policy: stop
>
> commitlog_directory: /cassandra/log
>
> commitlog_segment_size_in_mb: 32
>
> commitlog_sync: periodic
>
> commitlog_sync_period_in_ms: 1
>
> commitlog_total_space_in_mb: 4096
>
>
>
> Thank you,
>
> *Meg Mara*
>
>
>
>
>
-- 

- John

multiple table directories for system_schema keyspace

[John Sanda]

On a couple different occasions I have run into this exception at start up:

Exception (org.apache.cassandra.exceptions.InvalidRequestException)
encountered during startup: Unknown type 
org.apache.cassandra.exceptions.InvalidRequestException: Unknown type 
at
org.apache.cassandra.cql3.CQL3Type$Raw$RawUT.prepare(CQL3Type.java:745)
at
org.apache.cassandra.cql3.CQL3Type$Raw.prepareInternal(CQL3Type.java:533)
at
org.apache.cassandra.schema.CQLTypeParser.parse(CQLTypeParser.java:53)
at
org.apache.cassandra.schema.SchemaKeyspace.createColumnFromRow(SchemaKeyspace.java:1052)
at
org.apache.cassandra.schema.SchemaKeyspace.lambda$fetchColumns$12(SchemaKeyspace.java:1038)

This was with Cassandra 3.0.12 running in Kubernetes, which means that IP
address changes for the Cassandra node can and will happen. Nowhere in
client code does the UDT get dropped. I came across
https://issues.apache.org/jira/browse/CASSANDRA-13739 which got me
wondering if this particular Cassandra node wound up with another version
of the system_schema.types table which did not have the UDT.

In what circumstances could I end up with multiple table directories for
the tables in system_schema? Right now I am just guessing that I wound up
with a newer (or different) version of the system_schema.types table.
Unfortunately, I no longer have access to the environment to confirm/deny
what was happening. I just want to better understand so I can avoid it in
the future.


- John

Re: Understanding Blocked and All Time Blocked columns in tpstats

[John Sanda]

We do small inserts. For a modest size environment we do about 90,000
inserts every 30 seconds. For a larger environment, we could be doing
300,000 or more inserts every 30 seconds. In earlier versions of the
project, each insert was a separate request as each insert targets a
different partition. In more recent versions though, we introduced micro
batching. We batch up to about 25 inserts where inserts are grouped by
token range. Even though batches are used, I assume that does not reduce
the overall number of inserts or mutations. Inserts are always async,
prepared statements. Client code is written with RxJava which makes doing
async, concurrent writes a lot easier.

On Fri, Mar 23, 2018 at 1:29 PM, Chris Lohfink <clohf...@apple.com> wrote:

> Increasing queue would increase the number of requests waiting. It could
> make GCs worse if the requests are like large INSERTs, but for a lot of
> super tiny queries it helps to increase queue size (to a point). Might want
> to look into what and how queries are being made, since there are possibly
> options to help with that (ie prepared queries, what queries are, limiting
> number of async inflight queries)
>
> Chris
>
>
> On Mar 23, 2018, at 11:42 AM, John Sanda <john.sa...@gmail.com> wrote:
>
> Thanks for the explanation. In the past when I have run into problems
> related to CASSANDRA-11363, I have increased the queue size via the
> cassandra.max_queued_native_transport_requests system property. If I find
> that the queue is frequently at capacity, would that be an indicator that
> the node is having trouble keeping up with the load? And if so, will
> increasing the queue size just exacerbate the problem?
>
> On Fri, Mar 23, 2018 at 11:51 AM, Chris Lohfink <clohf...@apple.com>
> wrote:
>
>> It blocks the caller attempting to add the task until theres room in
>> queue, applying back pressure. It does not reject it. It mimics the
>> behavior from pre-SEP DebuggableThreadPoolExecutor's
>> RejectionExecutionHandler that the other thread pools use (exception on
>> sampling/trace which just throw away on rejections).
>>
>> Worth noting this is only really possible in the native transport pool
>> (sep pool) last I checked. Since 2.1 at least, before that there were a few
>> others. That changes version to version. For (basically) all other thread
>> pools the queue is limited by memory.
>>
>> Chris
>>
>>
>> On Mar 22, 2018, at 10:44 PM, John Sanda <john.sa...@gmail.com> wrote:
>>
>> I have been doing some work on a cluster that is impacted by
>> https://issues.apache.org/jira/browse/CASSANDRA-11363. Reading through
>> the ticket prompted me to take a closer look at
>> org.apache.cassandra.concurrent.SEPExecutor. I am looking at the 3.0.14
>> code. I am a little confused about the Blocked and All Time Blocked columns
>> reported in nodetool tpstats and reported by StatusLogger. I understand
>> that there is a queue for tasks. In the case of RequestThreadPoolExecutor,
>> the size of that queue can be controlled via the
>> cassandra.max_queued_native_transport_requests system property.
>>
>> I have been looking at SEPExecutor.addTask(FutureTask task), and here
>> is my question. If the queue is full, as defined by
>> SEPExector.maxTasksQueued, are tasks rejected? I do not fully grok the
>> code, but it looks like it is possible for tasks to be rejected here (some
>> code and comments omitted for brevity):
>>
>> public void addTask(FutureTask task)
>> {
>> tasks.add(task);
>> ...
>> else if (taskPermits >= maxTasksQueued)
>> {
>> WaitQueue.Signal s = hasRoom.register();
>>
>> if (taskPermits(permits.get()) > maxTasksQueued)
>> {
>> if (takeWorkPermit(true))
>> pool.schedule(new Work(this))
>>
>> metrics.totalBlocked.inc();
>> metrics.currentBlocked.inc();
>> s.awaitUninterruptibly();
>> metrics.currentBlocked.dec();
>> }
>> else
>> s.cancel();
>> }
>> }
>>
>> The first thing that happens is that the task is added to the tasks
>> queue. pool.schedule() only gets called if takeWorkPermit() returns true. I
>> am still studying the code, but can someone explain what exactly happens
>> when the queue is full?
>>
>>
>> - John
>>
>>
>>
>
>
> --
>
> - John
>
>
>


-- 

- John

Re: Understanding Blocked and All Time Blocked columns in tpstats

[John Sanda]

Thanks for the explanation. In the past when I have run into problems
related to CASSANDRA-11363, I have increased the queue size via the
cassandra.max_queued_native_transport_requests system property. If I find
that the queue is frequently at capacity, would that be an indicator that
the node is having trouble keeping up with the load? And if so, will
increasing the queue size just exacerbate the problem?

On Fri, Mar 23, 2018 at 11:51 AM, Chris Lohfink <clohf...@apple.com> wrote:

> It blocks the caller attempting to add the task until theres room in
> queue, applying back pressure. It does not reject it. It mimics the
> behavior from pre-SEP DebuggableThreadPoolExecutor's
> RejectionExecutionHandler that the other thread pools use (exception on
> sampling/trace which just throw away on rejections).
>
> Worth noting this is only really possible in the native transport pool
> (sep pool) last I checked. Since 2.1 at least, before that there were a few
> others. That changes version to version. For (basically) all other thread
> pools the queue is limited by memory.
>
> Chris
>
>
> On Mar 22, 2018, at 10:44 PM, John Sanda <john.sa...@gmail.com> wrote:
>
> I have been doing some work on a cluster that is impacted by
> https://issues.apache.org/jira/browse/CASSANDRA-11363. Reading through
> the ticket prompted me to take a closer look at 
> org.apache.cassandra.concurrent.SEPExecutor.
> I am looking at the 3.0.14 code. I am a little confused about the Blocked
> and All Time Blocked columns reported in nodetool tpstats and reported by
> StatusLogger. I understand that there is a queue for tasks. In the case of
> RequestThreadPoolExecutor, the size of that queue can be controlled via the
> cassandra.max_queued_native_transport_requests system property.
>
> I have been looking at SEPExecutor.addTask(FutureTask task), and here
> is my question. If the queue is full, as defined by
> SEPExector.maxTasksQueued, are tasks rejected? I do not fully grok the
> code, but it looks like it is possible for tasks to be rejected here (some
> code and comments omitted for brevity):
>
> public void addTask(FutureTask task)
> {
> tasks.add(task);
> ...
> else if (taskPermits >= maxTasksQueued)
> {
> WaitQueue.Signal s = hasRoom.register();
>
> if (taskPermits(permits.get()) > maxTasksQueued)
> {
> if (takeWorkPermit(true))
> pool.schedule(new Work(this))
>
> metrics.totalBlocked.inc();
> metrics.currentBlocked.inc();
> s.awaitUninterruptibly();
> metrics.currentBlocked.dec();
> }
> else
> s.cancel();
> }
> }
>
> The first thing that happens is that the task is added to the tasks queue.
> pool.schedule() only gets called if takeWorkPermit() returns true. I am
> still studying the code, but can someone explain what exactly happens when
> the queue is full?
>
>
> - John
>
>
>


-- 

- John

Understanding Blocked and All Time Blocked columns in tpstats

[John Sanda]

I have been doing some work on a cluster that is impacted by
https://issues.apache.org/jira/browse/CASSANDRA-11363. Reading through the
ticket prompted me to take a closer look at
org.apache.cassandra.concurrent.SEPExecutor. I am looking at the 3.0.14
code. I am a little confused about the Blocked and All Time Blocked columns
reported in nodetool tpstats and reported by StatusLogger. I understand
that there is a queue for tasks. In the case of RequestThreadPoolExecutor,
the size of that queue can be controlled via the
cassandra.max_queued_native_transport_requests system property.

I have been looking at SEPExecutor.addTask(FutureTask task), and here is
my question. If the queue is full, as defined by SEPExector.maxTasksQueued,
are tasks rejected? I do not fully grok the code, but it looks like it is
possible for tasks to be rejected here (some code and comments omitted for
brevity):

public void addTask(FutureTask task)
{
tasks.add(task);
...
else if (taskPermits >= maxTasksQueued)
{
WaitQueue.Signal s = hasRoom.register();

if (taskPermits(permits.get()) > maxTasksQueued)
{
if (takeWorkPermit(true))
pool.schedule(new Work(this))

metrics.totalBlocked.inc();
metrics.currentBlocked.inc();
s.awaitUninterruptibly();
metrics.currentBlocked.dec();
}
else
s.cancel();
}
}

The first thing that happens is that the task is added to the tasks queue.
pool.schedule() only gets called if takeWorkPermit() returns true. I am
still studying the code, but can someone explain what exactly happens when
the queue is full?


- John

Not marking node down due to local pause

[John Sanda]

I have a small, two-node cluster running Cassandra 2.2.1. I am seeing a lot
of these messages in both logs:

WARN  07:23:16 Not marking nodes down due to local pause of 7219277694 >
50

I am fairly certain that they are not due to GC. I am not seeing a whole of
GC being logged and nothing over 500 ms. I do think it is I/O related.

I am seeing lots of read timeouts for queries to a table that has a large
growing number of SSTables. At last count there are over 1800 SSTables on
one node. The count is lower on the other node, and I suspect that this is
due to data distribution. Slowly but surely the number of SSTables keeps
going up, and not surprisingly nodetool tablehistograms reports high
latencies. The table is using STCS.

I am seeing some but not a whole lot of dropped mutations. nodetool tpstats
looks ok.

The growing number of SSTables really makes me think this is an I/O issue.
Casssandra is running in a kubernetes cluster using a SAN which is another
reason I suspect I/O.

What are some things I can look at/test to determine what is causing all of
local pauses?

- John

Re: Cassandra - Nodes can't restart due to java.lang.OutOfMemoryError: Direct buffer memory

[John Sanda]

I am not sure which version of Netty is in 3.9, but maybe you are hitting
https://issues.apache.org/jira/browse/CASSANDRA-13114. I hit this in
Cassandra 3.0.9 which uses Netty 4.0.23. Here is the upstream netty ticket
https://github.com/netty/netty/issues/3057.

On Thu, Aug 31, 2017 at 10:15 AM, Chris Lohfink 
wrote:

> What version of java are you running? There is a "kinda leak" in jvm
> around this you may run into, can try with -Djdk.nio.maxCachedBufferSize=
> 262144 if above 8u102. You can also try increasing the size allowed for
> direct byte buffers. It defaults to size of heap -XX:MaxDirectMemorySize=?
> G
>
> Some NIO channel operations use temporary DirectByteBuffers which are
>> cached in thread-local caches to avoid having to allocate / free a buffer
>> at every operation.
>> Unfortunately, there is no bound imposed on the size of buffers added to
>> the thread-local caches. So, infrequent channel operations that require a
>> very large buffer can create a native memory leak.
>
>
>
>> *Ability to limit the capacity of buffers that can be held in the
>> temporary buffer cache*The system property jdk.nio.maxCachedBufferSize has
>> been introduced in 8u102 to limit the memory used by the "temporary buffer
>> cache." The temporary buffer cache is a per-thread cache of direct memory
>> used by the NIO implementation to support applications that do I/O with
>> buffers backed by arrays in the java heap. The value of the property is the
>> maximum capacity of a direct buffer that can be cached. If the property is
>> not set, then no limit is put on the size of buffers that are cached.
>> Applications with certain patterns of I/O usage may benefit from using this
>> property. In particular, an application that does I/O with large
>> multi-megabyte buffers at startup but does I/O with small buffers may see a
>> benefit to using this property. Applications that do I/O using direct
>> buffers will not see any benefit to using this system property.
>> See JDK-8147468 
>
>
> Chris
>
> On Thu, Aug 31, 2017 at 4:59 AM, Jonathan Baynes <
> jonathan.bay...@tradeweb.com> wrote:
>
>> I wonder if its related to this bug  (below) that’s currently unresolved,
>> albeit it being reproduced way back in 2.1.11
>>
>>
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-10689
>>
>>
>>
>>
>>
>> *From:* qf zhou [mailto:zhouqf2...@gmail.com]
>> *Sent:* 31 August 2017 10:58
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: Cassandra - Nodes can't restart due to
>> java.lang.OutOfMemoryError: Direct buffer memory
>>
>>
>>
>> I am usingCassandra 3.9 with cqlsh 5.0.1.
>>
>> 在 2017年8月31日，下午5:54，Jonathan Baynes  写道：
>>
>>
>>
>> again
>>
>>
>>
>> 
>>
>> This e-mail may contain confidential and/or privileged information. If
>> you are not the intended recipient (or have received this e-mail in error)
>> please notify the sender immediately and destroy it. Any unauthorized
>> copying, disclosure or distribution of the material in this e-mail is
>> strictly forbidden. Tradeweb reserves the right to monitor all e-mail
>> communications through its networks. If you do not wish to receive
>> marketing emails about our products / services, please let us know by
>> contacting us, either by email at contac...@tradeweb.com or by writing
>> to us at the registered office of Tradeweb in the UK, which is: Tradeweb
>> Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y
>> 9DT. To see our privacy policy, visit our website @ www.tradeweb.com.
>>
>
>


-- 

- John

Netty SSL memory leak

[John Sanda]

I have Cassandra 3.0.9 cluster that is hitting OutOfMemoryErrors with byte
buffer allocation. The stack trace looks like:

java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_131]
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
~[na:1.8.0_131]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
~[na:1.8.0_131]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:434)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:168)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:98)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:250)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:83)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1265)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1275)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:453)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:432)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]

I do not yet have a heap dump. The two relevant tickets are CASSANDRA-13114
 and CASSANDRA-13126
. The upstream Netty
ticket is 3057 . Cassandra
3.0.11 upgraded Netty to the version with the fix. Is there anything I can
check to confirm that this is in fact the issue I am hitting?

Secondly, is there a way to monitor for this? The OOME does not cause the
JVM to exit. Instead, the logs are getting filled up with OutOfMemoryErrors.
nodetool status reports UN, and nodetool statusbinary reports running.

-- 

- John

Re: manual deletes with TWCS

[John Sanda]

This is involving TTLed data, and I actually would want to delete all
related partitions across all time windows. Let's say I have a time series
partitioned by day with a 7 day TTL and a window size of one day. If I
delete partitions for the past seven days, would I still run into the issue
of data purge being postponed?

On Fri, May 5, 2017 at 4:57 PM, Jon Haddad <jonathan.had...@gmail.com>
wrote:

> You cannot.
>
> From Alex’s TLP post: http://thelastpickle.com/blog/2016/12/08/TWCS-
> part1.html
>
> TWCS is no fit for workload that perform deletes on non TTLed data.
> Consider that SSTables from different time windows will never be compacted
> together, so data inserted on day 1 and deleted on day 2 will have the
> tombstone and the shadowed cells living in different time windows. Unless a
> major compaction is performed (which shouldn’t), and while the deletion
> will seem effective when running queries, space will never be reclaimed on
> disk.
> Deletes can be performed on TTLed data if needed, but the partition will
> then exist in different time windows, which will postpone actual deletion
> from disk until both time windows fully expire.
>
>
> On May 5, 2017, at 1:54 PM, John Sanda <john.sa...@gmail.com> wrote:
>
> How problematic is it to perform deletes when using TWCS? I am currently
> using TWCS and have some new use cases for performing deletes. So far I
> have avoided performing deletes, but I am wondering what issues I might run
> into.
>
>
> - John
>
>
>


-- 

- John

manual deletes with TWCS

[John Sanda]

How problematic is it to perform deletes when using TWCS? I am currently
using TWCS and have some new use cases for performing deletes. So far I
have avoided performing deletes, but I am wondering what issues I might run
into.


- John

partition sizes reported by nodetool tablehistograms

[John Sanda]

I am working on some issues involving really big partitions. I have been
making extensive use of nodetool tablehistograms. What exactly is the
partition size being reported? I have a table for which the max value
reported is about 3.5 GB, but running du -h against the table data
directory reports 548 MB. Are the partition sizes reported by
tablehistograms the decompressed size on disk?

- John

Problems with large partitions and compaction

[John Sanda]

I have a table that uses LCS and has wound up with partitions upwards of
700 MB. I am seeing lots of the large partition warnings. Client requests
are subsequently failing. The driver is not reporting timeout exception,
just NoHostAvailableExceptions (in the logs I have reviewed so far). I know
that I need to redesign the table to avoid such large partitions. What
specifically goes wrong that results in the instability I am seeing? Or put
another way, what issues will compacting really large partitions cause?
Initially I thought that there was high GC activity, but after closer
inspection that does not really seem to happening. And most of the failures
I am seeing are on reads, but for an entirely different table. Lastly, does
anyone has anyone had success to switching to STCS in this situation as a
work around?

Thanks

- John

compaction falling behind

[John Sanda]

What is a good way to determine whether or not compaction is falling
behind? I read a couple things earlier that suggest nodetool
compactionstats might not be the most reliable thing to use.


- John

Re: Time series data model and tombstones

[John Sanda]

I wanted to provide a quick update. I was able to patch one of the
environments that is hitting the tombstone problem. It has been running
TWCS for five days now, and things are stable so far. I also had a patch to
the application code to implement date partitioning ready to go, but I
wanted to see how things went with only making the compaction changes.

On Sun, Jan 29, 2017 at 4:05 PM, DuyHai Doan <doanduy...@gmail.com> wrote:

> In theory, you're right and Cassandra should possibly skip reading cells
> having time < 50. But it's all theory, in practice Cassandra read chunks of
> xxx kilobytes worth of data (don't remember the exact value of xxx, maybe
> 64k or far less) so you may end up reading tombstones.
>
> On Sun, Jan 29, 2017 at 9:24 PM, John Sanda <john.sa...@gmail.com> wrote:
>
>> Thanks for the clarification. Let's say I have a partition in an SSTable
>> where the values of time range from 100 to 10 and everything < 50 is
>> expired. If I do a query with time < 100 and time >= 50, are there
>> scenarios in which Cassandra will have to read cells where time < 50? In
>> particular I am wondering if compression might have any affect.
>>
>> On Sun, Jan 29, 2017 at 3:01 PM DuyHai Doan <doanduy...@gmail.com> wrote:
>>
>>> "Should the data be sorted by my time column regardless of the
>>> compaction strategy" --> It does
>>>
>>> What I mean is that an old "chunk" of expired data in SSTABLE-12 may be
>>> compacted together with a new chunk of SSTABLE-2 containing fresh data so
>>> in the new resulting SSTable will contain tombstones AND fresh data inside
>>> the same partition, but of course sorted by clustering column "time".
>>>
>>> On Sun, Jan 29, 2017 at 8:55 PM, John Sanda <john.sa...@gmail.com>
>>> wrote:
>>>
>>> Since STCS does not sort data based on timestamp, your wide partition
>>> may span over multiple SSTables and inside each SSTable, old data (+
>>> tombstones) may sit on the same partition as newer data.
>>>
>>>
>>> Should the data be sorted by my time column regardless of the compaction
>>> strategy? I didn't think that the column timestamp came into play with
>>> respect to sorting. I have been able to review some SSTables with
>>> sstablemetadata and I can see that old/expired data is definitely living
>>> with live data.
>>>
>>>
>>> On Sun, Jan 29, 2017 at 2:38 PM, DuyHai Doan <doanduy...@gmail.com>
>>> wrote:
>>>
>>> Ok so give it a try with TWCS. Since STCS does not sort data based on
>>> timestamp, your wide partition may span over multiple SSTables and inside
>>> each SSTable, old data (+ tombstones) may sit on the same partition as
>>> newer data.
>>>
>>> When reading by slice, even if you request for fresh data, Cassandra has
>>> to scan over a lot tombstones to fetch the correct range of data thus your
>>> issue
>>>
>>> On Sun, Jan 29, 2017 at 8:19 PM, John Sanda <john.sa...@gmail.com>
>>> wrote:
>>>
>>> It was with STCS. It was on a 2.x version before TWCS was available.
>>>
>>> On Sun, Jan 29, 2017 at 10:58 AM DuyHai Doan <doanduy...@gmail.com>
>>> wrote:
>>>
>>> Did you get this Overwhelming tombstonne behavior with STCS or with TWCS
>>> ?
>>>
>>> If you're using DTCS, beware of its weird behavior and tricky
>>> configuration.
>>>
>>> On Sun, Jan 29, 2017 at 3:52 PM, John Sanda <john.sa...@gmail.com>
>>> wrote:
>>>
>>> Your partitioning key is text. If you have multiple entries per id you
>>> are likely hitting older cells that have expired. Descending only affects
>>> how the data is stored on disk, if you have to read the whole partition to
>>> find whichever time you are querying for you could potentially hit
>>> tombstones in other SSTables that contain the same "id". As mentioned
>>> previously, you need to add a time bucket to your partitioning key and
>>> definitely use DTCS/TWCS.
>>>
>>>
>>> As I mentioned previously, the UI only queries recent data, e.g., the
>>> past hour, past two hours, past day, past week. The UI does not query for
>>> anything older than the TTL which is 7 days. My understanding and
>>> expectation was that Cassandra would only scan live cells. The UI is a
>>> separate application that I do not maintain, so I am not 100% certain about
>>> the queries.

Questions about TWCS

[John Sanda]

In Jeff Jirsa C* 2016 summit presentation, TimeWindowCompactionStrategy for
Time Series Workloads, there is a slide which talks about optimizations. It
says to align partition keys to your TWCS windows. Is it generally the case
that calendar/date based partitions would align nicely with TWCS windows
such that we would end up with one SSTable per partition after the major
compaction runs?

http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html says to aim for <
50 buckets per table based on TTL. Are there any recommendations on a range
to stay within for number of buckets? What are some of the tradeoffs of
smaller vs larger number of buckets? For example I know that a smaller
number of buckets means more SSTable to compact during the major compaction
that runs when we get past a given window.

Are tombstone compactions disabled by default?

Can you ever wind up in a situation where the major compaction that is
supposed to run at the end of a window does not run? Not sure if this is
realistic but consider this scenario. Suppose compaction falls behind such
that there are 5 windows for which the major compactions have not run. Will
TWCS run the major compactions for those window serially oldest to newest?

With respect to TWCS would a write be considered out of order when it
arrives after its window has already finished? If am I using a window size
of one day, it it current 02:00 AM Tuesday, and I receive a write for 11:45
PM Monday, should I consider that out of order?

Thanks

- John

Re: Time series data model and tombstones

[John Sanda]

Thanks for the clarification. Let's say I have a partition in an SSTable
where the values of time range from 100 to 10 and everything < 50 is
expired. If I do a query with time < 100 and time >= 50, are there
scenarios in which Cassandra will have to read cells where time < 50? In
particular I am wondering if compression might have any affect.

On Sun, Jan 29, 2017 at 3:01 PM DuyHai Doan <doanduy...@gmail.com> wrote:

> "Should the data be sorted by my time column regardless of the compaction
> strategy" --> It does
>
> What I mean is that an old "chunk" of expired data in SSTABLE-12 may be
> compacted together with a new chunk of SSTABLE-2 containing fresh data so
> in the new resulting SSTable will contain tombstones AND fresh data inside
> the same partition, but of course sorted by clustering column "time".
>
> On Sun, Jan 29, 2017 at 8:55 PM, John Sanda <john.sa...@gmail.com> wrote:
>
> Since STCS does not sort data based on timestamp, your wide partition may
> span over multiple SSTables and inside each SSTable, old data (+
> tombstones) may sit on the same partition as newer data.
>
>
> Should the data be sorted by my time column regardless of the compaction
> strategy? I didn't think that the column timestamp came into play with
> respect to sorting. I have been able to review some SSTables with
> sstablemetadata and I can see that old/expired data is definitely living
> with live data.
>
>
> On Sun, Jan 29, 2017 at 2:38 PM, DuyHai Doan <doanduy...@gmail.com> wrote:
>
> Ok so give it a try with TWCS. Since STCS does not sort data based on
> timestamp, your wide partition may span over multiple SSTables and inside
> each SSTable, old data (+ tombstones) may sit on the same partition as
> newer data.
>
> When reading by slice, even if you request for fresh data, Cassandra has
> to scan over a lot tombstones to fetch the correct range of data thus your
> issue
>
> On Sun, Jan 29, 2017 at 8:19 PM, John Sanda <john.sa...@gmail.com> wrote:
>
> It was with STCS. It was on a 2.x version before TWCS was available.
>
> On Sun, Jan 29, 2017 at 10:58 AM DuyHai Doan <doanduy...@gmail.com> wrote:
>
> Did you get this Overwhelming tombstonne behavior with STCS or with TWCS ?
>
> If you're using DTCS, beware of its weird behavior and tricky
> configuration.
>
> On Sun, Jan 29, 2017 at 3:52 PM, John Sanda <john.sa...@gmail.com> wrote:
>
> Your partitioning key is text. If you have multiple entries per id you are
> likely hitting older cells that have expired. Descending only affects how
> the data is stored on disk, if you have to read the whole partition to find
> whichever time you are querying for you could potentially hit tombstones in
> other SSTables that contain the same "id". As mentioned previously, you
> need to add a time bucket to your partitioning key and definitely use
> DTCS/TWCS.
>
>
> As I mentioned previously, the UI only queries recent data, e.g., the past
> hour, past two hours, past day, past week. The UI does not query for
> anything older than the TTL which is 7 days. My understanding and
> expectation was that Cassandra would only scan live cells. The UI is a
> separate application that I do not maintain, so I am not 100% certain about
> the queries. I have been told that it does not query for anything older
> than 7 days.
>
> On Sun, Jan 29, 2017 at 4:14 AM, kurt greaves <k...@instaclustr.com>
> wrote:
>
>
> Your partitioning key is text. If you have multiple entries per id you are
> likely hitting older cells that have expired. Descending only affects how
> the data is stored on disk, if you have to read the whole partition to find
> whichever time you are querying for you could potentially hit tombstones in
> other SSTables that contain the same "id". As mentioned previously, you
> need to add a time bucket to your partitioning key and definitely use
> DTCS/TWCS.
>
>
>
>
>
> --
>
> - John
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> - John
>
>
>
>
>
>
>
>

Re: Time series data model and tombstones

[John Sanda]

>
> Since STCS does not sort data based on timestamp, your wide partition may
> span over multiple SSTables and inside each SSTable, old data (+
> tombstones) may sit on the same partition as newer data.


Should the data be sorted by my time column regardless of the compaction
strategy? I didn't think that the column timestamp came into play with
respect to sorting. I have been able to review some SSTables with
sstablemetadata and I can see that old/expired data is definitely living
with live data.


On Sun, Jan 29, 2017 at 2:38 PM, DuyHai Doan <doanduy...@gmail.com> wrote:

> Ok so give it a try with TWCS. Since STCS does not sort data based on
> timestamp, your wide partition may span over multiple SSTables and inside
> each SSTable, old data (+ tombstones) may sit on the same partition as
> newer data.
>
> When reading by slice, even if you request for fresh data, Cassandra has
> to scan over a lot tombstones to fetch the correct range of data thus your
> issue
>
> On Sun, Jan 29, 2017 at 8:19 PM, John Sanda <john.sa...@gmail.com> wrote:
>
>> It was with STCS. It was on a 2.x version before TWCS was available.
>>
>> On Sun, Jan 29, 2017 at 10:58 AM DuyHai Doan <doanduy...@gmail.com>
>> wrote:
>>
>>> Did you get this Overwhelming tombstonne behavior with STCS or with TWCS
>>> ?
>>>
>>> If you're using DTCS, beware of its weird behavior and tricky
>>> configuration.
>>>
>>> On Sun, Jan 29, 2017 at 3:52 PM, John Sanda <john.sa...@gmail.com>
>>> wrote:
>>>
>>> Your partitioning key is text. If you have multiple entries per id you
>>> are likely hitting older cells that have expired. Descending only affects
>>> how the data is stored on disk, if you have to read the whole partition to
>>> find whichever time you are querying for you could potentially hit
>>> tombstones in other SSTables that contain the same "id". As mentioned
>>> previously, you need to add a time bucket to your partitioning key and
>>> definitely use DTCS/TWCS.
>>>
>>>
>>> As I mentioned previously, the UI only queries recent data, e.g., the
>>> past hour, past two hours, past day, past week. The UI does not query for
>>> anything older than the TTL which is 7 days. My understanding and
>>> expectation was that Cassandra would only scan live cells. The UI is a
>>> separate application that I do not maintain, so I am not 100% certain about
>>> the queries. I have been told that it does not query for anything older
>>> than 7 days.
>>>
>>> On Sun, Jan 29, 2017 at 4:14 AM, kurt greaves <k...@instaclustr.com>
>>> wrote:
>>>
>>>
>>> Your partitioning key is text. If you have multiple entries per id you
>>> are likely hitting older cells that have expired. Descending only affects
>>> how the data is stored on disk, if you have to read the whole partition to
>>> find whichever time you are querying for you could potentially hit
>>> tombstones in other SSTables that contain the same "id". As mentioned
>>> previously, you need to add a time bucket to your partitioning key and
>>> definitely use DTCS/TWCS.
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> - John
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>


-- 

- John

Re: Time series data model and tombstones

[John Sanda]

It was with STCS. It was on a 2.x version before TWCS was available.

On Sun, Jan 29, 2017 at 10:58 AM DuyHai Doan <doanduy...@gmail.com> wrote:

> Did you get this Overwhelming tombstonne behavior with STCS or with TWCS ?
>
> If you're using DTCS, beware of its weird behavior and tricky
> configuration.
>
> On Sun, Jan 29, 2017 at 3:52 PM, John Sanda <john.sa...@gmail.com> wrote:
>
> Your partitioning key is text. If you have multiple entries per id you are
> likely hitting older cells that have expired. Descending only affects how
> the data is stored on disk, if you have to read the whole partition to find
> whichever time you are querying for you could potentially hit tombstones in
> other SSTables that contain the same "id". As mentioned previously, you
> need to add a time bucket to your partitioning key and definitely use
> DTCS/TWCS.
>
>
> As I mentioned previously, the UI only queries recent data, e.g., the past
> hour, past two hours, past day, past week. The UI does not query for
> anything older than the TTL which is 7 days. My understanding and
> expectation was that Cassandra would only scan live cells. The UI is a
> separate application that I do not maintain, so I am not 100% certain about
> the queries. I have been told that it does not query for anything older
> than 7 days.
>
> On Sun, Jan 29, 2017 at 4:14 AM, kurt greaves <k...@instaclustr.com>
> wrote:
>
>
> Your partitioning key is text. If you have multiple entries per id you are
> likely hitting older cells that have expired. Descending only affects how
> the data is stored on disk, if you have to read the whole partition to find
> whichever time you are querying for you could potentially hit tombstones in
> other SSTables that contain the same "id". As mentioned previously, you
> need to add a time bucket to your partitioning key and definitely use
> DTCS/TWCS.
>
>
>
>
>
> --
>
> - John
>
>
>
>
>
>
>
>

Re: Time series data model and tombstones

[John Sanda]

>
> Your partitioning key is text. If you have multiple entries per id you are
> likely hitting older cells that have expired. Descending only affects how
> the data is stored on disk, if you have to read the whole partition to find
> whichever time you are querying for you could potentially hit tombstones in
> other SSTables that contain the same "id". As mentioned previously, you
> need to add a time bucket to your partitioning key and definitely use
> DTCS/TWCS.


As I mentioned previously, the UI only queries recent data, e.g., the past
hour, past two hours, past day, past week. The UI does not query for
anything older than the TTL which is 7 days. My understanding and
expectation was that Cassandra would only scan live cells. The UI is a
separate application that I do not maintain, so I am not 100% certain about
the queries. I have been told that it does not query for anything older
than 7 days.

On Sun, Jan 29, 2017 at 4:14 AM, kurt greaves  wrote:

>
> Your partitioning key is text. If you have multiple entries per id you are
> likely hitting older cells that have expired. Descending only affects how
> the data is stored on disk, if you have to read the whole partition to find
> whichever time you are querying for you could potentially hit tombstones in
> other SSTables that contain the same "id". As mentioned previously, you
> need to add a time bucket to your partitioning key and definitely use
> DTCS/TWCS.
>



-- 

- John

Re: Time series data model and tombstones

[John Sanda]

Thanks for the response. This version of the code is using STCS.
gc_grace_seconds was set to one day and then I changed it to zero since RF
= 1. I understand that expired data will still generate tombstones and that
STCS is not the best. More recent versions of the code use DTCS, and we'll
be switching over to TWCS shortly. The suggestions raised are excellent
ones, but I tend to think of them as optimizations that might not address
my issue which I think may be 1) a problem with my data model, 2) problem
with the queries used or 3) some misunderstanding of Cassandra performs
range scans.

I am doing append-only writes. There is no out of order data. There are no
deletes, just TTLs. Data is stored on disk in descending order, and queries
access recent data and never query past the TTL of seven days. Given this I
would not except to be reading tombstones, certainly not the large numbers
that I am seeing.

On Sat, Jan 28, 2017 at 12:15 PM, Jonathan Haddad <j...@jonhaddad.com> wrote:

> Since you didn't specify a compaction strategy I'm guessing you're using
> STCS. Your TTL'ed data is becoming a tombstone. TWCS is a better strategy
> for this type of workload.
> On Sat, Jan 28, 2017 at 8:30 AM John Sanda <john.sa...@gmail.com> wrote:
>
>> I have a time series data model that is basically:
>>
>> CREATE TABLE metrics (
>> id text,
>> time timeuuid,
>> value double,
>> PRIMARY KEY (id, time)
>> ) WITH CLUSTERING ORDER BY (time DESC);
>>
>> I do append-only writes, no deletes, and use a TTL of seven days. Data
>> points are written every seconds. The UI queries data for the past hour,
>> two hours, day, or week. The UI refreshes and executes queries every 30
>> seconds. In one test environment I am seeing lots of tombstone threshold
>> warnings and Cassandra has even OOME'd. Since I am storing data in
>> descending order and always query for recent data, I do not understand why
>> I am running into this problem.
>>
>> I know that it is recommended to do some date partitioning in part to
>> ensure partitions do not grow too large. I already have some changes in
>> place to partition by day.. Before I make those changes I want to
>> understand why I am scanning so many tombstones so that I can be more
>> confident that the date partitioning changes will help.
>>
>> Thanks
>>
>> - John
>>
>


-- 

- John

Time series data model and tombstones

[John Sanda]

I have a time series data model that is basically:

CREATE TABLE metrics (
id text,
time timeuuid,
value double,
PRIMARY KEY (id, time)
) WITH CLUSTERING ORDER BY (time DESC);

I do append-only writes, no deletes, and use a TTL of seven days. Data
points are written every seconds. The UI queries data for the past hour,
two hours, day, or week. The UI refreshes and executes queries every 30
seconds. In one test environment I am seeing lots of tombstone threshold
warnings and Cassandra has even OOME'd. Since I am storing data in
descending order and always query for recent data, I do not understand why
I am running into this problem.

I know that it is recommended to do some date partitioning in part to
ensure partitions do not grow too large. I already have some changes in
place to partition by day.. Before I make those changes I want to
understand why I am scanning so many tombstones so that I can be more
confident that the date partitioning changes will help.

Thanks

- John

empty buckets with STCS

[John Sanda]

I have 2.2.1 Cassandra node that does not appear to be compacting
SSTables.  The table is currently configured with STCS. I turned on some
debug logging and when the compaction checks run, they log:

Compaction buckets are []

I have been going over SizeTieredCompactionStrategy.java and looking in
particular at the getNextBackgroundTables method since that is where the
above message is logged. Assuming I am reading the logs correctly, what if
anything besides all SSTables being blacklisted would result in empty
buckets? As I understand the code if an SSTable falls outside the threshold
of all existing buckets, it will be added to a new bucket.


- John

Re: commit log on NFS volume

[John Sanda]

>
> Using nfs for a distribited System like Cassandra is like putting a
> Ferrari on a Truck and going for a Race with the Truck. It is simply
> nonsense.


As I mentioned in my original post, I am aware that using NFS is considered
bad and even documented as an anti-pattern. Your analogy, interesting as it
may be, is not helpful. It simply restating what has already been said. I
don't even know that NFS is to blame for the CommitLogReplayException that
I cited.

On Tue, Nov 1, 2016 at 2:43 PM, Benjamin Roth <benjamin.r...@jaumo.com>
wrote:

> Using nfs for a distribited System like Cassandra is like putting a
> Ferrari on a Truck and going for a Race with the Truck. It is simply
> nonsense.
>
> Am 01.11.2016 19:39 schrieb "Vladimir Yudovin" <vla...@winguzone.com>:
>
>> Hi,
>>
>> it's not only performance issue. In case of network problem writer tread
>> can be blocked, also in case of failure loss of data can occur.
>>
>> Best regards, Vladimir Yudovin,
>>
>> *Winguzone <https://winguzone.com?from=list> - Hosted Cloud
>> CassandraLaunch your cluster in minutes.*
>>
>>
>>  On Tue, 01 Nov 2016 14:10:10 -0400*John Sanda <john.sa...@gmail.com
>> <john.sa...@gmail.com>>* wrote 
>>
>> I know that using NFS is discouraged, particularly for the commit log.
>> Can anyone shed some light into what kinds of problems I might encounter
>> aside from performance? The reason for my inquiry is because I have some
>> deployments with Cassandra 2.2.1 that use NFS and are experiencing some
>> problems like reoccurring corrupted commit log segments on start up:
>>
>> ERROR 19:38:42 Exiting due to error while processing commit log during 
>> initialization.
>> 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
>>  Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
>> at 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) 
>> [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) 
>> [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266) 
>> [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) 
>> [apache-cassandra-2.2.1.jar]
>>
>>
>> In one deployment after removing all of corrupted commit log segments I
>> got a different error:
>>
>> Exception (java.lang.RuntimeException) encountered during startup: 
>> java.nio.file.NoSuchFileException: 
>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>> java.lang.RuntimeException: java.nio.file.NoSuchFileException: 
>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>  at 
>> org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
>>  at 
>> org.apache.cassandra.io.util.ChannelProxy.(ChannelProxy.java:66)
>>  at 
>> org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
>>  at 
>> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
>>  at 
>> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
>>  at 
>> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
>>  at 
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
>>  at 
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>>  at 
>> org.apache.cassandra.service.

commit log on NFS volume

[John Sanda]

I know that using NFS is discouraged, particularly for the commit log. Can
anyone shed some light into what kinds of problems I might encounter aside
from performance? The reason for my inquiry is because I have some
deployments with Cassandra 2.2.1 that use NFS and are experiencing some
problems like reoccurring corrupted commit log segments on start up:

ERROR 19:38:42 Exiting due to error while processing commit log
during initialization.

org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
[apache-cassandra-2.2.1.jar]


In one deployment after removing all of corrupted commit log segments I got
a different error:

Exception (java.lang.RuntimeException) encountered during startup:
java.nio.file.NoSuchFileException:
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
java.lang.RuntimeException: java.nio.file.NoSuchFileException:
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
at 
org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
at 
org.apache.cassandra.io.util.ChannelProxy.(ChannelProxy.java:66)
at 
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
at 
org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
at 
org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
at 
org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
Caused by: java.nio.file.NoSuchFileException:
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at 
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at 
org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
... 8 more


The latter error looks like it involves compaction and might be unrelated.
I don't know if it matters, but I have commit log compression enabled in
these environments.

-- 

- John

Re: Client-side timeouts after dropping table

[John Sanda]

I was able to get metrics, but nothing stands out. When the applications
start up and a table is dropped, shortly thereafter on a subsequent write I
get a NoHostAvailableException that is caused by an
OperationTimedOutException. I am not 100% certain on which write the
timeout occurs because there are multiple apps running, but it does happen
fairly consistently almost immediately after the table is dropped. I don't
see any indication of a server side timeout or any dropped mutations being
reported in the log.

On Tue, Sep 20, 2016 at 11:07 PM, John Sanda <john.sa...@gmail.com> wrote:

> Thanks Nate. We do not have monitoring set up yet, but I should be able to
> get the deployment updated with a metrics reporter. I'll update the thread
> with my findings.
>
> On Tue, Sep 20, 2016 at 10:30 PM, Nate McCall <n...@thelastpickle.com>
> wrote:
>
>> If you can get to them in the test env. you want to look in
>> o.a.c.metrics.CommitLog for:
>> - TotalCommitlogSize: if this hovers near commitlog_size_in_mb and never
>> goes down, you are thrashing on segment allocation
>> - WaitingOnCommit: this is the time spent waiting on calls to sync and
>> will start to climb real fast if you cant sync within sync_interval
>> - WaitingOnSegmentAllocation: how long it took to allocate a new
>> commitlog segment, if it is all over the place it is IO bound
>>
>> Try turning all the commit log settings way down for low-IO test
>> infrastructure like this. Maybe total commit log size of like 32mb with 4mb
>> segments (or even lower depending on test data volume) so they basically
>> flush constantly and don't try to hold any tables open. Also lower
>> concurrent_writes substantially while you are at it to add some write
>> throttling.
>>
>> On Wed, Sep 21, 2016 at 2:14 PM, John Sanda <john.sa...@gmail.com> wrote:
>>
>>> I have seen in various threads on the list that 3.0.x is probably best
>>> for prod. Just wondering though if there is anything in particular in 3.7
>>> to be weary of.
>>>
>>> I need to check with one of our QA engineers to get specifics on the
>>> storage. Here is what I do know. We have a blade center running lots of
>>> virtual machines for various testing. Some of those vm's are running
>>> Cassandra and the Java web apps I previously mentioned via docker
>>> containers. The storage is shared. Beyond that I don't have any more
>>> specific details at the moment. I can also tell you that the storage can be
>>> quite slow.
>>>
>>> I have come across different threads that talk to one degree or another
>>> about the flush queue getting full. I have been looking at the code in
>>> ColumnFamilyStore.java. Is perDiskFlushExecutors the thread pool I should
>>> be interested in? It uses an unbounded queue, so I am not really sure what
>>> it means for it to get full. Is there anything I can check or look for to
>>> see if writes are getting blocked?
>>>
>>> On Tue, Sep 20, 2016 at 8:41 PM, Jonathan Haddad <j...@jonhaddad.com>
>>> wrote:
>>>
>>>> If you haven't yet deployed to prod I strongly recommend *not* using
>>>> 3.7.
>>>>
>>>> What network storage are you using?  Outside of a handful of highly
>>>> experienced experts using EBS in very specific ways, it usually ends in
>>>> failure.
>>>>
>>>> On Tue, Sep 20, 2016 at 3:30 PM John Sanda <john.sa...@gmail.com>
>>>> wrote:
>>>>
>>>>> I am deploying multiple Java web apps that connect to a Cassandra 3.7
>>>>> instance. Each app creates its own schema at start up. One of the schema
>>>>> changes involves dropping a table. I am seeing frequent client-side
>>>>> timeouts reported by the DataStax driver after the DROP TABLE statement is
>>>>> executed. I don't see this behavior in all environments. I do see it
>>>>> consistently in a QA environment in which Cassandra is running in docker
>>>>> with network storage, so writes are pretty slow from the get go. In my 
>>>>> logs
>>>>> I see a lot of tables getting flushed, which I guess are all of the dirty
>>>>> column families in the respective commit log segment. Then I seen a whole
>>>>> bunch of flushes getting queued up. Can I reach a point in which too many
>>>>> table flushes get queued such that writes would be blocked?
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> - John
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> - John
>>>
>>
>>
>>
>> --
>> -
>> Nate McCall
>> Wellington, NZ
>> @zznate
>>
>> CTO
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>
>
>
> --
>
> - John
>



-- 

- John

Re: Client-side timeouts after dropping table

[John Sanda]

Thanks Nate. We do not have monitoring set up yet, but I should be able to
get the deployment updated with a metrics reporter. I'll update the thread
with my findings.

On Tue, Sep 20, 2016 at 10:30 PM, Nate McCall <n...@thelastpickle.com>
wrote:

> If you can get to them in the test env. you want to look in
> o.a.c.metrics.CommitLog for:
> - TotalCommitlogSize: if this hovers near commitlog_size_in_mb and never
> goes down, you are thrashing on segment allocation
> - WaitingOnCommit: this is the time spent waiting on calls to sync and
> will start to climb real fast if you cant sync within sync_interval
> - WaitingOnSegmentAllocation: how long it took to allocate a new commitlog
> segment, if it is all over the place it is IO bound
>
> Try turning all the commit log settings way down for low-IO test
> infrastructure like this. Maybe total commit log size of like 32mb with 4mb
> segments (or even lower depending on test data volume) so they basically
> flush constantly and don't try to hold any tables open. Also lower
> concurrent_writes substantially while you are at it to add some write
> throttling.
>
> On Wed, Sep 21, 2016 at 2:14 PM, John Sanda <john.sa...@gmail.com> wrote:
>
>> I have seen in various threads on the list that 3.0.x is probably best
>> for prod. Just wondering though if there is anything in particular in 3.7
>> to be weary of.
>>
>> I need to check with one of our QA engineers to get specifics on the
>> storage. Here is what I do know. We have a blade center running lots of
>> virtual machines for various testing. Some of those vm's are running
>> Cassandra and the Java web apps I previously mentioned via docker
>> containers. The storage is shared. Beyond that I don't have any more
>> specific details at the moment. I can also tell you that the storage can be
>> quite slow.
>>
>> I have come across different threads that talk to one degree or another
>> about the flush queue getting full. I have been looking at the code in
>> ColumnFamilyStore.java. Is perDiskFlushExecutors the thread pool I should
>> be interested in? It uses an unbounded queue, so I am not really sure what
>> it means for it to get full. Is there anything I can check or look for to
>> see if writes are getting blocked?
>>
>> On Tue, Sep 20, 2016 at 8:41 PM, Jonathan Haddad <j...@jonhaddad.com>
>> wrote:
>>
>>> If you haven't yet deployed to prod I strongly recommend *not* using
>>> 3.7.
>>>
>>> What network storage are you using?  Outside of a handful of highly
>>> experienced experts using EBS in very specific ways, it usually ends in
>>> failure.
>>>
>>> On Tue, Sep 20, 2016 at 3:30 PM John Sanda <john.sa...@gmail.com> wrote:
>>>
>>>> I am deploying multiple Java web apps that connect to a Cassandra 3.7
>>>> instance. Each app creates its own schema at start up. One of the schema
>>>> changes involves dropping a table. I am seeing frequent client-side
>>>> timeouts reported by the DataStax driver after the DROP TABLE statement is
>>>> executed. I don't see this behavior in all environments. I do see it
>>>> consistently in a QA environment in which Cassandra is running in docker
>>>> with network storage, so writes are pretty slow from the get go. In my logs
>>>> I see a lot of tables getting flushed, which I guess are all of the dirty
>>>> column families in the respective commit log segment. Then I seen a whole
>>>> bunch of flushes getting queued up. Can I reach a point in which too many
>>>> table flushes get queued such that writes would be blocked?
>>>>
>>>>
>>>> --
>>>>
>>>> - John
>>>>
>>>
>>
>>
>> --
>>
>> - John
>>
>
>
>
> --
> -
> Nate McCall
> Wellington, NZ
> @zznate
>
> CTO
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>



-- 

- John

Re: Client-side timeouts after dropping table

[John Sanda]

I have seen in various threads on the list that 3.0.x is probably best for
prod. Just wondering though if there is anything in particular in 3.7 to be
weary of.

I need to check with one of our QA engineers to get specifics on the
storage. Here is what I do know. We have a blade center running lots of
virtual machines for various testing. Some of those vm's are running
Cassandra and the Java web apps I previously mentioned via docker
containers. The storage is shared. Beyond that I don't have any more
specific details at the moment. I can also tell you that the storage can be
quite slow.

I have come across different threads that talk to one degree or another
about the flush queue getting full. I have been looking at the code in
ColumnFamilyStore.java. Is perDiskFlushExecutors the thread pool I should
be interested in? It uses an unbounded queue, so I am not really sure what
it means for it to get full. Is there anything I can check or look for to
see if writes are getting blocked?

On Tue, Sep 20, 2016 at 8:41 PM, Jonathan Haddad <j...@jonhaddad.com> wrote:

> If you haven't yet deployed to prod I strongly recommend *not* using 3.7.
>
> What network storage are you using?  Outside of a handful of highly
> experienced experts using EBS in very specific ways, it usually ends in
> failure.
>
> On Tue, Sep 20, 2016 at 3:30 PM John Sanda <john.sa...@gmail.com> wrote:
>
>> I am deploying multiple Java web apps that connect to a Cassandra 3.7
>> instance. Each app creates its own schema at start up. One of the schema
>> changes involves dropping a table. I am seeing frequent client-side
>> timeouts reported by the DataStax driver after the DROP TABLE statement is
>> executed. I don't see this behavior in all environments. I do see it
>> consistently in a QA environment in which Cassandra is running in docker
>> with network storage, so writes are pretty slow from the get go. In my logs
>> I see a lot of tables getting flushed, which I guess are all of the dirty
>> column families in the respective commit log segment. Then I seen a whole
>> bunch of flushes getting queued up. Can I reach a point in which too many
>> table flushes get queued such that writes would be blocked?
>>
>>
>> --
>>
>> - John
>>
>


-- 

- John

Client-side timeouts after dropping table

[John Sanda]

I am deploying multiple Java web apps that connect to a Cassandra 3.7
instance. Each app creates its own schema at start up. One of the schema
changes involves dropping a table. I am seeing frequent client-side
timeouts reported by the DataStax driver after the DROP TABLE statement is
executed. I don't see this behavior in all environments. I do see it
consistently in a QA environment in which Cassandra is running in docker
with network storage, so writes are pretty slow from the get go. In my logs
I see a lot of tables getting flushed, which I guess are all of the dirty
column families in the respective commit log segment. Then I seen a whole
bunch of flushes getting queued up. Can I reach a point in which too many
table flushes get queued such that writes would be blocked?

-- 

- John

mutation checksum failure during commit log replay

[John Sanda]

What could cause an error like:

ERROR 07:11:56 Exiting due to error while processing commit log during
initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
Mutation checksum failure at 818339 in CommitLog-5-1470234746867.log


This is with Cassandra 2.2.4. It's clear that the commit log segment
is corrupted in some way. Cassandra is running in docker. This happens
after destroying the container and starting Cassandra in a new
container on a different host machine which is using the same commit
log and data directories which are attached via network storage.

-- 

- John

Re: Forming a cluster of embedded Cassandra instances

[John Sanda]

The motivation was to make it easy for someone to get up and running
quickly with the project. Clone the git repo, run the maven build, and then
you are all set. It definitely does lower the learning curve for someone
just getting started with a project and who is not really thinking about
Cassandra. It also is convenient for non-devs who need to quickly get the
project up and running. For development, we have people working on Linux,
Mac OS X, and Windows. I am not a Windows user and not even sure if ccm
works on Windows, so ccm can't be the de factor standard for development.

On Sun, Feb 14, 2016 at 2:52 PM, Jack Krupansky <jack.krupan...@gmail.com>
wrote:

> What motivated the use of an embedded instance for development - as
> opposed to simply spawning a process for Cassandra?
>
>
>
> -- Jack Krupansky
>
> On Sun, Feb 14, 2016 at 2:05 PM, John Sanda <john.sa...@gmail.com> wrote:
>
>> The project I work on day to day uses an embedded instance of Cassandra,
>> but it is intended for primarily for development. We embed Cassandra in a
>> WildFly (i.e., JBoss) server. It is packaged and deployed as an EAR. I
>> personally do not do this. I use and recommend ccm
>> <https://github.com/pcmanus/ccm> for development. If you do you WildFly,
>> there is also wildfly-cassandra
>> <https://github.com/hawkular/wildfly-cassandra> which deploys Cassandra
>> as a custom WildFly extension. In other words it is deployed in WildFly
>> like other subsystems like EJB, web, etc, not like an application. There
>> isn't a whole lot of active development on this, but it could be another
>> option.
>>
>> For production, we have to support single node clusters (not embedded
>> though), and it has been challenging for pretty much all the reasons you
>> find people saying not to do so.
>>
>> As for failure detection and cluster membership changes, are you using
>> the Datastax driver? You can register an event listener with the driver to
>> receive notifications for those things.
>>
>> On Sat, Feb 13, 2016 at 6:33 PM, Jonathan Haddad <j...@jonhaddad.com>
>> wrote:
>>
>>> +1 to what jack said. Don't mess with embedded till you understand the
>>> basics of the db. You're not making your system any less complex, I'd say
>>> you're most likely going to shoot yourself in the foot.
>>> On Sat, Feb 13, 2016 at 2:22 PM Jack Krupansky <jack.krupan...@gmail.com>
>>> wrote:
>>>
>>>> HA requires an odd number of replicas - 3, 5, 7 - so that split-brain
>>>> can be avoided. Two nodes would not support HA. You need to be able to
>>>> reach a quorum, which is defined as n/2+1 where n is the number of
>>>> replicas. IOW, you cannot update the data if a quorum cannot be reached.
>>>> The data on any given node needs to be replicated on at least two other
>>>> nodes.
>>>>
>>>> Embedded Cassandra is only for extremely sophisticated developers - not
>>>> those who are new to Cassandra, with a "superficial understanding".
>>>>
>>>> As a general proposition, you should not be running application code on
>>>> Cassandra nodes.
>>>>
>>>> That said, if any of the senior Cassandra developers wish to personally
>>>> support your efforts towards embedded clusters, they are certainly free to
>>>> do so. we'll see if any of them step forward.
>>>>
>>>>
>>>> -- Jack Krupansky
>>>>
>>>> On Sat, Feb 13, 2016 at 3:47 PM, Binil Thomas <
>>>> binil.thomas.pub...@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> TL;DR: I have a very superficial understanding of Cassandra and am
>>>>> currently evaluating it for a project.
>>>>>
>>>>> * Can Cassandra be embedded into another JVM application?
>>>>> * Can such embedded instances form a cluster?
>>>>> * Can the application use the the failure detection and cluster
>>>>> membership dissemination infrastructure of embedded Cassandra?
>>>>>
>>>>> 
>>>>>
>>>>> I am in the process of re-packaging a SaaS system written in Java to
>>>>> be deployed on-premise by customers. The SaaS system currently uses AWS
>>>>> DynamoDB. The data storage needs for this application are modest, but I
>>>>> would like to keep the deployment complexity to a minimum. Here are three
>>>>> different usecases the on-premise system should support:
>>&

Re: Forming a cluster of embedded Cassandra instances

[John Sanda]

The project I work on day to day uses an embedded instance of Cassandra,
but it is intended for primarily for development. We embed Cassandra in a
WildFly (i.e., JBoss) server. It is packaged and deployed as an EAR. I
personally do not do this. I use and recommend ccm
 for development. If you do you WildFly,
there is also wildfly-cassandra
 which deploys Cassandra as
a custom WildFly extension. In other words it is deployed in WildFly like
other subsystems like EJB, web, etc, not like an application. There isn't a
whole lot of active development on this, but it could be another option.

For production, we have to support single node clusters (not embedded
though), and it has been challenging for pretty much all the reasons you
find people saying not to do so.

As for failure detection and cluster membership changes, are you using the
Datastax driver? You can register an event listener with the driver to
receive notifications for those things.

On Sat, Feb 13, 2016 at 6:33 PM, Jonathan Haddad  wrote:

> +1 to what jack said. Don't mess with embedded till you understand the
> basics of the db. You're not making your system any less complex, I'd say
> you're most likely going to shoot yourself in the foot.
> On Sat, Feb 13, 2016 at 2:22 PM Jack Krupansky 
> wrote:
>
>> HA requires an odd number of replicas - 3, 5, 7 - so that split-brain can
>> be avoided. Two nodes would not support HA. You need to be able to reach a
>> quorum, which is defined as n/2+1 where n is the number of replicas. IOW,
>> you cannot update the data if a quorum cannot be reached. The data on any
>> given node needs to be replicated on at least two other nodes.
>>
>> Embedded Cassandra is only for extremely sophisticated developers - not
>> those who are new to Cassandra, with a "superficial understanding".
>>
>> As a general proposition, you should not be running application code on
>> Cassandra nodes.
>>
>> That said, if any of the senior Cassandra developers wish to personally
>> support your efforts towards embedded clusters, they are certainly free to
>> do so. we'll see if any of them step forward.
>>
>>
>> -- Jack Krupansky
>>
>> On Sat, Feb 13, 2016 at 3:47 PM, Binil Thomas <
>> binil.thomas.pub...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> TL;DR: I have a very superficial understanding of Cassandra and am
>>> currently evaluating it for a project.
>>>
>>> * Can Cassandra be embedded into another JVM application?
>>> * Can such embedded instances form a cluster?
>>> * Can the application use the the failure detection and cluster
>>> membership dissemination infrastructure of embedded Cassandra?
>>>
>>> 
>>>
>>> I am in the process of re-packaging a SaaS system written in Java to be
>>> deployed on-premise by customers. The SaaS system currently uses AWS
>>> DynamoDB. The data storage needs for this application are modest, but I
>>> would like to keep the deployment complexity to a minimum. Here are three
>>> different usecases the on-premise system should support:
>>>
>>> 1. single-node deployments with minimal complexity
>>> 2. two-node HA deployments; the data and processing needs dictated by
>>> the load on the system are well under what a single node can do, but the
>>> second node is there to satisfy the HA requirement as a hot standby
>>> 3. a multi-node clustered deployment, where higher operational
>>> complexity is justified
>>>
>>> I am considering Cassandra for these usecases.
>>>
>>> For usecase #1, I hope to embed Cassandra into the same JVM as my
>>> application. I read on the web that CassandraDaemon can be used this way.
>>> Is that accurate? What other applications embed Cassandra this way? I
>>> *think* JetBrains Upsource does, but do you know other ones? (Incidentally,
>>> my Java application embeds Jetty webserver also).
>>>
>>> For usecase #2, I am hoping that I can deploy two instances of this
>>> ensemble and have the embedded Cassandra instances form a cluster. If I
>>> configure every write to be replicated on both nodes synchronously, then it
>>> will satisfy the HA needs of this usecase. Is it feasible to form clusters
>>> of embedded Cassandra instances?
>>>
>>> For usecase #3, I can form a large cluster of the ensemble where all
>>> writes are replicated synchronously to a quorum of nodes.
>>>
>>> Finally, in usecase #2 and #3, I'd like to use the failure detection and
>>> cluster membership dissemination infrastructure of Cassandra from within my
>>> application. Is it possible to be notified of membership changes when
>>> embedding Cassandra? I could use a separate library to do this (say, with
>>> JGroups or Akka) but I fear that if this library and the embedded Cassandra
>>> instances disagrees, it could lead to subtle bugs.
>>>
>>> Thanks,
>>> Binil
>>>
>>> PS: Cross-posted at
>>> 

Re: Schema Versioning

[John Sanda]

If you are interested in a solution that maintains scripts, there are at
least a few projects available,

https://github.com/comeara/pillar - Runs on the JVM and written in Scala.
Scripts are CQL files.
https://github.com/Contrast-Security-OSS/cassandra-migration - Runs on JVM
and I believe a port of Flyway
https://github.com/hsgubert/cassandra_migrations - Ruby based and similar
to ActiveRecord
https://github.com/jsanda/cassalog - A project I have started. Runs on JVM
and scripts are groovy files.

On Thu, Feb 11, 2016 at 4:57 AM, Carlos Alonso  wrote:

> Here we use the Cassanity gem: https://github.com/jnunemaker/cassanity
> This one suggests using schema migration files that are then registered in
> a column family to keep track of the version.
>
> Carlos Alonso | Software Engineer | @calonso 
>
> On 10 February 2016 at 21:29, Alex Popescu  wrote:
>
>>
>> On Wed, Feb 10, 2016 at 12:05 PM, Joe Bako  wrote:
>>
>>> Modern RDBMS tools can compare schemas between DDL object definitions
>>> and live databases and generate change scripts accordingly.  Older
>>> techniques included maintaining a version and script table in the database,
>>> storing schema change scripts in a sequential fashion on disk, and
>>> iterating over them to apply them against the target database based on
>>> whether they had been run previously or not (indicated in the script table).
>>
>>
>> Using DevCenter will give you some of these features (and future versions
>> will add more). Just to give you a quick example, if using DevCenter to
>> make schema changes it will offer the options of saving the final
>> definition or just the set of changes applied (to an existing CQL file or a
>> new one).
>>
>>
>> --
>> Bests,
>>
>> Alex Popescu | @al3xandru
>> Sen. Product Manager @ DataStax
>>
>>
>


-- 

- John

Consistent reads and first write wins

[John Sanda]

Suppose I have the following schema,

CREATE TABLE foo (
id text,
time timeuuid,
prop1 text,
PRIMARY KEY (id, time)
)
WITHCLUSTERING ORDER BY (time ASC);

And I have two clients who execute quorum writes, e.g.,

// client 1
INSERT INTO FOO (id, time, prop1) VALUES ('test', time_uuid_1, 'bar');

// client 2
INSERT INTO FOO (id, time, prop1) VALUES ('test', time_uuid_2, 'bam');

If time_uuid_1 comes before time_uuid_2 and if both clients follow up the
writes with quorum reads, then will both clients see the value 'bar' for
prop1? Are there situations in which clients might see different values?


-- 

- John

Re: Example Data Modelling

[John Sanda]

25 MB seems very specific. Is there a reason why?

On Tuesday, July 7, 2015, Peer, Oded oded.p...@rsa.com wrote:

  The data model suggested isn’t optimal for the “end of month” query you
 want to run since you are not querying by partition key.

 The query would look like “select EmpID, FN, LN, basic from salaries where
 month = 1” which requires filtering and has unpredictable performance.



 For this type of query to be fast you can use the “month” column as the
 partition key and the “EmpID” and the clustering column.

 This approach also has drawbacks:

 1. This data model creates a wide row. Depending on the number of
 employees this partition might be very large. You should limit partition
 sizes to 25MB

 2. Distributing data according to month means that only a small number of
 nodes will hold all of the salary data for a specific month which might
 cause hotspots on those nodes.



 Choose the approach that works best for you.





 *From:* Carlos Alonso [mailto:i...@mrcalonso.com
 javascript:_e(%7B%7D,'cvml','i...@mrcalonso.com');]
 *Sent:* Monday, July 06, 2015 7:04 PM
 *To:* user@cassandra.apache.org
 javascript:_e(%7B%7D,'cvml','user@cassandra.apache.org');
 *Subject:* Re: Example Data Modelling



 Hi Srinivasa,



 I think you're right, In Cassandra you should favor denormalisation when
 in RDBMS you find a relationship like this.



 I'd suggest a cf like this

 CREATE TABLE salaries (

   EmpID varchar,

   FN varchar,

   LN varchar,

   Phone varchar,

   Address varchar,

   month integer,

   basic integer,

   flexible_allowance float,

   PRIMARY KEY(EmpID, month)

 )



 That way the salaries will be partitioned by EmpID and clustered by month,
 which I guess is the natural sorting you want.



 Hope it helps,

 Cheers!


   Carlos Alonso | Software Engineer | @calonso
 https://twitter.com/calonso



 On 6 July 2015 at 13:01, Srinivasa T N seen...@gmail.com
 javascript:_e(%7B%7D,'cvml','seen...@gmail.com'); wrote:

 Hi,

I have basic doubt: I have an RDBMS with the following two tables:

Emp - EmpID, FN, LN, Phone, Address
Sal - Month, Empid, Basic, Flexible Allowance

My use case is to print the Salary slip at the end of each month and
 the slip contains emp name and his other details.

Now, if I want to have the same in cassandra, I will have a single cf
 with emp personal details and his salary details.  Is this the right
 approach?  Should we have the employee personal details duplicated each
 month?

 Regards,
 Seenu.





-- 

- John

Re: Migrate table data to another table

[John Sanda]

You might want to take a look at CQLSSTableWriter[1] in the Cassandra
source tree.

http://www.datastax.com/dev/blog/using-the-cassandra-bulk-loader-updated

On Tue, Jun 30, 2015 at 1:18 PM, Umut Kocasaraç ukocasa...@gmail.com
wrote:

 Hi,
 I want to change clustering order column of my table. As far as i know it
 is not possible to use alter command so i have created new table and i
 would like to move data from old table to this one.

 I am using Cassandra 2.0.7 and there is almost 100GB data on table. Is
 there any easy method to move data except Copy command.

 Thanks

 Umut




-- 

- John

Large SSTable not compacted with size tiered compaction

[John Sanda]

I have a write-heavy table that is using size tiered compaction. I am
running C* 1.2.9. There is an SSTable that is not getting compacted. It is
disproportionately larger than the other SSTables. The data file sizes are,

1.70 GB
0.18 GB
0.16 GB
0.05 GB
8.61 GB

If I set the bucket_high compaction property on the table to a sufficiently
large value, will the 8.61 GB get compacted? What if any drawbacks are
there to increasing the bucket_high property?

In what scenarios could I wind up with such a disproportionately large
SSTable like this? One thing that comes to mind is major compactions, but I
have not that.

- John

Re: Performance problem with large wide row inserts using CQL

[John Sanda]

From a quick glance at your code, it looks like you are preparing your
insert statement multiple times. You only need to prepare it once. I would
expect to see some improvement with that change.


On Wed, Feb 19, 2014 at 5:27 AM, Rüdiger Klaehn rkla...@gmail.com wrote:

 Hi all,

 I am evaluating Cassandra for satellite telemetry storage and analysis. I
 set up a little three node cluster on my local development machine and
 wrote a few simple test programs.

 My use case requires storing incoming telemetry updates in the database at
 the same rate as they are coming in. A telemetry update is a map of
 name/value pairs that arrives at a certain time.

 The idea is that I want to store the data as quickly as possible, and then
 later store it in an additional format that is more amenable to analysis.

 The format I have chosen for my test is the following:

 CREATE TABLE IF NOT EXISTS test.wide (
   time varchar,
   name varchar,
   value varchar,
   PRIMARY KEY (time,name))
   WITH COMPACT STORAGE

 The layout I want to achieve with this is something like this:

 +---+---+---+---+---+---+
 |   | name1 | name2 | name3 | ...   | nameN |
 | time  +---+---+---+---+---+
 |   | val1  | val2  | val3  | ...   | valN  |
 +---+---+---+---|---+---+

 (Time will at some point be some kind of timestamp, and value will become
 a blob. But this is just for initial testing)

 The problem is the following: I am getting very low performance for bulk
 inserts into the above table. In my test program, each insert has a new,
 unique time and creates a row with 1 name/value pairs. This should map
 into creating a new row in the underlying storage engine, correct? I do
 that 1000 times and measure both time per insert and total time.

 I am getting about 0.5s for each insert of 1 name/value pairs, which
 is much lower than the rate at which the telemetry is coming in at my
 system. I have read a few previous threads on this subject and am using
 batch prepared statements for maximum performance (
 https://issues.apache.org/jira/browse/CASSANDRA-4693 ). But that does not
 help.

 Here is the CQL benchmark:
 https://gist.github.com/rklaehn/9089304#file-cassandratestminimized-scala

 I have written the exact same thing using the thrift API of astyanax, and
 I am getting much better performance. Each insert of 1 name/values
 takes 0.04s using a ColumnListMutation. When I use async calls for both
 programs, as suggested by somebody on Stackoverflow, the difference gets
 even larger. The CQL insert remains at 0.5s per insert on average, whereas
 the astyanax ColumnListMutation approach takes 0.01s per insert on
 average, even on my test cluster. That's the kind of performance I need.

 Here is the thrift benchmark, modified from an ast example:
 https://gist.github.com/rklaehn/9089304#file-astclient-java

 I realize that running on a test cluster on localhost is not a 100%
 realistic test. But nevertheless you would expect both tests to have
 roughly similar performance.

 I saw a few suggestions to create a table with CQL and fill it using the
 thrift API. For example in this thread
 http://mail-archives.apache.org/mod_mbox/cassandra-user/201309.mbox/%3c523334b8.8070...@gmail.com%3E.
  But I would very much prefer to use pure CQL for this. It seems that the
 thrift API is considered deprecated, so I would not feel comfortable
 starting a new project using a legacy API.

 I already posted a question on SO about this, but did not get any
 satisfactory answer. Just general performance tuning tips that do nothing
 to explain the difference between the CQL and thrift approaches.

 http://stackoverflow.com/questions/21778671/cassandra-how-to-insert-a-new-wide-row-with-good-performance-using-cql

 Am I doing something wrong, or is this a fundamental limitation of CQL. If
 the latter is the case, what's the plan to mitigate the issue?

 There is a JIRA issue about this (
 https://issues.apache.org/jira/browse/CASSANDRA-5959 ), but it is marked
 as a duplicate of https://issues.apache.org/jira/browse/CASSANDRA-4693 .
 But according to my benchmarks batch prepared statements do not solve this
 issue!

 I would really appreciate any help on this issue. The telemetry data I
 would like to import into C* for testing contains ~2*10^12 samples, where
 each sample consists of time, value and status. If quick batch insertion is
 not possible, I would not even be able to insert it in an acceptable time.

 best regards,

 Rüdiger




-- 

- John

Re: user / password authentication advice

[John Sanda]

You could use CassandraAuthorizer and PaaswordAuthenticator which ships
with Cassandra. See this article[1] for a good overview.

[1]
http://www.datastax.com/dev/blog/a-quick-tour-of-internal-authentication-and-authorization-security-in-datastax-enterprise-and-apache-cassandra

On Thursday, December 12, 2013, onlinespending wrote:

 OK, thanks for getting me going in the right direction. I imagine most
 people would store password and tokenized authentication information in a
 single table, using the username (e.g. email address) as the key?


 On Dec 11, 2013, at 10:44 PM, Janne Jalkanen 
 janne.jalka...@ecyrd.comjavascript:_e({}, 'cvml', 
 'janne.jalka...@ecyrd.com');
 wrote:


 Hi!

 You're right, this isn't really Cassandra-specific. Most languages/web
 frameworks have their own way of doing user authentication, and then you
 just typically write a plugin that just stores whatever data the system
 needs in Cassandra.

 For example, if you're using Java (or Scala or Groovy or anything else
 JVM-based), Apache Shiro is a good way of doing user authentication and
 authorization. http://shiro.apache.org/. Just implement a custom Realm
 for Cassandra and you should be set.

 /Janne

 On Dec 12, 2013, at 05:31 , onlinespending 
 onlinespend...@gmail.comjavascript:_e({}, 'cvml', 
 'onlinespend...@gmail.com');
 wrote:

 Hi,

 I’m using Cassandra in an environment where many users can login to use an
 application I’m developing. I’m curious if anyone has any advice or links
 to documentation / blogs where it discusses common implementations or best
 practices for user and password authentication. My cursory search online
 didn’t bring much up on the subject. I suppose the information needn’t even
 be specific to Cassandra.

 I imagine a few basic steps will be as follows:


- user types in username (e.g. email address) and password
- this is verified against a table storing username and passwords
(encrypted in some way)
- a token is return to the app / web browser to allow further
transactions using secure token (e.g. cookie)


 Obviously I’m only scratching the surface and it’s the detail and best
 practices of implementing this user / password authentication that I’m
 curious about.

 Thank you,
 Ben






-- 

- John

Re: What is the fastest way to get data into Cassandra 2 from a Java application?

[John Sanda]

The session.execute blocks until the C* returns the response. Use the async
version, but do so with caution. If you don't throttle the requests, you
will start seeing timeouts on the client side pretty quickly. For
throttling I've used a Semaphore, but I think Guava's RateLimiter is better
suited. And if you want to wait until all the writes have finished,
definitely use Guava's futures API. Try something like,

PreparedStatement ps = session.prepare(INSERT INTO perf_test.wibble
(id, info) VALUES (?, ?));
RateLimiter permits = RateLimiter.create(500);// you will need to tune
this to your environment
int count = 1000;
final CountDownLatch latch = new CountDownLatch(count);
for (int i = 0; i  count; i++) {
ResultSetFuture future = session.executeAsync(ps.bind( + i, aa +
i));
Futures.addCallback(future, new FutureCallbackResultSet() {
public void onSuccess(ResultSet rows) {
latch.countDown();
}

public void onFailure(Throwable t) {
latch.countDown();
// log the error or other error handling
}
});
}
latch.await();   // need to handle and/or throw InterruptedException



On Tue, Dec 10, 2013 at 8:16 PM, graham sanderson gra...@vast.com wrote:

 I can’t speak for Astyanax; their thrift transport I believe is abstracted
 out, however the object model is very CF wide row vs table-y.

 I have no idea what the plans are for further Astyanax dev (maybe someone
 on this list), but I believe the thrift API is not going away, so
 considering Astyanax/thrift is an option, thought I’d imagine you wouldn’t
 gain much going down the CQL over thrift method, so you need to be able to
 model your data in “internal” form.

 Two reasons we may want to move to the binary protocol
 for reads: asynchronous ability (which is now in thrift but it seems
 unlikely to be utilized in cassandra)
 for writes: compression, since we are (currently) network bandwidth
 limited for enormous batch inserts (from hadoop)

 On Dec 10, 2013, at 6:44 AM, David Tinker david.tin...@gmail.com wrote:

  Hmm. I have read that the thrift interface to Cassandra is out of
  favour and the CQL interface is in. Where does that leave Astyanax?
 
  On Tue, Dec 10, 2013 at 1:14 PM, graham sanderson gra...@vast.com
 wrote:
  Perhaps not the way forward, however I can bulk insert data via
 astyanax at a rate that maxes out our (fast) networks. That said for our
 next release (of this part of our product - our other current is node.js
 via binary protocol) we will be looking at insert speed via java driver,
 and also alternative scala/java implementations of the binary protocol.
 
  On Dec 10, 2013, at 4:49 AM, David Tinker david.tin...@gmail.com
 wrote:
 
  I have tried the DataStax Java driver and it seems the fastest way to
  insert data is to compose a CQL string with all parameters inline.
 
  This loop takes 2500ms or so on my test cluster:
 
  PreparedStatement ps = session.prepare(INSERT INTO perf_test.wibble
  (id, info) VALUES (?, ?))
  for (int i = 0; i  1000; i++) session.execute(ps.bind( + i, aa +
 i));
 
  The same loop with the parameters inline is about 1300ms. It gets
  worse if there are many parameters. I know I can use batching to
  insert all the rows at once but thats not the purpose of this test. I
  also tried using session.execute(cql, params) and it is faster but
  still doesn't match inline values.
 
  Composing CQL strings is certainly convenient and simple but is there
  a much faster way?
 
  Thanks
  David
 
  I have also posted this on Stackoverflow if anyone wants the points:
 
 http://stackoverflow.com/questions/20491090/what-is-the-fastest-way-to-get-data-into-cassandra-2-from-a-java-application
 
 
 
 
  --
  http://qdb.io/ Persistent Message Queues With Replay and #RabbitMQ
 Integration




-- 

- John

Re: calculating sizes on disk

[John Sanda]

I finally got the math right for the partition index after tracing through
SSTableWriter.IndexWriter.append(DecoratedKey key, RowIndexEntry
indexEntry). I should also note that I am working off of the source for
1.2.9. Here is the break down for what gets written to disk in the append()
call (my keys are 4 bytes  while column names and values are both 8 bytes).

// key
key length - 2 bytes
key - 4 bytes

// index entry
index entry position - 8 bytes
index entry size - 4 bytes

// when the index entry contains columns index, the following entries will
be written ceil(total_row_size / column_index_size_in_kb) times
local deletion time - 4 bytes
marked for delete at - 8 bytes
columns index entry first name length - 2 bytes
columns index entry first name - 8 bytes
columns index entry last name length - 2 bytes
columns index entry last name - 8 bytes
columns index entry offset - 8 bytes
columns index entry width - 8 bytes

I also went through the serialization code for bloom filters, but I do not
understand the math. Even with my slightly improved understanding, I am
still uncertain about how effective any sizing analysis will be since the
numbers of rows and columns will vary per SSTable.


On Fri, Dec 6, 2013 at 3:53 PM, John Sanda john.sa...@gmail.com wrote:

 I have done that, but it only gets me so far because the cluster and app
 that manages it is run by 3rd parties. Ideally, I would like to provide my
 end users with a formula or heuristic for establishing some sort of
 baselines that at least gives them a general idea for planning. Generating
 data as you have suggested and as I have done is helpful, but it is hard
 for users to extrapolate out from that.


 On Fri, Dec 6, 2013 at 3:47 PM, Jacob Rhoden jacob.rho...@me.com wrote:

 Not sure what your end setup will be, but I would probably just spin up a
 cluster and fill it with typical data to and measure the size on disk.

 __
 Sent from iPhone

 On 7 Dec 2013, at 6:08 am, John Sanda john.sa...@gmail.com wrote:

 I am trying to do some disk capacity planning. I have been referring the
 datastax docs[1] and this older blog post[2]. I have a column family with
 the following,

 row key - 4 bytes
 column name - 8 bytes
 column value - 8 bytes
 max number of non-deleted columns per row - 20160

 Is there an effective way to calculate the sizes (or at least a decent
 approximation) of the bloom filters and partition indexes on disk?

 [1] Calculating user data 
 sizehttp://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docsversion=1.2file=index#cassandra/architecture/../../cassandra/architecture/architecturePlanningUserData_t.html
 [2] Cassandra Storage Sizing http://btoddb-cass-storage.blogspot.com/

 --

 - John




 --

 - John




-- 

- John

calculating sizes on disk

[John Sanda]

I am trying to do some disk capacity planning. I have been referring the
datastax docs[1] and this older blog post[2]. I have a column family with
the following,

row key - 4 bytes
column name - 8 bytes
column value - 8 bytes
max number of non-deleted columns per row - 20160

Is there an effective way to calculate the sizes (or at least a decent
approximation) of the bloom filters and partition indexes on disk?

[1] Calculating user data
sizehttp://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docsversion=1.2file=index#cassandra/architecture/../../cassandra/architecture/architecturePlanningUserData_t.html
[2] Cassandra Storage Sizing http://btoddb-cass-storage.blogspot.com/

-- 

- John

Re: calculating sizes on disk

[John Sanda]

I should have also mentioned that I have tried using the calculations from
the storage sizing post. My lack of success may be due to the post basing
things off of Cassandra 0.8 as well as a lack of understanding in how to do
some of the calculations.


On Fri, Dec 6, 2013 at 3:08 PM, John Sanda john.sa...@gmail.com wrote:

 I am trying to do some disk capacity planning. I have been referring the
 datastax docs[1] and this older blog post[2]. I have a column family with
 the following,

 row key - 4 bytes
 column name - 8 bytes
 column value - 8 bytes
 max number of non-deleted columns per row - 20160

 Is there an effective way to calculate the sizes (or at least a decent
 approximation) of the bloom filters and partition indexes on disk?

 [1] Calculating user data 
 sizehttp://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docsversion=1.2file=index#cassandra/architecture/../../cassandra/architecture/architecturePlanningUserData_t.html
 [2] Cassandra Storage Sizing http://btoddb-cass-storage.blogspot.com/

 --

 - John




-- 

- John

Re: calculating sizes on disk

[John Sanda]

I have done that, but it only gets me so far because the cluster and app
that manages it is run by 3rd parties. Ideally, I would like to provide my
end users with a formula or heuristic for establishing some sort of
baselines that at least gives them a general idea for planning. Generating
data as you have suggested and as I have done is helpful, but it is hard
for users to extrapolate out from that.


On Fri, Dec 6, 2013 at 3:47 PM, Jacob Rhoden jacob.rho...@me.com wrote:

 Not sure what your end setup will be, but I would probably just spin up a
 cluster and fill it with typical data to and measure the size on disk.

 __
 Sent from iPhone

 On 7 Dec 2013, at 6:08 am, John Sanda john.sa...@gmail.com wrote:

 I am trying to do some disk capacity planning. I have been referring the
 datastax docs[1] and this older blog post[2]. I have a column family with
 the following,

 row key - 4 bytes
 column name - 8 bytes
 column value - 8 bytes
 max number of non-deleted columns per row - 20160

 Is there an effective way to calculate the sizes (or at least a decent
 approximation) of the bloom filters and partition indexes on disk?

 [1] Calculating user data 
 sizehttp://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docsversion=1.2file=index#cassandra/architecture/../../cassandra/architecture/architecturePlanningUserData_t.html
 [2] Cassandra Storage Sizing http://btoddb-cass-storage.blogspot.com/

 --

 - John




-- 

- John

reads and compression

[John Sanda]

This article[1] cites gains in read performance can be achieved when
compression is enabled. The more I thought about it, even after reading the
DataStax docs about reads[2], I realized I do not understand how
compression improves read performance. Can someone provide some details on
this?

Is the compression offsets map still used if compression is disabled for a
table? If so what is its rate of growth like as compared to the growth of
the map when compression is enabled?

[1] 
whats-new-in-cassandra-1-0-compressionhttp://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-compression
[2] about 
readshttp://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docsversion=1.2file=index#cassandra/dml/dml_about_reads_c.html

Thanks

- John

Re: Cassandra crashes

[John Sanda]

Check your file limits -
http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docsversion=1.2file=#cassandra/troubleshooting/trblshootInsufficientResources_r.html

On Friday, September 6, 2013, Jan Algermissen wrote:


 On 06.09.2013, at 13:12, Alex Major al3...@gmail.com javascript:;
 wrote:

  Have you changed the appropriate config settings so that Cassandra will
 run with only 2GB RAM? You shouldn't find the nodes go down.
 
  Check out this blog post
 http://www.opensourceconnections.com/2013/08/31/building-the-perfect-cassandra-test-environment/,
  it outlines the configuration settings needed to run Cassandra on 64MB
 RAM and might give you some insights.

 Yes, I have my fingers on the knobs and have also seen the article you
 mention - very helpful indeed. As well as the replies so far. Thanks very
 much.

 However, I still manage to kill 2 or 3 nodes of my 3-node cluster with my
 data import :-(

 Now, while it would be easy to scale out and up a bit until the default
 config of C* is sufficient, I really like to dive deep and try to
 understand why the thing is still going down, IOW, which of my config
 settings is so darn wrong that in most cases kill -9 remains the only way
 to shutdown the Java process in the end.


 The problem seems to be the heap size (set to MAX_HEAP_SIZE=640M   and
 HEAP_NEWSIZE=120M ) in combination with some cassandra activity that
 demands too much heap, right?

 So how do I find out what activity this is and how do I sufficiently
 reduce that activity.

 What bugs me in general is that AFAIU C* is so eager at giving massive
 write speed, that it sort of forgets to protect itself from client demand.
 I would very much like to understand why and how that happens.  I mean: no
 matter how many clients are flooding the database, it should not die due to
 out of memory situations, regardless of any configuration specifics, or?


 tl;dr

 Currently my client side (with java-driver) after a while reports more and
 more timeouts and then the following exception:

 com.datastax.driver.core.ex
 ceptions.DriverInternalError: An unexpected error occured server side:
 java.lang.OutOfMemoryError: unable
 to create new native thread ;

 On the server side, my cluster remains more or less in this condition:

 DN  x 71,33 MB   256 34,1%
  2f5e0b70-dbf4-4f37-8d5e-746ab76efbae  rack1
 UN  x  189,38 MB  256 32,0%  e6d95136-f102-49ce-81ea-72bd6a52ec5f
  rack1
 UN  x198,49 MB  256 33,9%
  0c2931a9-6582-48f2-b65a-e406e0bf1e56  rack1

 The host that is down (it is the seed host, if that matters) still shows
 the running java process, but I cannot shut down cassandra or connect with
 nodetool, hence kill -9 to the rescue.

 In that host, I still see a load of around 1.

 jstack -F lists 892 threads, all blocked, except for 5 inactive ones.


 The system.log after a few seconds of import shows the following exception:

 java.lang.AssertionError: incorrect row data size 771030 written to
 /var/lib/cassandra/data/products/product/products-product-tmp-ic-6-Data.db;
 correct is 771200
 at
 org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
 at
 org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)
 at
 org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
 at
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
 at
 org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
 at
 org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
 at
 org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
 at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at
 java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
 at java.util.concurrent.FutureTask.run(FutureTask.java:166)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:724)


 And then, after about 2 minutes there are out of memory errors:

  ERROR [CompactionExecutor:5] 2013-09-06 11:02:28,630 CassandraDaemon.java
 (line 192) Exception in thread Thread[CompactionExecutor
 :5,1,main]
 java.lang.OutOfMemoryError: unable to create new native thread
 at java.lang.Thread.start0(Native Method)
 at java.lang.Thread.start(Thread.java:693)
 at
 org.apache.cassandra.db.compaction.ParallelCompactionIterable$Deserializer.init(ParallelCompactionIterable.java:296)
 at
 org.apache.cassandra.db.compaction.ParallelCompactionIterable.iterator(ParallelCompactionIterable.java:73)
 at
 

Re: system.peers and decommissioned nodes

[John Sanda]

I did not see any errors during decommission. I have gone ahead though and
upgraded to 1.2.9.


On Wed, Sep 4, 2013 at 1:13 PM, Nate McCall n...@thelastpickle.com wrote:

 Did you get any exceptions on decommission? Does CASSANDRA-5857 sound
 related at all? (1.2.4 is a few revs behind now and there have been a few
 fixes there, so and upgrade might not hurt).


 On Tue, Aug 27, 2013 at 5:53 PM, John Sanda john.sa...@gmail.com wrote:

 Forgot to mention before, the host_id column is null for one of the rows.
 Running nodetool removenode on the other one failed. StorageService threw
 an exception because it could not find the host id (of the decommissioned
 node with the non-null host_id in system.peers).


 On Tuesday, August 27, 2013, John Sanda wrote:

 I had a 4 node cluster running C* 1.2.4. I am testing some client code
 for adding/removing nodes to/from the cluster. I decommissioned 3 nodes. I
 only have one node now; however, the system.peers table still has rows for
 two of the nodes that were decommissioned. nodetool status only reports the
 one remaining node.

 Will it be a problem still having those decommissioned nodes in
 system.peers? And if so, what should be done to resolve it?

 - John



 --

 - John





-- 

- John

system.peers and decommissioned nodes

[John Sanda]

I had a 4 node cluster running C* 1.2.4. I am testing some client code for
adding/removing nodes to/from the cluster. I decommissioned 3 nodes. I only
have one node now; however, the system.peers table still has rows for two
of the nodes that were decommissioned. nodetool status only reports the one
remaining node.

Will it be a problem still having those decommissioned nodes in
system.peers? And if so, what should be done to resolve it?

- John

Re: system.peers and decommissioned nodes

[John Sanda]

Forgot to mention before, the host_id column is null for one of the rows.
Running nodetool removenode on the other one failed. StorageService threw
an exception because it could not find the host id (of the decommissioned
node with the non-null host_id in system.peers).

On Tuesday, August 27, 2013, John Sanda wrote:

 I had a 4 node cluster running C* 1.2.4. I am testing some client code for
 adding/removing nodes to/from the cluster. I decommissioned 3 nodes. I only
 have one node now; however, the system.peers table still has rows for two
 of the nodes that were decommissioned. nodetool status only reports the one
 remaining node.

 Will it be a problem still having those decommissioned nodes in
 system.peers? And if so, what should be done to resolve it?

 - John



-- 

- John

Re: insert performance (1.2.8)

[John Sanda]

I'd suggest using prepared statements that you initialize at application
start up and switching to use Session.executeAsync coupled with Google
Guava Futures API to get better throughput on the client side.


On Mon, Aug 19, 2013 at 10:14 PM, Keith Freeman 8fo...@gmail.com wrote:

  Sure, I've tried different numbers for batches and threads, but generally
 I'm running 10-30 threads at a time on the client, each sending a batch of
 100 insert statements in every call, using the QueryBuilder.batch() API
 from the latest datastax java driver, then calling the Session.execute()
 function (synchronous) on the Batch.

 I can't post my code, but my client does this on each iteration:
 -- divides up the set of inserts by the number of threads
 -- stores the current time
 -- tells all the threads to send their inserts
 -- then when they've all returned checks the elapsed time

 At about 2000 rows for each iteration, 20 threads with 100 inserts each
 finish in about 1 second.  For 4000 rows, 40 threads with 100 inserts each
 finish in about 1.5 - 2 seconds, and as I said all 3 cassandra nodes have a
 heavy CPU load while the client is hardly loaded.  I've tried with 10
 threads and more inserts per batch, or up to 60 threads with fewer, doesn't
 seem to make a lot of difference.


 On 08/19/2013 05:00 PM, Nate McCall wrote:

  How big are the batch sizes? In other words, how many rows are you
 sending per insert operation?

  Other than the above, not much else to suggest without seeing some
 example code (on pastebin, gist or similar, ideally).

 On Mon, Aug 19, 2013 at 5:49 PM, Keith Freeman 8fo...@gmail.com wrote:

 I've got a 3-node cassandra cluster (16G/4-core VMs ESXi v5 on 2.5Ghz
 machines not shared with any other VMs).  I'm inserting time-series data
 into a single column-family using wide rows (timeuuids) and have a 3-part
 partition key so my primary key is something like ((a, b, day),
 in-time-uuid), x, y, z).

 My java client is feeding rows (about 1k of raw data size each) in
 batches using multiple threads, and the fastest I can get it run reliably
 is about 2000 rows/second.  Even at that speed, all 3 cassandra nodes are
 very CPU bound, with loads of 6-9 each (and the client machine is hardly
 breaking a sweat).  I've tried turning off compression in my table which
 reduced the loads slightly but not much.  There are no other updates or
 reads occurring, except the datastax opscenter.

 I was expecting to be able to insert at least 10k rows/second with this
 configuration, and after a lot of reading of docs, blogs, and google, can't
 really figure out what's slowing my client down.  When I increase the
 insert speed of my client beyond 2000/second, the server responses are just
 too slow and the client falls behind.  I had a single-node Mysql database
 that can handle 10k of these data rows/second, so I really feel like I'm
 missing something in Cassandra.  Any ideas?






-- 

- John

sstable_compression for system tables

[John Sanda]

Is there a way to change the sstable_compression for system tables? I am
trying to deploy Cassandra 1.2.2 on a platform with IBM Java and 32 bit
arch where the snappy-java native library fails to load. The error I get
looks like,

ERROR [SSTableBatchOpen:1] 2013-05-02 14:42:42,485
CassandraDaemon.java (line 132) Exception in thread
Thread[SSTableBatchOpen:1,5,main]
java.lang.RuntimeException: Cannot create CompressionParameters for
stored parameters
at 
org.apache.cassandra.io.compress.CompressionMetadata.init(CompressionMetadata.java:99)
at 
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at 
org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:51)
at 
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:404)
at 
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:198)
at 
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
at 
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:238)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:482)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:345)
at java.util.concurrent.FutureTask.run(FutureTask.java:177)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:626)
at java.lang.Thread.run(Thread.java:780)
Caused by: org.apache.cassandra.exceptions.ConfigurationException:
SnappyCompressor.create() threw an error:
java.lang.NoClassDefFoundError org.xerial.snappy.Snappy
(initialization failure)
at 
org.apache.cassandra.io.compress.CompressionParameters.createCompressor(CompressionParameters.java:179)
at 
org.apache.cassandra.io.compress.CompressionParameters.init(CompressionParameters.java:71)
at 
org.apache.cassandra.io.compress.CompressionMetadata.init(CompressionMetadata.java:95)
... 12 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at 
org.apache.cassandra.io.compress.CompressionParameters.createCompressor(CompressionParameters.java:156)
... 14 more
Caused by: java.lang.NoClassDefFoundError: org.xerial.snappy.Snappy
(initialization failure)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:176)
at 
org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor.java:45)
... 19 more
Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229)
at org.xerial.snappy.Snappy.clinit(Snappy.java:44)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:150)
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:366)
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:409)


I am not able to change sstable_compression for system tables from either
cassandra-cli or from cqlsh. I should point out that the DataStax docs do
state that system tables cannot be altered. I was wondering though if there
might be another way to do so.

Simply not using IBM Java is not an option for me. There is already an
issue[1] open with the snappy-java project that I think will address this
issue; however, that would involve packaging a new version of snappy-java
with Cassandra (when the fix is available). I would like to better
understand the impact of switching to a patched and/or upgraded version of
snappy-java before making that change.

[1] https://github.com/xerial/snappy-java/issues/34

Thanks

- John

Re: sstable_compression for system tables

[John Sanda]

The root cause was as I described. System tables were creating while
running OpenJDK. Files were written to disk using snappy compression.
Cassandra was later restarted with IBM Java. With the IBM JRE on a 32 bit
arch, the native snappy library is not found; consequently, Cassandra is
not able to read the SSTables on disk If IBM's JRE was used from the get
go, there would have been no SSTable compression and hence no error.


On Fri, May 3, 2013 at 5:28 PM, Robert Coli rc...@eventbrite.com wrote:

 On Fri, May 3, 2013 at 11:07 AM, John Sanda john.sa...@gmail.com wrote:
  The machine where this error occurred had both OpenJDK and IBM's Java
  installed. The only way I have been able to reproduce is by installing
  Cassandra with OpenJDK, shutting it down, the starting it back up with
 IBM
  Java.

 Maybe the root cause of the (non-repro)
 https://issues.apache.org/jira/browse/CASSANDRA-5059 ?

 =Rob




-- 

- John

question about internode_compression

[John Sanda]

When internode_compression is enabled, will the compression algorithm used
be the same as whatever I am using for sstable_compression?


- John

Re: DB Change management tools for Cassandra?

[John Sanda]

I had cobbled together a solution using Liquibase and the Cassandra JDBC
driver. I started implemented it before the CQL driver was announced. The
solution involved a patch and some Liquibase extensions which live at
https://github.com/jsanda/cassandra-liquibase-ext. The patch will go into
the 3.0 release of Liquibase. I developed the extensions against the 2.x
code base. Changes were introduced for Liquibase 3.0 that broken the
extensions I developed. At that point I decided to pursue a simpler
solution (though I think Liquibase is a great tool for relational
databases). Please update the list with any efforts as I am definitely
interested!


On Thu, Apr 25, 2013 at 10:19 AM, Brian O'Neill b...@alumni.brown.eduwrote:


 I haven't seen any, which has one of our developers (CC'd) looking at
 extending myBatis migrations and/or Flyway with CQL to do it.

 -brian

 ---
 Brian O'Neill
 Lead Architect, Software Development
 Health Market Science
 The Science of Better Results
 2700 Horizon Drive € King of Prussia, PA € 19406
 M: 215.588.6024 € @boneill42 http://www.twitter.com/boneill42  €
 healthmarketscience.com

 This information transmitted in this email message is for the intended
 recipient only and may contain confidential and/or privileged material. If
 you received this email in error and are not the intended recipient, or
 the person responsible to deliver it to the intended recipient, please
 contact the sender at the email above and delete this email and any
 attachments and destroy any copies thereof. Any review, retransmission,
 dissemination, copying or other use of, or taking any action in reliance
 upon, this information by persons or entities other than the intended
 recipient is strictly prohibited.







 On 4/25/13 7:35 AM, Marko Asplund marko.aspl...@gmail.com wrote:

 hi,
 
 Do database change management tools similar to Liquibase and dbdeploy
 exist for Cassandra?
 I need to handle change management for CQL3 schema.
 
 
 thanks,
 
 marko





-- 

- John

Re: Datatype Conversion in CQL-Client?

[John Sanda]

You might want to take  look a org.apache.cassandra.transport.SimpleClient
and org.apache.cassandra.transport.messages.ResultMessage.


On Mon, Nov 19, 2012 at 9:48 AM, Timmy Turner timm.t...@gmail.com wrote:

 What I meant was the method that the Cassandra-jars give you when you
 include them in your project:

   TTransport tr = new TFramedTransport(new TSocket(localhost, 9160));
   TProtocol proto = new TBinaryProtocol(tr);
   Cassandra.Client client = new Cassandra.Client(proto);
   tr.open();
   client.execute_cql_query(ByteBuffer.wrap(cql.getBytes()),
 Compression.NONE);



 2012/11/19 Brian O'Neill b...@alumni.brown.edu

 I don't think Michael and/or Jonathan have published the CQL java driver
 yet.  (CCing them)

 Hopefully they'll find a public home for it soon, I hope to include it in
 the Webinar in December.
 (http://www.datastax.com/resources/webinars/collegecredit)

 -brian

 ---

 Brian O'Neill

 Lead Architect, Software Development

 *Health Market Science*

 *The Science of Better Results*

 2700 Horizon Drive • King of Prussia, PA • 19406

 M: 215.588.6024 • @boneill42 http://www.twitter.com/boneill42  •

 healthmarketscience.com


 This information transmitted in this email message is for the intended
 recipient only and may contain confidential and/or privileged material. If
 you received this email in error and are not the intended recipient, or the
 person responsible to deliver it to the intended recipient, please contact
 the sender at the email above and delete this email and any attachments and
 destroy any copies thereof. Any review, retransmission, dissemination,
 copying or other use of, or taking any action in reliance upon, this
 information by persons or entities other than the intended recipient is
 strictly prohibited.

 ** **


 From: Tommi Laukkanen tlaukka...@gmail.com
 Reply-To: user@cassandra.apache.org
 Date: Monday, November 19, 2012 2:36 AM

 To: user@cassandra.apache.org
 Subject: Re: Datatype Conversion in CQL-Client?

 I think Timmy might be referring to the upcoming native CQL Java driver
 that might be coming with 1.2 - It was mentioned here:

 http://www.datastax.com/wp-content/uploads/2012/08/7_Datastax_Upcoming_Changes_in_Drivers.pdf

 I would also be interested on testing that but I can't find it from
 repositories. Any hints?

 Regards,
 Tommi L.

 *From:* Brian O'Neill [mailto:boneil...@gmail.com] *On Behalf Of *Brian
 O'Neill

 *Sent:* 18. marraskuuta 2012 17:47
 *To:* user@cassandra.apache.org
 *Subject:* Re: Datatype Conversion in CQL-Client?
 *Importance:* Low

 ** **

 ** **

 If you are talking about the CQL-client that comes with Cassandra
 (cqlsh), it is actually written in Python:

 https://github.com/apache/cassandra/blob/trunk/bin/cqlsh

 ** **

 For information on datatypes (and conversion) take a look at the CQL
 definition:

 http://www.datastax.com/docs/1.0/references/cql/index

 (Look at the CQL Data Types section)

 ** **

 If that's not the client you are referencing, let us know which one you
 mean:

 http://brianoneill.blogspot.com/2012/08/cassandra-apis-laundry-list.html
 

 ** **

 -brian

 ** **

 On Nov 17, 2012, at 9:54 PM, Timmy Turner wrote:



 

 Thanks for the links, however I'm interested in the functionality that
 the official Cassandra client/API (which is in Java) offers.

 ** **

 2012/11/17 aaron morton aa...@thelastpickle.com

 Does the official/built-in Cassandra CQL client (in 1.2) 

 What language ? 

 ** **

 Check the Java
 http://code.google.com/a/apache-extras.org/p/cassandra-jdbc/ and python
 http://code.google.com/a/apache-extras.org/p/cassandra-dbapi2/ drivers.*
 ***

 ** **

 Cheers

 ** **

 ** **

 -

 Aaron Morton

 Freelance Cassandra Developer

 New Zealand

 ** **

 @aaronmorton

 http://www.thelastpickle.com

 ** **

 On 16/11/2012, at 11:21 AM, Timmy Turner timm.t...@gmail.com wrote:***
 *



 

 Does the official/built-in Cassandra CQL client (in 1.2) offer any
 built-in option to get direct values/objects when reading a field, instead
 of just a byte array? 

 ** **

 ** **

 ** **

 --
 Brian ONeill
 Lead Architect, Health Market Science (http://healthmarketscience.com)
 mobile:215.588.6024
 blog: http://weblogs.java.net/blog/boneill42/
 blog: http://brianoneill.blogspot.com/ 

 ** **






-- 

- John

Re: Datastax Java Driver

[John Sanda]

Fantastic! As for the object mapping API, has there been any
discussion/consideration of http://www.hibernate.org/subprojects/ogm.html?


On Mon, Nov 19, 2012 at 1:50 PM, Sylvain Lebresne sylv...@datastax.comwrote:

 Everyone,

 We've just open-sourced a new Java driver we have been working on here at
 DataStax. This driver is CQL3 only and is built to use the new binary
 protocol
 that will be introduced with Cassandra 1.2. It will thus only work with
 Cassandra 1.2 onwards. Currently, it means that testing it requires
 1.2.0-beta2. This is also alpha software at this point. You are welcome to
 try
 and play with it and we would very much welcome feedback, but be sure that
 break, it will. The driver is accessible at:
   http://github.com/datastax/java-driver

 Today we're open-sourcing the core part of this driver. This main goal of
 this
 core module is to handle connections to the Cassandra cluster with all the
 features that one would expect. The currently supported features are:
   - Asynchronous: the driver uses the new CQL binary protocol asynchronous
 capabilities.
   - Nodes discovery.
   - Configurable load balancing/routing.
   - Transparent fail-over.
   - C* tracing handling.
   - Convenient schema access.
   - Configurable retry policy.

 This core module provides a simple low-level API (that works directly with
 query strings). We plan to release a higher-level, thin object mapping API
 based on top of this core shortly.

 Please refer to the project README for more information.

 --
 The DataStax Team




-- 

- John

distribution of token ranges with virtual nodes

[John Sanda]

I am not entirely clear on what
http://wiki.apache.org/cassandra/VirtualNodes/Balance#imbalance is saying
with respect to random vs. manual token selection. Can/should i assume that
i will get even range distribution or close to it with random token
selection? For the sake of discussion, what is a reasonable default to
start with for num_tokens assuming nodes are homogenous? That wiki page
mentions a default of 256 which I see commented out in cassandra.yaml;
however, Config.num_tokens is set to 1. Maybe I missed where the default of
256 is used. From some initial testing though, it looks like 1 token per
node is being used. Using defaults in cassandra.yaml, I see this in my logs,


WARN [main] 2012-10-31 12:06:48,591 StorageService.java (line 639)
Generated random token [-8703249769453332665]. Random tokens will result in
an unbalanced ring; see http://wiki.apache.org/cassandra/Operations

-- 

- John

CQL load balancing

[John Sanda]

Hector provides load balancing so that requests can be distributed across
cluster nodes based on a specified policy, like round robin. Is there
anything similar planned for CQL? I see that there is an open issue (
http://code.google.com/a/apache-extras.org/p/cassandra-jdbc/issues/detail?id=41)
to add support for pooled connections in cassandra-jdbc.Could load
balancing be incorporated into the connection pooling support? I have also
started reading up on the CQL transport protocol and am wondering if it
does or will provide anything for load balancing.

Thanks

- John

question about where clause of CQL update statement

[John Sanda]

I am using CQL 3 and trying to execute the following,

UPDATE CHANGELOGLOCK SET LOCKED = 'true', LOCKEDBY = '10.11.8.242
(10.11.8.242)', LOCKGRANTED = '2012-10-05 16:58:01' WHERE ID = 1 AND LOCKED
= 'false';


It gives me the error, Bad Request: PRIMARY KEY part locked found in SET
part. The primary key consists only of the ID column, but I do have a
secondary index on the locked column. Is it not possible to include a
column in both the set clause and in the where clause? And if it is not
possible, how come?

Thanks

- John

schema change management tools

[John Sanda]

I have been looking to see if there are any schema change management tools
for Cassandra. I have not come across any so far. I figured I would check
to see if anyone can point me to something before I start trying to
implement something on my own. I have used liquibase (
http://www.liquibase.org) for relational databases. Earlier today I tried
using it with the cassandra-jdbc driver, but ran into some exceptions due
to the SQL generated. I am not looking specifically for something
CQL-based. Something that uses the Thrift API via CLI scripts for example
would work as well.

Thanks

- John

Re: schema change management tools

[John Sanda]

For the project I work on and for previous projects as well that support
multiple upgrade paths, this kind of tooling is a necessity. And I would
prefer to avoid duplicating effort if there is already something out there.
If not though, I will be sure to post back to the list with whatever I wind
up doing.

On Thu, Oct 4, 2012 at 9:34 PM, Jonathan Haddad j...@jonhaddad.com wrote:

 Not that I know of.  I've always been really strict about dumping my
 schemas (to start) and keeping my changes in migration files.  I don't do a
 ton of schema changes so I haven't had a need to really automate it.

 Even with MySQL I never bothered.

 Jon


 On Thu, Oct 4, 2012 at 6:27 PM, John Sanda john.sa...@gmail.com wrote:

 I have been looking to see if there are any schema change management
 tools for Cassandra. I have not come across any so far. I figured I would
 check to see if anyone can point me to something before I start trying to
 implement something on my own. I have used liquibase (
 http://www.liquibase.org) for relational databases. Earlier today I
 tried using it with the cassandra-jdbc driver, but ran into some exceptions
 due to the SQL generated. I am not looking specifically for something
 CQL-based. Something that uses the Thrift API via CLI scripts for example
 would work as well.

 Thanks

 - John




 --
 Jon Haddad
 http://www.rustyrazorblade.com
 skype: rustyrazorblade