Understanding Blocked and All Time Blocked columns in tpstats

I have been doing some work on a cluster that is impacted by https://issues.apache.org/jira/browse/CASSANDRA-11363. Reading through the ticket prompted me to take a closer look at org.apache.cassandra.concurrent.SEPExecutor. I am looking at the 3.0.14 code. I am a little confused about the Blocked

Re: Understanding Blocked and All Time Blocked columns in tpstats

gt; (sep pool) last I checked. Since 2.1 at least, before that there were a few > others. That changes version to version. For (basically) all other thread > pools the queue is limited by memory. > > Chris > > > On Mar 22, 2018, at 10:44 PM, John Sanda wrote: > > I ha

Re: Understanding Blocked and All Time Blocked columns in tpstats

> On Mar 23, 2018, at 11:42 AM, John Sanda wrote: > > Thanks for the explanation. In the past when I have run into problems > related to CASSANDRA-11363, I have increased the queue size via the > cassandra.max_queued_native_transport_requests system property. If I find > that the q

multiple table directories for system_schema keyspace

On a couple different occasions I have run into this exception at start up: Exception (org.apache.cassandra.exceptions.InvalidRequestException) encountered during startup: Unknown type org.apache.cassandra.exceptions.InvalidRequestException: Unknown type at org.apache.cassandra.cql3.CQL3

Re: Commit Log sync problems

Hi Meg, I believe that the average duration reported is the total amount of time that exceeded the interval divided by the number of syncs that exceeded the interval. Cassandra is not complaining because commit log syncs took 0.66 ms but rather on average 4 commit log syncs 10060 ms. Cheers, Joh

Re: CassKop : a Cassandra operator for Kubernetes developped by Orange

There is also https://github.com/sky-uk/cassandra-operator On Fri, May 24, 2019 at 2:34 PM Rahul Singh wrote: > Fantastic! Now there are three teams making k8s operators for C*: > Datastax, Instaclustr, and now Orange. > > rahul.xavier.si...@gmail.com > > http://cassandra.link > > I'm speaking a

Re: Cassandra DataStax Java Driver in combination with Java EE / EJBs

Hi Ralph, A session is intended to be a long-lived, i.e., application-scoped object. You only need one session per cluster. I think what you are doing with the @Singleton is fine. In my opinion though, EJB really does not offer much value when working with Cassandra. I would be inclined to just us

Re: Rebuilding a node without clients hitting it

Assuming the rebuild is happening on a node in another DC, then there should not be an issue if you are using LOCAL_ONE. If the node is in the local DC (i.e., same DC as the client), I am inclined to think repair would be more appropriate than rebuild but I am not 100% certain. On Mon, Aug 5, 2019

Re: Cassandra on Kubernetes

One of the problems I have experienced in the past has more to do with Java than Cassandra in particular, and that is the JVM ignoring cgroups. With Cassandra in particular I would often see memory usage go higher than what was desired. This would lead to pods getting oom killed. This was fixed in

mutation checksum failure during commit log replay

What could cause an error like: ERROR 07:11:56 Exiting due to error while processing commit log during initialization. org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Mutation checksum failure at 818339 in CommitLog-5-1470234746867.log This is with Cassandra 2.2.4.

Client-side timeouts after dropping table

I am deploying multiple Java web apps that connect to a Cassandra 3.7 instance. Each app creates its own schema at start up. One of the schema changes involves dropping a table. I am seeing frequent client-side timeouts reported by the DataStax driver after the DROP TABLE statement is executed. I d

Re: Client-side timeouts after dropping table

Outside of a handful of highly > experienced experts using EBS in very specific ways, it usually ends in > failure. > > On Tue, Sep 20, 2016 at 3:30 PM John Sanda wrote: > >> I am deploying multiple Java web apps that connect to a Cassandra 3.7 >> instance. Each app cr

Re: Client-side timeouts after dropping table

2mb with 4mb > segments (or even lower depending on test data volume) so they basically > flush constantly and don't try to hold any tables open. Also lower > concurrent_writes substantially while you are at it to add some write > throttling. > > On Wed, Sep 21, 2016 at 2:14 PM,

Re: Client-side timeouts after dropping table

are multiple apps running, but it does happen fairly consistently almost immediately after the table is dropped. I don't see any indication of a server side timeout or any dropped mutations being reported in the log. On Tue, Sep 20, 2016 at 11:07 PM, John Sanda wrote: > Thanks Nate. W

commit log on NFS volume

I know that using NFS is discouraged, particularly for the commit log. Can anyone shed some light into what kinds of problems I might encounter aside from performance? The reason for my inquiry is because I have some deployments with Cassandra 2.2.1 that use NFS and are experiencing some problems l

Re: commit log on NFS volume

can be blocked, also in case of failure loss of data can occur. >> >> Best regards, Vladimir Yudovin, >> >> *Winguzone <https://winguzone.com?from=list> - Hosted Cloud >> CassandraLaunch your cluster in minutes.* >> >> >> On Tue, 01 Nov 20

empty buckets with STCS

I have 2.2.1 Cassandra node that does not appear to be compacting SSTables. The table is currently configured with STCS. I turned on some debug logging and when the compaction checks run, they log: Compaction buckets are [] I have been going over SizeTieredCompactionStrategy.java and looking in

Time series data model and tombstones

I have a time series data model that is basically: CREATE TABLE metrics ( id text, time timeuuid, value double, PRIMARY KEY (id, time) ) WITH CLUSTERING ORDER BY (time DESC); I do append-only writes, no deletes, and use a TTL of seven days. Data points are written every seconds. T

Re: Time series data model and tombstones

mbers that I am seeing. On Sat, Jan 28, 2017 at 12:15 PM, Jonathan Haddad wrote: > Since you didn't specify a compaction strategy I'm guessing you're using > STCS. Your TTL'ed data is becoming a tombstone. TWCS is a better strategy > for this type of workload. >

Re: Time series data model and tombstones

> > Your partitioning key is text. If you have multiple entries per id you are > likely hitting older cells that have expired. Descending only affects how > the data is stored on disk, if you have to read the whole partition to find > whichever time you are querying for you could potentially hit to

Re: Time series data model and tombstones

ation. > > On Sun, Jan 29, 2017 at 3:52 PM, John Sanda wrote: > > Your partitioning key is text. If you have multiple entries per id you are > likely hitting older cells that have expired. Descending only affects how > the data is stored on disk, if you have to read the whole partiti

Re: Time series data model and tombstones

Cassandra has > to scan over a lot tombstones to fetch the correct range of data thus your > issue > > On Sun, Jan 29, 2017 at 8:19 PM, John Sanda wrote: > >> It was with STCS. It was on a 2.x version before TWCS was available. >> >> On Sun, Jan 29, 2017 at 10:58 AM Du

Re: Time series data model and tombstones

pired data in SSTABLE-12 may be > compacted together with a new chunk of SSTABLE-2 containing fresh data so > in the new resulting SSTable will contain tombstones AND fresh data inside > the same partition, but of course sorted by clustering column "time". > > On Sun, Jan 29

Questions about TWCS

In Jeff Jirsa C* 2016 summit presentation, TimeWindowCompactionStrategy for Time Series Workloads, there is a slide which talks about optimizations. It says to align partition keys to your TWCS windows. Is it generally the case that calendar/date based partitions would align nicely with TWCS window

Re: Time series data model and tombstones

ilobytes worth of data (don't remember the exact value of xxx, maybe > 64k or far less) so you may end up reading tombstones. > > On Sun, Jan 29, 2017 at 9:24 PM, John Sanda wrote: > >> Thanks for the clarification. Let's say I have a partition in an SSTable >> wher

compaction falling behind

What is a good way to determine whether or not compaction is falling behind? I read a couple things earlier that suggest nodetool compactionstats might not be the most reliable thing to use. - John

Problems with large partitions and compaction

I have a table that uses LCS and has wound up with partitions upwards of 700 MB. I am seeing lots of the large partition warnings. Client requests are subsequently failing. The driver is not reporting timeout exception, just NoHostAvailableExceptions (in the logs I have reviewed so far). I know tha

partition sizes reported by nodetool tablehistograms

I am working on some issues involving really big partitions. I have been making extensive use of nodetool tablehistograms. What exactly is the partition size being reported? I have a table for which the max value reported is about 3.5 GB, but running du -h against the table data directory reports 5

Re: Migrate table data to another table

You might want to take a look at CQLSSTableWriter[1] in the Cassandra source tree. http://www.datastax.com/dev/blog/using-the-cassandra-bulk-loader-updated On Tue, Jun 30, 2015 at 1:18 PM, Umut Kocasaraç wrote: > Hi, > I want to change clustering order column of my table. As far as i know it >

Re: Example Data Modelling

25 MB seems very specific. Is there a reason why? On Tuesday, July 7, 2015, Peer, Oded wrote: > The data model suggested isn’t optimal for the “end of month” query you > want to run since you are not querying by partition key. > > The query would look like “select EmpID, FN, LN, basic from sala

Consistent reads and first write wins

Suppose I have the following schema, CREATE TABLE foo ( id text, time timeuuid, prop1 text, PRIMARY KEY (id, time) ) WITHCLUSTERING ORDER BY (time ASC); And I have two clients who execute quorum writes, e.g., // client 1 INSERT INTO FOO (id, time, prop1) VALUES ('test', , 'bar');

Re: Schema Versioning

If you are interested in a solution that maintains scripts, there are at least a few projects available, https://github.com/comeara/pillar - Runs on the JVM and written in Scala. Scripts are CQL files. https://github.com/Contrast-Security-OSS/cassandra-migration - Runs on JVM and I believe a port

Re: Forming a cluster of embedded Cassandra instances

The project I work on day to day uses an embedded instance of Cassandra, but it is intended for primarily for development. We embed Cassandra in a WildFly (i.e., JBoss) server. It is packaged and deployed as an EAR. I personally do not do this. I use and recommend ccm

Re: Forming a cluster of embedded Cassandra instances

, Feb 14, 2016 at 2:52 PM, Jack Krupansky wrote: > What motivated the use of an embedded instance for development - as > opposed to simply spawning a process for Cassandra? > > > > -- Jack Krupansky > > On Sun, Feb 14, 2016 at 2:05 PM, John Sanda wrote: > >> Th

schema change management tools

I have been looking to see if there are any schema change management tools for Cassandra. I have not come across any so far. I figured I would check to see if anyone can point me to something before I start trying to implement something on my own. I have used liquibase ( http://www.liquibase.org) f

Re: schema change management tools

ly automate it. > > Even with MySQL I never bothered. > > Jon > > > On Thu, Oct 4, 2012 at 6:27 PM, John Sanda wrote: > >> I have been looking to see if there are any schema change management >> tools for Cassandra. I have not come across any so far. I figured I woul

question about where clause of CQL update statement

I am using CQL 3 and trying to execute the following, UPDATE CHANGELOGLOCK SET LOCKED = 'true', LOCKEDBY = '10.11.8.242 (10.11.8.242)', LOCKGRANTED = '2012-10-05 16:58:01' WHERE ID = 1 AND LOCKED = 'false'; It gives me the error, Bad Request: PRIMARY KEY part locked found in SET part. The primar

CQL load balancing

Hector provides load balancing so that requests can be distributed across cluster nodes based on a specified policy, like round robin. Is there anything similar planned for CQL? I see that there is an open issue ( http://code.google.com/a/apache-extras.org/p/cassandra-jdbc/issues/detail?id=41) to a

distribution of token ranges with virtual nodes

I am not entirely clear on what http://wiki.apache.org/cassandra/VirtualNodes/Balance#imbalance is saying with respect to random vs. manual token selection. Can/should i assume that i will get even range distribution or close to it with random token selection? For the sake of discussion, what is a

Re: Datatype Conversion in CQL-Client?

You might want to take look a org.apache.cassandra.transport.SimpleClient and org.apache.cassandra.transport.messages.ResultMessage. On Mon, Nov 19, 2012 at 9:48 AM, Timmy Turner wrote: > What I meant was the method that the Cassandra-jars give you when you > include them in your project: > >

Re: Datastax Java Driver

Fantastic! As for the object mapping API, has there been any discussion/consideration of http://www.hibernate.org/subprojects/ogm.html? On Mon, Nov 19, 2012 at 1:50 PM, Sylvain Lebresne wrote: > Everyone, > > We've just open-sourced a new Java driver we have been working on here at > DataStax. T

manual deletes with TWCS

How problematic is it to perform deletes when using TWCS? I am currently using TWCS and have some new use cases for performing deletes. So far I have avoided performing deletes, but I am wondering what issues I might run into. - John

Re: manual deletes with TWCS

ll seem effective when running queries, space will never be reclaimed on > disk. > Deletes can be performed on TTLed data if needed, but the partition will > then exist in different time windows, which will postpone actual deletion > from disk until both time windows fully expire. > >

Netty SSL memory leak

I have Cassandra 3.0.9 cluster that is hitting OutOfMemoryErrors with byte buffer allocation. The stack trace looks like: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_131] at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)

Re: Cassandra - Nodes can't restart due to java.lang.OutOfMemoryError: Direct buffer memory

I am not sure which version of Netty is in 3.9, but maybe you are hitting https://issues.apache.org/jira/browse/CASSANDRA-13114. I hit this in Cassandra 3.0.9 which uses Netty 4.0.23. Here is the upstream netty ticket https://github.com/netty/netty/issues/3057. On Thu, Aug 31, 2017 at 10:15 AM, Ch

Not marking node down due to local pause

I have a small, two-node cluster running Cassandra 2.2.1. I am seeing a lot of these messages in both logs: WARN 07:23:16 Not marking nodes down due to local pause of 7219277694 > 50 I am fairly certain that they are not due to GC. I am not seeing a whole of GC being logged and nothing o

reads and compression

This article[1] cites gains in read performance can be achieved when compression is enabled. The more I thought about it, even after reading the DataStax docs about reads[2], I realized I do not understand how compression improves read performance. Can someone provide some details on this? Is the

Re: Nodetool cleanup

Couldn't another reason for doing cleanup sequentially be to avoid data loss? If data is being streamed from a node during bootstrap and cleanup is run too soon, couldn't you wind up in a situation with data loss if the new node being bootstrapped goes down (permanently)? On Thu, Nov 28, 2013 at

calculating sizes on disk

I am trying to do some disk capacity planning. I have been referring the datastax docs[1] and this older blog post[2]. I have a column family with the following, row key - 4 bytes column name - 8 bytes column value - 8 bytes max number of non-deleted columns per row - 20160 Is there an effective

Re: calculating sizes on disk

I should have also mentioned that I have tried using the calculations from the storage sizing post. My lack of success may be due to the post basing things off of Cassandra 0.8 as well as a lack of understanding in how to do some of the calculations. On Fri, Dec 6, 2013 at 3:08 PM, John Sanda

Re: calculating sizes on disk

re the size on disk. > > __ > Sent from iPhone > > On 7 Dec 2013, at 6:08 am, John Sanda wrote: > > I am trying to do some disk capacity planning. I have been referring the > datastax docs[1] and this older blog post[2]. I have a column family with &g

Re: calculating sizes on disk

SSTable. On Fri, Dec 6, 2013 at 3:53 PM, John Sanda wrote: > I have done that, but it only gets me so far because the cluster and app > that manages it is run by 3rd parties. Ideally, I would like to provide my > end users with a formula or heuristic for establishing some sort of > ba

Re: What is the fastest way to get data into Cassandra 2 from a Java application?

The session.execute blocks until the C* returns the response. Use the async version, but do so with caution. If you don't throttle the requests, you will start seeing timeouts on the client side pretty quickly. For throttling I've used a Semaphore, but I think Guava's RateLimiter is better suited.

Re: user / password authentication advice

You could use CassandraAuthorizer and PaaswordAuthenticator which ships with Cassandra. See this article[1] for a good overview. [1] http://www.datastax.com/dev/blog/a-quick-tour-of-internal-authentication-and-authorization-security-in-datastax-enterprise-and-apache-cassandra On Thursday, Decembe

Re: Performance problem with large wide row inserts using CQL

>From a quick glance at your code, it looks like you are preparing your insert statement multiple times. You only need to prepare it once. I would expect to see some improvement with that change. On Wed, Feb 19, 2014 at 5:27 AM, Rüdiger Klaehn wrote: > Hi all, > > I am evaluating Cassandra for

Large SSTable not compacted with size tiered compaction

I have a write-heavy table that is using size tiered compaction. I am running C* 1.2.9. There is an SSTable that is not getting compacted. It is disproportionately larger than the other SSTables. The data file sizes are, 1.70 GB 0.18 GB 0.16 GB 0.05 GB 8.61 GB If I set the bucket_high compaction

When will a node's host ID change?

Under what circumstances, if any, will a node's host ID change? - John

Re: DB Change management tools for Cassandra?

I had cobbled together a solution using Liquibase and the Cassandra JDBC driver. I started implemented it before the CQL driver was announced. The solution involved a patch and some Liquibase extensions which live at https://github.com/jsanda/cassandra-liquibase-ext. The patch will go into the 3.0

question about internode_compression

When internode_compression is enabled, will the compression algorithm used be the same as whatever I am using for sstable_compression? - John

sstable_compression for system tables

Is there a way to change the sstable_compression for system tables? I am trying to deploy Cassandra 1.2.2 on a platform with IBM Java and 32 bit arch where the snappy-java native library fails to load. The error I get looks like, ERROR [SSTableBatchOpen:1] 2013-05-02 14:42:42,485 CassandraDaemon.j

Re: sstable_compression for system tables

a static object static object that > requires initialization even though it is not used? > > > On Fri, May 3, 2013 at 10:19 AM, John Sanda wrote: > >> Is there a way to change the sstable_compression for system tables? I am >> trying to deploy Cassandra 1.2.2 on a

Re: sstable_compression for system tables

m the get go, tables are created with compression disabled as expected. System tables though are in fact created with snappy compression if it is available. On Fri, May 3, 2013 at 12:35 PM, John Sanda wrote: > I am still trying to sort this out. When I run with Oracle's JRE, it does &

Re: sstable_compression for system tables

read the SSTables on disk If IBM's JRE was used from the get go, there would have been no SSTable compression and hence no error. On Fri, May 3, 2013 at 5:28 PM, Robert Coli wrote: > On Fri, May 3, 2013 at 11:07 AM, John Sanda wrote: > > The machine where this error occurred ha

Re: insert performance (1.2.8)

I'd suggest using prepared statements that you initialize at application start up and switching to use Session.executeAsync coupled with Google Guava Futures API to get better throughput on the client side. On Mon, Aug 19, 2013 at 10:14 PM, Keith Freeman <8fo...@gmail.com> wrote: > Sure, I've t

conflict resolution in range scans

How is conflict resolution done with range scans? I understand when reading a single column, the latest timestamp wins. Is the timestamp for each column compared with a range scan such that some columns in the result could come from one replica while other columns come from another? - John

system.peers and decommissioned nodes

I had a 4 node cluster running C* 1.2.4. I am testing some client code for adding/removing nodes to/from the cluster. I decommissioned 3 nodes. I only have one node now; however, the system.peers table still has rows for two of the nodes that were decommissioned. nodetool status only reports the on

Re: system.peers and decommissioned nodes

, 2013, John Sanda wrote: > I had a 4 node cluster running C* 1.2.4. I am testing some client code for > adding/removing nodes to/from the cluster. I decommissioned 3 nodes. I only > have one node now; however, the system.peers table still has rows for two > of the nodes that were de

Re: system.peers and decommissioned nodes

a few > fixes there, so and upgrade might not hurt). > > > On Tue, Aug 27, 2013 at 5:53 PM, John Sanda wrote: > >> Forgot to mention before, the host_id column is null for one of the rows. >> Running nodetool removenode on the other one failed. StorageService threw >

Re: Cassandra crashes

Check your file limits - http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docs&version=1.2&file=#cassandra/troubleshooting/trblshootInsufficientResources_r.html On Friday, September 6, 2013, Jan Algermissen wrote: > > On 06.09.2013, at 13:12, Alex Major > > wrote: >

Re: slurm for cluster job scheduling and coordination

> > I've been working towards organizing an effort around using Kubernetes for > cluster management. There is a lot of work to do but this could be > something really important to tackle as a community if you(or anyone else) > are interested in getting involved. > This is a big area of interest fo

Re: Minimum System Requirements

I recently had to set up an integration testing environment that involves running multiple C* instances in containers using docker-compose. I am able to do so with a total memory for the container set at 512 MB and a 256 MB heap for C*. This is with C* 3.11.4. Going below 512 MB causes the containe

Re: Question on cass-operator

Hi Manu, The 2/2 indicates that there are two containers and each is in the ready state. As Vishal suggested, run kubectl describe pod to get more details. You also use kubectl get pod -o yaml. The former will include events in the output. You can run nodetool commands like this: $ kubectl -n c

Re: What is the way to scale down Cassandra/Kubernetes cluster from 3 to 1 nodes using cass-operator

Cass Operator currently does not support scaling down. Thanks John On Thu, Jul 2, 2020 at 1:02 PM Manu Chadha wrote: > Hi > > > > I changed the file and applied it but the new configuration hasn’t got > applied. > > > > > > metadata: > > name: dc1 > > spec: > > clusterName: cluster1 > >

Re: Replacing a Cassandra node in K8S

The Cassandra pod will get scheduled to run on a different worker node, provided there is an available node that satisfies affinity rules, resource requirements, etc. And you are correct that the volume will get remounted. If however you are using a local or hostPath volume, then it will be lost an

Re: Connect java application to Cassandra in Kubernetes

Hi Pushpendra You should use the headless service, e.g., // Note that this code snippet is using v3.x of the driver. // Assume the service is deployed in namespace dev and is // named cassandra-service. The FQDN of the service would then // be cassandra-service.dev.svc.cluster.local. If your clie

Re: Connect java application to Cassandra in Kubernetes

the client container when the liveness probe fails. And if you configure the driver to connect via the headless service, you will get the update endpoints. On Thu, Aug 6, 2020 at 11:00 PM John Sanda wrote: > Hi Pushpendra > > You should use the headless service, e.g., > > // Note

Re: backup data with Cass-operator and Medusa

Hi Manu I have started doing the preliminary work. I have a PR[1] for Medusa that needs to be reviewed/merged. I have done some work[2] for initial, low-level integration directly against statefulsets. I hope to have medusa integration land within the next couple releases of cass-operator. [1] ht