Re: best practices for time-series data with massive amounts of records
Hi all, Thanks for the responses, this was very helpful. I don't know yet what the distribution of clicks and users will be, but I expect to see a few users with an enormous amount of interactions and most users having very few. The idea of doing some additional manual partitioning, and then maintaining another table that contains the head partition for each user makes sense, although it would add additional latency when we want to get say the most recent 1000 interactions for a given user (which is something that we have to do sometimes for applications with tight SLAs). FWIW I doubt that any users will have so many interactions that they exceed what we could reasonably put in a row, but I wanted to have a strategy to deal with this. Having a nice design pattern in Cassandra for maintaining a row with the N-most-recent interactions would also solve this reasonably well, but I don't know of any way to implement that without running batch jobs that periodically clean out data (which might be okay). Best regards, Clint On Tue, Mar 3, 2015 at 8:10 AM, mck m...@apache.org wrote: Here partition is a random digit from 0 to (N*M) where N=nodes in cluster, and M=arbitrary number. Hopefully it was obvious, but here (unless you've got hot partitions), you don't need N. ~mck
best practices for time-series data with massive amounts of records
Hi all, I am designing an application that will capture time series data where we expect the number of records per user to potentially be extremely high. I am not sure if we will eclipse the max row size of 2B elements, but I assume that we would not want our application to approach that size anyway. If we wanted to put all of the interactions in a single row, then I would make a data model that looks like: CREATE TABLE events ( id text, event_time timestamp, event blob, PRIMARY KEY (id, event_time)) WITH CLUSTERING ORDER BY (event_time DESC); The best practice for breaking up large rows of time series data is, as I understand it, to put part of the time into the partitioning key ( http://planetcassandra.org/getting-started-with-time-series-data-modeling/): CREATE TABLE events ( id text, date text, // Could also use year+month here or year+week or something else event_time timestamp, event blob, PRIMARY KEY ((id, date), event_time)) WITH CLUSTERING ORDER BY (event_time DESC); The downside of this approach is that we can no longer do a simple continuous scan to get all of the events for a given user. Some users may log lots and lots of interactions every day, while others may interact with our application infrequently, so I'd like a quick way to get the most recent interaction for a given user. Has anyone used different approaches for this problem? The only thing I can think of is to use the second table schema described above, but switch to an order-preserving hashing function, and then manually hash the id field. This is essentially what we would do in HBase. Curious if anyone else has any thoughts. Best regards, Clint
Re: how to scan all rows of cassandra using multiple threads
Hi Gaurav, I recommend you just run a MapReduce job for this computation. Alternatively, you can look at the code for the C* MapReduce input format: https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/hadoop/cql3/CqlInputFormat.java That should give you what you need to iterate over independent token ranges. If you want, you can also just divide up the total token range for the partitioner you are using into equal chunks and have each of your threads execute a separate scan. Best regards, Clint On Tue, Feb 24, 2015 at 9:50 AM, Gaurav Bhatnagar gauravb...@gmail.com wrote: Hi, I have a cassandra cluster of 3 nodes holding around 300 million rows of items. I have a replication factor of 3 with read/write consistency as Quorum. I want to scan all rows of database to generate sum of items having value available in column name state and value batch1 in column name batch. Row key for item is a 15 digit random number. I want to do this processing in multiple threads for instance one thread generating sum for one portion of data and other thread generating sum for another disjoint portion of data and later I would add up total from these 2 threads to get final sum. What can be the possible way to achieve this? Can I use concept of virtual nodes here. Each node owns set of virtual nodes. Can I get data owned by a particular node and this way generate sum on different nodes by iterating over data from virtual nodes and later generate total sum by doing sum of data from all virtual nodes. Regards, Gaurav
Re: Why no virtual nodes for Cassandra on EC2?
Hi mck, I'm not familiar with this ticket, but my understanding was that performance of Hadoop jobs on C* clusters with vnodes was poor because a given Hadoop input split has to run many individual scans (one for each vnode) rather than just a single scan. I've run C* and Hadoop in production with a custom input format that used vnodes (and just combined multiple vnodes in a single input split) and didn't have any issues (the jobs had many other performance bottlenecks besides starting multiple scans from C*). This is one of the videos where I recall an off-hand mention of the Spark connector working with vnodes: https://www.youtube.com/watch?v=1NtnrdIUlg0 Best regards, Clint On Sat, Feb 21, 2015 at 2:58 PM, mck m...@apache.org wrote: At least the problem of hadoop and vnodes described in CASSANDRA-6091 doesn't apply to spark. (Spark already allows multiple token ranges per split). If this is the reason why DSE hasn't enabled vnodes then fingers crossed that'll change soon. Some of the DataStax videos that I watched discussed how the Cassandra Spark connecter has optimizations to deal with vnodes. Are these videos public? if so got any link to them? ~mck
Re: Running Cassandra + Spark on AWS - architecture questions
These are both good suggestions, thanks! I thought I had remembered reading that different virtual datacenters should always have the same number of nodes. I think I was mistaken about that. In the past we had major issues running huge analytics jobs on data stored in HBase (it would bring down our real-time performance), so this capability of Cassandra is great! Best regards, Clint On Sun, Feb 22, 2015 at 8:02 AM, Eric Stevens migh...@gmail.com wrote: I'm not sure if this is a good use case for you, but you might also consider setting up several keyspaces, one for any data you want available for analytics (such as business object tables), and one for data you don't want to do analytics on (such as custom secondary indices). Maybe a third one for data which should only exist in the analytics space, such as for temporary rollup data. This can reduce the amount of data you replicate into your analytics space, and allow you to run a smaller analytics cluster than your production cluster. On Fri, Feb 20, 2015 at 2:43 PM, DuyHai Doan doanduy...@gmail.com wrote: Cassandra would take care of keeping the data synced between the two sets of five nodes. Is that correct? Correct But doing so means that we need 2x as many nodes as we need for the real-time cluster alone Not necessarily. With multi DC you can configure the replication factor value per DC, meaning that you can have RF = 3 for the real time DC and RF=1 or RF=2 for the analytics DC. Thus the number of nodes can be different for each DC In addition, you can also tune the hardware. If the realtime DC is mostly write only for incoming data and read-only from aggregated table, it is less IO intensive than the analytics DC with lot of read write to compute aggregations. On Fri, Feb 20, 2015 at 10:17 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi all, I read the DSE 4.6 documentation and I'm still not 100% sure what a mixed workload Cassandra + Spark installation would look like, especially on AWS. What I gather is that you use OpsCenter to set up the following: - One virtual data center for real-time processing (e.g., ingestion of time-series data, replying to requests for an interactive application) - Another virtual data center for batch analytics (Spark, possibly for machine learning) If I understand this correctly, if I estimate that I need a five-node cluster to handle all of my data, under the system described above, I would have five nodes serving real-time traffic and all of the data replicated in another five nodes that I use for batch processing. Cassandra would take care of keeping the data synced between the two sets of five nodes. Is that correct? I assume the motivation for such a dual-virtual-data-center architecture is to prevent the Spark jobs (which are going to do lots of scans from Cassandra, and maybe run computation on the same machines hosting Cassandra) from disrupting the real-time performance. But doing so means that we need 2x as many nodes as we need for the real-time cluster alone. *Could someone confirm that my interpretation above of what I read about in the DSE documentation is correct?* If my application needs to run analytics on Spark only a few hours a day, might we be better off spending our money to get a bigger Cassandra cluster and then just spin up Spark jobs on EMR for a few hours at night? (I know this is a hard question to answer, since it all depends on the application---just curious if anyone else here has had to make similar tradeoffs.) e.g., maybe instead of having a five-node real-time cluster, we would have an eight-node real-time cluster, and use our remaining budget on EMR jobs. I am curious if anyone has any thoughts / experience about this. Best regards, Clint
Any notion of unions in C* user-defined types?
Hi all, I am building an application that keeps a time-series record of clickstream data (clicks, impressions, etc.). The data model looks something like: CREATE TABLE clickstream ( userid text, event_time timestamp, interaction frozen interaction_type, PRIMARY KEY (userid, timestamp) ) WITH CLUSTERING ORDER BY (event_time DESC); I would like to create a user-defined type interaction_type such that it can be different depending on whether the interaction was a click, view, etc. Previously we encoded such data with Avro, using Avro's unions ( http://avro.apache.org/docs/1.7.5/idl.html#unions) and encoded the data as blobs. I was hoping to get away from blobs now that we have UDTs in Cassandra 2.1, but I don't see any support for unions. Does anyone have any suggestions? I think I may be better of just sticking with Avro serialization. :( Best regards, Clint
Running Cassandra + Spark on AWS - architecture questions
Hi all, I read the DSE 4.6 documentation and I'm still not 100% sure what a mixed workload Cassandra + Spark installation would look like, especially on AWS. What I gather is that you use OpsCenter to set up the following: - One virtual data center for real-time processing (e.g., ingestion of time-series data, replying to requests for an interactive application) - Another virtual data center for batch analytics (Spark, possibly for machine learning) If I understand this correctly, if I estimate that I need a five-node cluster to handle all of my data, under the system described above, I would have five nodes serving real-time traffic and all of the data replicated in another five nodes that I use for batch processing. Cassandra would take care of keeping the data synced between the two sets of five nodes. Is that correct? I assume the motivation for such a dual-virtual-data-center architecture is to prevent the Spark jobs (which are going to do lots of scans from Cassandra, and maybe run computation on the same machines hosting Cassandra) from disrupting the real-time performance. But doing so means that we need 2x as many nodes as we need for the real-time cluster alone. *Could someone confirm that my interpretation above of what I read about in the DSE documentation is correct?* If my application needs to run analytics on Spark only a few hours a day, might we be better off spending our money to get a bigger Cassandra cluster and then just spin up Spark jobs on EMR for a few hours at night? (I know this is a hard question to answer, since it all depends on the application---just curious if anyone else here has had to make similar tradeoffs.) e.g., maybe instead of having a five-node real-time cluster, we would have an eight-node real-time cluster, and use our remaining budget on EMR jobs. I am curious if anyone has any thoughts / experience about this. Best regards, Clint
Re: Why no virtual nodes for Cassandra on EC2?
BTW are the performance concerns with vnodes a big deal for Spark? Or were those more important for MapReduce? Some of the DataStax videos that I watched discussed how the Cassandra Spark connecter has optimizations to deal with vnodes. I would imagine that Spark's ability to cache RDDs would mean that paying a small efficiency cost when reading data out of Cassandra initially might not be the end of the world (especially given the benefits of using vnodes). On Fri, Feb 20, 2015 at 8:29 AM, Clint Kelly clint.ke...@gmail.com wrote: Hi Mark, Thanks for your reply. That makes sense. I recall looking at this back when we were going to run Hadoop against data in Cassandra tables at my previous company. Disabling virtual nodes seems unfortunate as it would make (as I understand it) scaling the cluster a lot trickier. I assume there is a tradeoff between the performance of analytics jobs and the ease with which you can change cluster size. -Clint On Fri, Feb 20, 2015 at 1:01 AM, Mark Reddy mark.l.re...@gmail.com wrote: Hey Clint, Someone for DataStax can correct me here, but I'm assuming that they have disabled vnodes because the AMI is built to make it easy to set up a pre-configured mixed workload cluster. A mixture of Real-Time/Transactional (Cassandra), Analytics (Hadoop), or Search (Solr). If you take a look at the getting started guide for both Hadoop and Solr you will see a paragraph instructing the user to disable vnodes for a mix workload cluster. http://www.datastax.com/documentation/datastax_enterprise/4.0/datastax_enterprise/srch/srchIntro.html http://www.datastax.com/documentation/datastax_enterprise/4.0/datastax_enterprise/ana/anaStrt.html This is specific to the example AMI and that type of workload. This is by no means a warning for users to disable vnodes on their Real-Time/Transactional Cassandra only clusters on EC2. I've used vnodes on EC2 without issue. Regards, Mark On 20 February 2015 at 05:08, Clint Kelly clint.ke...@gmail.com wrote: Hi all, The guide for installing Cassandra on EC2 says that Note: The DataStax AMI does not install DataStax Enterprise nodes with virtual nodes enabled. http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installAMI.html Just curious why this is the case. It was my understanding that virtual nodes make taking Cassandra nodes on and offline an easier process, and that seems like something that an EC2 user would want to do quite frequently. -Clint
AMI to use to launch a cluster with OpsCenter on AWS
Hi all, I am trying to follow the instructions here for installing DSE 4.6 on AWS: http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installAMIOpsc.html I was successful creating a single-node instance running OpsCenter, which I intended to bootstrap creating a larger cluster running Cassandra and Spark. During my first attempt, however, OpsCenter reported problems talking to agents in the new cluster I was creating. I ssh'ed into one of the new instances that I created with OpsCenter and saw that this was the problem: DataStax AMI for DataStax Enterprise and DataStax Community AMI version 2.4 DataStax AMI 2.5 released 02.25.2014 http://goo.gl/g1RRd7 This AMI (version 2.4) will be left available, but no longer updated. These notices occurred during the startup of this instance: [ERROR] 02/21/15-00:53:01 sudo chown -R cassandra:cassandra /mnt/cassandra: [WARN] Permissions not set correctly. Please run manually: [WARN] sudo chown -hR cassandra:cassandra /mnt/cassandra [WARN] sudo service dse restart It looks like by default, the OpsCenter GUI selects an out-of-date AMI (ami-4c32ba7c) when you click on Create Cluster and attempt to create a brand-new cluster on EC2. What is the recommended image to use here? I found a version 2.5.1 of the autoclustering AMI ( http://thecloudmarket.com/image/ami-ada2b6c4--datastax-auto-clustering-ami-2-5-1-hvm). Is that correct? Or should I be using one of the regular AMIs listed at http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installAMIOpsc.html ? Or just a standard ubuntu image? FWIW, I tried just using one of the AMIs listed on the DSE 4.6 page (ami-32f7c977), and I still see the Waiting for the agent to start message, although if I log in, things look like they have kind of worked: Cluster started with these options: None Raiding complete. Waiting for nodetool... The cluster is now in it's finalization phase. This should only take a moment... Note: You can also use CTRL+C to view the logs if desired: AMI log: ~/datastax_ami/ami.log Cassandra log: /var/log/cassandra/system.log Datacenter: us-west-2 = Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.28.24.19 51.33 KB 256 100.0% ce0d365a-d58b-4700-b861-9f30af400476 2a Opscenter: http://ec2-54-71-102-180.us-west-2.compute.amazonaws.com:/ Please wait 60 seconds if this is the cluster's first start... Tools: Run: datastax_tools Demos: Run: datastax_demos Support: Run: datastax_support DataStax AMI for DataStax Enterprise and DataStax Community AMI version 2.5 DataStax Community version 2.1.3-1 These notices occurred during the startup of this instance: [ERROR] 02/21/15-01:18:55 sudo chown opscenter-agent:opscenter-agent /var/lib/datastax-agent/conf: [ERROR] 02/21/15-01:19:04 sudo chown -R opscenter-agent:opscenter-agent /var/log/datastax-agent: [ERROR] 02/21/15-01:19:04 sudo chown -R opscenter-agent:opscenter-agent /mnt/datastax-agent: I would appreciate any help... I assume what I'm trying to do here is pretty common.
Re: AMI to use to launch a cluster with OpsCenter on AWS
BTW I was able to use this script: https://github.com/joaquincasares/cassandralauncher to get a cluster up and running pretty easily on AWS. Cheers to the author for this. Still curious for answers to my questions above, but not as urgent. Best regards, Clint On Fri, Feb 20, 2015 at 5:36 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi all, I am trying to follow the instructions here for installing DSE 4.6 on AWS: http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installAMIOpsc.html I was successful creating a single-node instance running OpsCenter, which I intended to bootstrap creating a larger cluster running Cassandra and Spark. During my first attempt, however, OpsCenter reported problems talking to agents in the new cluster I was creating. I ssh'ed into one of the new instances that I created with OpsCenter and saw that this was the problem: DataStax AMI for DataStax Enterprise and DataStax Community AMI version 2.4 DataStax AMI 2.5 released 02.25.2014 http://goo.gl/g1RRd7 This AMI (version 2.4) will be left available, but no longer updated. These notices occurred during the startup of this instance: [ERROR] 02/21/15-00:53:01 sudo chown -R cassandra:cassandra /mnt/cassandra: [WARN] Permissions not set correctly. Please run manually: [WARN] sudo chown -hR cassandra:cassandra /mnt/cassandra [WARN] sudo service dse restart It looks like by default, the OpsCenter GUI selects an out-of-date AMI (ami-4c32ba7c) when you click on Create Cluster and attempt to create a brand-new cluster on EC2. What is the recommended image to use here? I found a version 2.5.1 of the autoclustering AMI ( http://thecloudmarket.com/image/ami-ada2b6c4--datastax-auto-clustering-ami-2-5-1-hvm). Is that correct? Or should I be using one of the regular AMIs listed at http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installAMIOpsc.html ? Or just a standard ubuntu image? FWIW, I tried just using one of the AMIs listed on the DSE 4.6 page (ami-32f7c977), and I still see the Waiting for the agent to start message, although if I log in, things look like they have kind of worked: Cluster started with these options: None Raiding complete. Waiting for nodetool... The cluster is now in it's finalization phase. This should only take a moment... Note: You can also use CTRL+C to view the logs if desired: AMI log: ~/datastax_ami/ami.log Cassandra log: /var/log/cassandra/system.log Datacenter: us-west-2 = Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.28.24.19 51.33 KB 256 100.0% ce0d365a-d58b-4700-b861-9f30af400476 2a Opscenter: http://ec2-54-71-102-180.us-west-2.compute.amazonaws.com:/ Please wait 60 seconds if this is the cluster's first start... Tools: Run: datastax_tools Demos: Run: datastax_demos Support: Run: datastax_support DataStax AMI for DataStax Enterprise and DataStax Community AMI version 2.5 DataStax Community version 2.1.3-1 These notices occurred during the startup of this instance: [ERROR] 02/21/15-01:18:55 sudo chown opscenter-agent:opscenter-agent /var/lib/datastax-agent/conf: [ERROR] 02/21/15-01:19:04 sudo chown -R opscenter-agent:opscenter-agent /var/log/datastax-agent: [ERROR] 02/21/15-01:19:04 sudo chown -R opscenter-agent:opscenter-agent /mnt/datastax-agent: I would appreciate any help... I assume what I'm trying to do here is pretty common.
Re: Why no virtual nodes for Cassandra on EC2?
Hi Mark, Thanks for your reply. That makes sense. I recall looking at this back when we were going to run Hadoop against data in Cassandra tables at my previous company. Disabling virtual nodes seems unfortunate as it would make (as I understand it) scaling the cluster a lot trickier. I assume there is a tradeoff between the performance of analytics jobs and the ease with which you can change cluster size. -Clint On Fri, Feb 20, 2015 at 1:01 AM, Mark Reddy mark.l.re...@gmail.com wrote: Hey Clint, Someone for DataStax can correct me here, but I'm assuming that they have disabled vnodes because the AMI is built to make it easy to set up a pre-configured mixed workload cluster. A mixture of Real-Time/Transactional (Cassandra), Analytics (Hadoop), or Search (Solr). If you take a look at the getting started guide for both Hadoop and Solr you will see a paragraph instructing the user to disable vnodes for a mix workload cluster. http://www.datastax.com/documentation/datastax_enterprise/4.0/datastax_enterprise/srch/srchIntro.html http://www.datastax.com/documentation/datastax_enterprise/4.0/datastax_enterprise/ana/anaStrt.html This is specific to the example AMI and that type of workload. This is by no means a warning for users to disable vnodes on their Real-Time/Transactional Cassandra only clusters on EC2. I've used vnodes on EC2 without issue. Regards, Mark On 20 February 2015 at 05:08, Clint Kelly clint.ke...@gmail.com wrote: Hi all, The guide for installing Cassandra on EC2 says that Note: The DataStax AMI does not install DataStax Enterprise nodes with virtual nodes enabled. http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installAMI.html Just curious why this is the case. It was my understanding that virtual nodes make taking Cassandra nodes on and offline an easier process, and that seems like something that an EC2 user would want to do quite frequently. -Clint
Why no virtual nodes for Cassandra on EC2?
Hi all, The guide for installing Cassandra on EC2 says that Note: The DataStax AMI does not install DataStax Enterprise nodes with virtual nodes enabled. http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installAMI.html Just curious why this is the case. It was my understanding that virtual nodes make taking Cassandra nodes on and offline an easier process, and that seems like something that an EC2 user would want to do quite frequently. -Clint
Re: No schema agreement from live replicas?
FWIW increasing the threshold for withMaxSchemaAgreementWaitSeconds to 30sec was enough to fix my problem---I would like to understand whether the cluster has some kind of configuration problem that made doing so necessary, however. Thanks! On Tue, Feb 3, 2015 at 7:44 AM, Clint Kelly clint.ke...@gmail.com wrote: Hi all, I have an application that uses the Java driver to create a table and then immediately write to it. I see the following warning in my logs: [10.241.17.134] out: 15/02/03 09:32:24 WARN com.datastax.driver.core.Cluster: No schema agreement from live replicas after 10 s. The schema may not be up to date on some nodes. ...this seems to happen after creating a table, and the schema not being up to date leads to errors when trying to write the the new tables: [10.241.17.134] out: Exception in thread main com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured columnfamily schema_hash Any suggestions on what to do about this (other than increasing withMaxSchemaAgreementWaitSeconds)? This is only a three-node test cluster. I have not gotten this warning before, even on much bigger clusters. Best regards, Clint
No schema agreement from live replicas?
Hi all, I have an application that uses the Java driver to create a table and then immediately write to it. I see the following warning in my logs: [10.241.17.134] out: 15/02/03 09:32:24 WARN com.datastax.driver.core.Cluster: No schema agreement from live replicas after 10 s. The schema may not be up to date on some nodes. ...this seems to happen after creating a table, and the schema not being up to date leads to errors when trying to write the the new tables: [10.241.17.134] out: Exception in thread main com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured columnfamily schema_hash Any suggestions on what to do about this (other than increasing withMaxSchemaAgreementWaitSeconds)? This is only a three-node test cluster. I have not gotten this warning before, even on much bigger clusters. Best regards, Clint
Best practice for emulating a Cassandra timeout during unit tests?
Hi all, I'd like to write some tests for my code that uses the Cassandra Java driver to see how it behaves if there is a read timeout while accessing Cassandra. Is there a best-practice for getting this done? I was thinking about adjusting the settings in the cluster builder to adjust the timeout settings to be something impossibly low (like 1ms), but I'd rather do something to my test Cassandra instance (using the EmbeddedCassandraService) to temporarily slow it down. Any suggestions? Best regards, Clint
any way to get nodetool proxyhistograms data for an entire cluster?
If I run this tool on a given host, it shows me stats for only the cases where that host was the coordinator node, correct? Is there any way (other than me cooking up a little script) to automatically get the proxyhistogram stats for my entire cluster? -Clint
Re: any way to get nodetool proxyhistograms data for an entire cluster?
Thanks for the reply. We have DSE so I can use opscenter. I was just looking for something more precise than the graphs that I get from opscenter. On Wed, Nov 19, 2014 at 5:53 PM, Rahul Neelakantan ra...@rahul.be wrote: So what do use as a good alternate to it? Rahul Neelakantan On Nov 19, 2014, at 8:48 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Nov 19, 2014 at 3:22 PM, Clint Kelly clint.ke...@gmail.com wrote: Is there any way (other than me cooking up a little script) to automatically get the proxyhistogram stats for my entire cluster? OpsCenter might expose this as an aggregate, and can be used with free Apache Cassandra. Notice that I say might because I have no idea, because I don't use OpsCenter. :D =Rob
What time range does nodetool cfhistograms use?
Hi all, Over what time range does nodetool cfhistograms operate? I am using Cassandra 2.0.8.39. I am trying to debug some very high 95th and 99th percentile read latencies in an application that I'm working on. I tried running nodetool cfhistograms to get a flavor for the distribution of read latencies and also to see how many SSTables our reads are using, and I saw different results yesterday versus this morning, so I assume the time window is fairly tight (like an hour or so). I vaguely recall Aaron Morton talking about this during the training course I took at Cassandra Summit, but I cannot find my notes and I think the behavior of the tool with regard to time windows changed from version to version. Thanks! -Clint
Best practices for route tracing
Hi all, I am trying to debug some high-latency outliers (99th percentile) in an application I'm working on. I thought that I could turn on route tracing, print the route traces to logs, and then examine my logs after a load test to find the highest-latency paths and figure out what is going on. I did something similar to what is described here: http://www.datastax.com/documentation/developer/java-driver/2.1/java-driver/tracing_t.html Under low amounts of traffic, this looks great and I can see nice trace logs: Total duration: 599 us (id: ae159110-6dd8-11e4-a6a1-cd4ac3ad1896) Executing single-partition query on lg_users_C | 13:36:52.898 | /10.29.2.9 | 204 Acquiring sstable references | 13:36:52.898 | /10.29.2.9 | 207 Merging memtable tombstones | 13:36:52.898 | /10.29.2.9 | 213 Bloom filter allows skipping sstable 40 | 13:36:52.898 | /10.29.2.9 | 231 Bloom filter allows skipping sstable 38 | 13:36:52.898 | /10.29.2.9 | 237 Bloom filter allows skipping sstable 37 | 13:36:52.898 | /10.29.2.9 | 243 Key cache hit for sstable 39 | 13:36:52.898 | /10.29.2.9 | 251 Seeking to partition indexed section in data file | 13:36:52.898 | /10.29.2.9 | 251 Skipped 0/4 non-slice-intersecting sstables, included 0 due to tombstones | 13:36:52.898 | /10.29.2.9 | 455 Merging data from memtables and 1 sstables | 13:36:52.898 | /10.29.2.9 | 457 Read 0 live and 0 tombstoned cells | 13:36:52.898 | /10.29.2.9 | 472 Under heavier load, however, things seem to fall over, and I get a lot of exceptions like: 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying /10.29.2.9:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying /10.29.2.8:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying /10.29.2.9:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying /10.29.2.12:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying /10.29.2.9:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying /10.29.2.9:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying /10.29.2.10:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying /10.29.2.11:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying xwing07.ul.wibidata.net/10.29.2.7:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying /10.29.2.10:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying /10.29.2.9:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying /10.29.2.8:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying xwing07.ul.wibidata.net/10.29.2.7:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG com.datastax.driver.core.RequestHandler: Error querying /10.29.2.8:9042, trying next host (error is: com.datastax.driver.core.exceptions.DriverException: Timeout during read) 14/11/16 13:39:41 DEBUG
Re: What time range does nodetool cfhistograms use?
Thanks for the quick reply! I completely forgot about proxyhistograms! On Sun, Nov 16, 2014 at 1:56 PM, DuyHai Doan doanduy...@gmail.com wrote: Does it show statistics for just the current node on which I am running, or for the entire cluster? -- only current node. Are the read latencies shown the latencies within a single host, or are they the end-to-end latencies from the coordinator node? -- cfhistograms shows metrics at table/node level, proxyhistograms shows metrics at cluster/coordinator level On Sun, Nov 16, 2014 at 10:31 PM, Clint Kelly clint.ke...@gmail.com wrote: Thanks, Mark! A couple of other questions about the command: * Does it show statistics for just the current node on which I am running, or for the entire cluster? * Are the read latencies shown the latencies within a single host, or are they the end-to-end latencies from the coordinator node? -Clint On Sun, Nov 16, 2014 at 10:05 AM, Mark Reddy mark.l.re...@gmail.com wrote: Hi Clint, The values of SSTables, Write Latency and Read Latency will be reset on node start/restart and after running the cfhistograms command itself. The values of Row Size and Column Count are calculated at startup and then re-evaluated during compaction. Mark On 16 November 2014 17:12, Clint Kelly clint.ke...@gmail.com wrote: Hi all, Over what time range does nodetool cfhistograms operate? I am using Cassandra 2.0.8.39. I am trying to debug some very high 95th and 99th percentile read latencies in an application that I'm working on. I tried running nodetool cfhistograms to get a flavor for the distribution of read latencies and also to see how many SSTables our reads are using, and I saw different results yesterday versus this morning, so I assume the time window is fairly tight (like an hour or so). I vaguely recall Aaron Morton talking about this during the training course I took at Cassandra Summit, but I cannot find my notes and I think the behavior of the tool with regard to time windows changed from version to version. Thanks! -Clint
best practice for waiting for schema changes to propagate
Hi all, I often have problems with code that I write that uses the DataStax Java driver to create / modify a keyspace or table and then soon after reads the metadata for the keyspace to verify that whatever changes I made the keyspace or table are complete. As an example, I may create a table called `myTableName` and then very soon after do something like: assert(session .getCluster() .getMetaData() .getKeyspace(myKeyspaceName) .getTable(myTableName) != null) I assume this fails sometimes because the default round-robin load balancing policy for the Java driver will send my create-table request to one node and the metadata read to another, and because it takes some time for the table creation to propagate across all of the nodes in my cluster. What is the best way to deal with this problem? Is there a standard way to wait for schema changes to propagate? Best regards, Clint
nondeterministic NoHostAvailableException occurs while dropping a table
Hi all,
TL;DR - I think my unit tests are sometimes failing because of read
timeouts to an EmbeddedCassandraService when dropping a table triggers a
compaction on a highly-loaded build slave. Does this sound reasonable?
What options should I change in my Cluster.Builder (or elsewhere) to
prevent this from happening?
Longer version of the question:
We use the EmbeddedCassandraService for unit testing and we're seeing
non-deterministic failures on some machines. The sequence of events that
cause the failures look something like this:
- We have a single EmbeddedCassandraService that runs for all of our
unit tests
- In every test class, we create a new keyspace, then create a bunch of
tables within that keyspace and run our tests
- When a given test class is finished, we execute some tear-down code
that deletes the tables in the keyspace and then drops the keyspace itself
- All of the unit tests share a single Session object
Our tests always fail when we are executing the tear-down code. We always
get an error that looks like:
error message=All host(s) tried for query failed (tried: localhost/
127.0.0.1:57905 (com.datastax.driver.core.exceptions.DriverException:
Timeout during read))
type=com.datastax.driver.core.exceptions.NoHostAvailableExceptioncom.datastax.driver.core.exceptions.NoHostAvailableException:
All host(s) tried for query failed (tried: localhost/127.0.0.1:57905
(com.datastax.driver.core.exceptions.DriverException: Timeout during read))
at
com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:65)
at
com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:256)
at
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:172)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36)
I captured the output of the EmbeddedCassandraService to a log file, and
the last few lines look like:
14/09/04 04:14:34 ERROR org.apache.cassandra.db.Memtable: MemoryMeter
uninitialized (jamm not specified as java agent); assuming liveRatio of
10.0. Usually this means cassandra-env.sh disabled jamm because you are
using a buggy JRE; upgrade to the Sun JRE instead
14/09/04 04:14:34 INFO org.kiji.schema.impl.cassandra.CassandraAdmin:
Deleting table kiji_testGet_4.schema_id
14/09/04 04:14:34 INFO org.apache.cassandra.service.MigrationManager: Drop
ColumnFamily 'kiji_testGet_4/schema_id'
14/09/04 04:14:34 ERROR org.apache.cassandra.db.Memtable: MemoryMeter
uninitialized (jamm not specified as java agent); assuming liveRatio of
10.0. Usually this means cassandra-env.sh disabled jamm because you are
using a buggy JRE; upgrade to the Sun JRE instead
14/09/04 04:14:34 ERROR org.apache.cassandra.db.Memtable: MemoryMeter
uninitialized (jamm not specified as java agent); assuming liveRatio of
10.0. Usually this means cassandra-env.sh disabled jamm because you are
using a buggy JRE; upgrade to the Sun JRE instead
14/09/04 04:14:34 ERROR org.apache.cassandra.db.Memtable: MemoryMeter
uninitialized (jamm not specified as java agent); assuming liveRatio of
10.0. Usually this means cassandra-env.sh disabled jamm because you are
using a buggy JRE; upgrade to the Sun JRE instead
14/09/04 04:14:34 INFO org.apache.cassandra.db.ColumnFamilyStore: Enqueuing
flush of Memtable-schema_keyspaces@822600288(138/1380 serialized/live
bytes, 3 ops)
14/09/04 04:14:34 INFO org.apache.cassandra.db.Memtable: Writing
Memtable-schema_keyspaces@822600288(138/1380 serialized/live bytes, 3 ops)
14/09/04 04:14:35 INFO org.apache.cassandra.db.compaction.CompactionTask:
Compacted 4 sstables to
[target/cassandra/data/system/local/system-local-jb-77,]. 6,147 bytes to
5,713 (~92% of original) in 497ms = 0.010962MB/s. 4 total partitions
merged to 1. Partition merge counts were {4:1, }
14/09/04 04:14:35 INFO org.apache.cassandra.db.Memtable: Completed flushing
target/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-151-Data.db
(167 bytes) for commitlog position ReplayPosition(segmentId=1409829145687,
position=619631)
14/09/04 04:14:35 INFO org.apache.cassandra.db.ColumnFamilyStore: Enqueuing
flush of Memtable-schema_columnfamilies@1313116194(0/0 serialized/live
bytes, 2 ops)
14/09/04 04:14:35 INFO org.apache.cassandra.db.Memtable: Writing
Memtable-schema_columnfamilies@1313116194(0/0 serialized/live bytes, 2 ops)
14/09/04 04:14:36 INFO org.apache.cassandra.db.Memtable: Completed flushing
target/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-jb-142-Data.db
(68 bytes) for commitlog position ReplayPosition(segmentId=1409829145687,
position=620056)
14/09/04 04:14:36 INFO org.apache.cassandra.db.ColumnFamilyStore: Enqueuing
flush of Memtable-schema_columns@1575679153(0/0 serialized/live bytes, 4
ops)
14/09/04 04:14:36 INFO
Re: cassandra-stress with clustering columns?
Hi Mikail, This plugin looks great! I have actually been using JMeter + a custom REST endpoint driving Cassandra. It would be great to compare the results I got from that against the pure JMeter + Cassandra (to evaluate the REST endpoint's performance). Thanks! I'll check this out. Best regards, Clint On Tue, Aug 19, 2014 at 1:38 AM, Mikhail Stepura mikhail.step...@outlook.com wrote: Are you interested in cassandra-stress in particular? Or in any tool which will allow you to stress test your schema? I believe Apache Jmeter + CQL plugin may be useful in the latter case. https://github.com/Mishail/CqlJmeter -M On 8/17/14 12:26, Clint Kelly wrote: Hi all, Is there a way to use the cassandra-stress tool with clustering columns? I am trying to figure out whether an application that I'm running on is slow because of my application logic, C* data model, or underlying C* setup (e.g., I need more nodes or to tune some parameters). My application uses tables with several clustering columns and a couple of additional indices and it is running quite slowly under a heavy write load. I think that the problem is my data model (and therefore table layout), but I'd like to confirm by replicating the problem with cassandra-stress. I don't see any option for using clustering columns or secondary indices, but I wanted to check before diving into the code and trying to add this functionality. Thanks! Best regards, Clint
Re: cassandra-stress with clustering columns?
Thanks for the update, Benedict. We are still using 2.0.9 unfortunately. :/ I will keep that in mind for when we upgrade. On Tue, Aug 19, 2014 at 10:51 AM, Benedict Elliott Smith belliottsm...@datastax.com wrote: The stress tool in 2.1 also now supports clustering columns: http://www.datastax.com/dev/blog/improved-cassandra-2-1-stress-tool-benchmark-any-schema There are however some features up for revision before release in order to help generate realistic workloads. See https://issues.apache.org/jira/browse/CASSANDRA-7519 for details. On Tue, Aug 19, 2014 at 10:46 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi Mikail, This plugin looks great! I have actually been using JMeter + a custom REST endpoint driving Cassandra. It would be great to compare the results I got from that against the pure JMeter + Cassandra (to evaluate the REST endpoint's performance). Thanks! I'll check this out. Best regards, Clint On Tue, Aug 19, 2014 at 1:38 AM, Mikhail Stepura mikhail.step...@outlook.com wrote: Are you interested in cassandra-stress in particular? Or in any tool which will allow you to stress test your schema? I believe Apache Jmeter + CQL plugin may be useful in the latter case. https://github.com/Mishail/CqlJmeter -M On 8/17/14 12:26, Clint Kelly wrote: Hi all, Is there a way to use the cassandra-stress tool with clustering columns? I am trying to figure out whether an application that I'm running on is slow because of my application logic, C* data model, or underlying C* setup (e.g., I need more nodes or to tune some parameters). My application uses tables with several clustering columns and a couple of additional indices and it is running quite slowly under a heavy write load. I think that the problem is my data model (and therefore table layout), but I'd like to confirm by replicating the problem with cassandra-stress. I don't see any option for using clustering columns or secondary indices, but I wanted to check before diving into the code and trying to add this functionality. Thanks! Best regards, Clint
cassandra-stress with clustering columns?
Hi all, Is there a way to use the cassandra-stress tool with clustering columns? I am trying to figure out whether an application that I'm running on is slow because of my application logic, C* data model, or underlying C* setup (e.g., I need more nodes or to tune some parameters). My application uses tables with several clustering columns and a couple of additional indices and it is running quite slowly under a heavy write load. I think that the problem is my data model (and therefore table layout), but I'd like to confirm by replicating the problem with cassandra-stress. I don't see any option for using clustering columns or secondary indices, but I wanted to check before diving into the code and trying to add this functionality. Thanks! Best regards, Clint
Re: cassandra-stress with clustering columns?
FWIW I tried just removing indices from my table and rerunning my write load test and I got about 2x the performance. I'm still interested, however, in seeing whether I can replicate my table layout in cassandra-stress, just so that I can normalize my results. Should the presence of clustering columns make a big difference in write performance? On Sun, Aug 17, 2014 at 12:26 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi all, Is there a way to use the cassandra-stress tool with clustering columns? I am trying to figure out whether an application that I'm running on is slow because of my application logic, C* data model, or underlying C* setup (e.g., I need more nodes or to tune some parameters). My application uses tables with several clustering columns and a couple of additional indices and it is running quite slowly under a heavy write load. I think that the problem is my data model (and therefore table layout), but I'd like to confirm by replicating the problem with cassandra-stress. I don't see any option for using clustering columns or secondary indices, but I wanted to check before diving into the code and trying to add this functionality. Thanks! Best regards, Clint
Re: question about OpsCenter agent
Thanks, Mark, this was exactly what I needed! On Thu, Aug 14, 2014 at 11:54 PM, Mark Reddy mark.re...@boxever.com wrote: Hi Clint, You need to configure the datastax-agents so they know what machine OpsCenter is running on. To do this you will need to edit the address.yaml of the datastax-agent, located in /var/lib/datastax-agent/conf/. In this file you need to add the following line: stomp_interface: opscenter-ip-address This will allow your agents to connect to the OpsCenter server (opscenterd) over the stomp protocol (stomp.github.io/). For a more detailed look at the configuration options available to the datastax-agent see this page: datastax.com/documentation/opscenter/5.0/opsc/configure/agentAddressConfiguration.html Mark On Fri, Aug 15, 2014 at 3:32 AM, Clint Kelly clint.ke...@gmail.com wrote: Hi all, I just installed DataStax Enterprise 4.5. I installed OpsCenter Server on one of my four machines. The port that OpsCenter usually uses () was used by something else, so I modified /usr/share/opscenter/conf/opscenterd.conf to set the port to 8889. When I log into OpsCenter, it says 0 of 4 agents connected. I have started datastax-agent on all four of the nodes in my Cassandra cluster. Is there another OpsCenter agent that I have to start? I'm not sure how to fix this. I clicked on the Fix link and it prompted me You have nodes that need the OpsCenter agent installed. How do I install the OpsCenter agent without using this GUI? I don't recall getting prompted for that when I installed DSE. Best regards, Clint
question about OpsCenter agent
Hi all, I just installed DataStax Enterprise 4.5. I installed OpsCenter Server on one of my four machines. The port that OpsCenter usually uses () was used by something else, so I modified /usr/share/opscenter/conf/opscenterd.conf to set the port to 8889. When I log into OpsCenter, it says 0 of 4 agents connected. I have started datastax-agent on all four of the nodes in my Cassandra cluster. Is there another OpsCenter agent that I have to start? I'm not sure how to fix this. I clicked on the Fix link and it prompted me You have nodes that need the OpsCenter agent installed. How do I install the OpsCenter agent without using this GUI? I don't recall getting prompted for that when I installed DSE. Best regards, Clint
Re: Cassandra process exiting mysteriously
Hi Or, For now I removed the test that was failing like this from our suite and made a note to revisit it in a couple of weeks. Unfortunately I still don't know what the issue is. I'll post here if I figure out it (please do the same!). My working hypothesis now is that we had some kind of OOM problem. Best regards, Clint On Tue, Aug 12, 2014 at 12:23 AM, Or Sher or.sh...@gmail.com wrote: Clint, did you find anything? I just noticed it happens to us too on only one node in our CI cluster. I don't think there is a special usage before it happens... The last line in the log before the shutdown lines in at least an hour before.. We're using C* 2.0.9. On Thu, Aug 7, 2014 at 12:49 AM, Clint Kelly clint.ke...@gmail.com wrote: Hi Rob, Thanks for the clarification; this is really useful. I'll run some experiments to see if the problem is a JVM OOM on our build machine. Best regards, Clint On Wed, Aug 6, 2014 at 1:14 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Aug 6, 2014 at 1:12 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Aug 6, 2014 at 1:11 AM, Duncan Sands duncan.sa...@gmail.com wrote: this doesn't look like an OOM to me. If the kernel OOM kills Cassandra then Cassandra instantly vaporizes, and there will be nothing in the Cassandra logs (you will find information about the OOM in the system logs though, eg in dmesg). In the log snippet above you see an orderly shutdown, this is completely different to the instant OOM kill. Not really. https://issues.apache.org/jira/browse/CASSANDRA-7507 To be clear, there's two different OOMs here, I am talking about the JVM OOM, not system level. As CASSANDRA-7507 indicates, JVM OOM does not necessarily result in the cassandra process dying, and can in fact trigger clean shutdown. System level OOM will in fact send the equivalent of KILL, which will not trigger the clean shutdown hook in Cassandra. =Rob -- Or Sher
Re: Cassandra process exiting mysteriously
Hi Duncan, Thanks for your help. I am at a loss as to what is causing this process to stop then. I would not expect the Cassandra process to finish until my code calls Process#destroy, but it seems to non-deterministically stop much earlier sometimes. FWIW I have seen failures on another machine this morning which also look orderly. These nodes never even get to the point where they announce they are listening for CQL clients. If anyone has any ideas on what to look for, I would really appreciate it. I will try turning logging up to DEBUG and see if that produces any useful errors. Best regards, Clint On Wed, Aug 6, 2014 at 1:11 AM, Duncan Sands duncan.sa...@gmail.com wrote: Hi Clint, INFO [StorageServiceShutdownHook] 2014-08-05 19:14:51,903 ThriftServer.java (line 141) Stop listening to thrift clients INFO [StorageServiceShutdownHook] 2014-08-05 19:14:51,920 Server.java (line 182) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-08-05 19:14:51,930 Gossiper.java (line 1279) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-08-05 19:14:53,930 MessagingService.java (line 683) Waiting for messaging service to quiesce INFO [ACCEPT-/127.0.0.10] 2014-08-05 19:14:53,931 MessagingService.java (line 923) MessagingService has terminated the accept() thread Does anyone have any ideas about how to debug this? Looking around on google I found some threads suggesting that this could occur from an OOM error (http://stackoverflow.com/questions/23755040/cassandra-exits-with-no-errors). this doesn't look like an OOM to me. If the kernel OOM kills Cassandra then Cassandra instantly vaporizes, and there will be nothing in the Cassandra logs (you will find information about the OOM in the system logs though, eg in dmesg). In the log snippet above you see an orderly shutdown, this is completely different to the instant OOM kill. Ciao, Duncan.
Re: Cassandra process exiting mysteriously
Hi Rob, Thanks for the clarification; this is really useful. I'll run some experiments to see if the problem is a JVM OOM on our build machine. Best regards, Clint On Wed, Aug 6, 2014 at 1:14 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Aug 6, 2014 at 1:12 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Aug 6, 2014 at 1:11 AM, Duncan Sands duncan.sa...@gmail.com wrote: this doesn't look like an OOM to me. If the kernel OOM kills Cassandra then Cassandra instantly vaporizes, and there will be nothing in the Cassandra logs (you will find information about the OOM in the system logs though, eg in dmesg). In the log snippet above you see an orderly shutdown, this is completely different to the instant OOM kill. Not really. https://issues.apache.org/jira/browse/CASSANDRA-7507 To be clear, there's two different OOMs here, I am talking about the JVM OOM, not system level. As CASSANDRA-7507 indicates, JVM OOM does not necessarily result in the cassandra process dying, and can in fact trigger clean shutdown. System level OOM will in fact send the equivalent of KILL, which will not trigger the clean shutdown hook in Cassandra. =Rob
Read timeouts with ALLOW FILTERING turned on
Hi all, Allow me to rephrase a question I asked last week. I am performing some queries with ALLOW FILTERING and getting consistent read timeouts like the following: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) These errors occur only during multi-row scans, and only during integration tests on our build server. I tried to see if I could replicate this error by reducing read_request_timeout_in_ms when I run Cassandra on my local machine (where I have not seen this error), but that is not working. Are there any other parameters that I need to adjust? I'd feel better if I could at least replicate this failure by reducing the read_request_timeout_in_ms (since doing so would mean I actually understand what is going wrong...). Best regards, Clint
Re: Read timeouts with ALLOW FILTERING turned on
Hi Rob, Thanks for your feedback. I understand that use of ALLOW FILTERING is not a best practice. In this case, however, I am building a tool on top of Cassandra that allows users to sometimes do things that are less than optimal. When they try to do expensive queries like this, I'd rather provide a higher limit before timing out, but I can't seem to change the behavior of Cassandra by tweaking any of the parameters in the cassandra.yaml file or in the DataStax Java driver's Cluster object. FWIW these queries are also in batch jobs where we can tolerate the extra latency. Thanks for your help! Best regards, Clint On Tue, Aug 5, 2014 at 10:54 AM, Robert Coli rc...@eventbrite.com wrote: On Tue, Aug 5, 2014 at 10:01 AM, Clint Kelly clint.ke...@gmail.com wrote: Allow me to rephrase a question I asked last week. I am performing some queries with ALLOW FILTERING and getting consistent read timeouts like the following: ALLOW FILTERING should be renamed PROBABLY TIMEOUT in order to properly describe its typical performance. As a general statement, if you have to ALLOW FILTERING, you are probably Doing It Wrong in terms of schema design. A correctly operated cluster is unlikely to need to increase the default timeouts. If you find yourself needing to do so, you are, again, probably Doing It Wrong. =Rob
Re: Read timeouts with ALLOW FILTERING turned on
Ah FWIW I was able to reproduce the problem by reducing range_request_timeout_in_ms. This is great since I want to increase the timeout for batch jobs where we scan a large set of rows, but leave the timeout for single-row queries alone. Best regards, Clint On Tue, Aug 5, 2014 at 11:42 AM, Clint Kelly clint.ke...@gmail.com wrote: Hi Rob, Thanks for your feedback. I understand that use of ALLOW FILTERING is not a best practice. In this case, however, I am building a tool on top of Cassandra that allows users to sometimes do things that are less than optimal. When they try to do expensive queries like this, I'd rather provide a higher limit before timing out, but I can't seem to change the behavior of Cassandra by tweaking any of the parameters in the cassandra.yaml file or in the DataStax Java driver's Cluster object. FWIW these queries are also in batch jobs where we can tolerate the extra latency. Thanks for your help! Best regards, Clint On Tue, Aug 5, 2014 at 10:54 AM, Robert Coli rc...@eventbrite.com wrote: On Tue, Aug 5, 2014 at 10:01 AM, Clint Kelly clint.ke...@gmail.com wrote: Allow me to rephrase a question I asked last week. I am performing some queries with ALLOW FILTERING and getting consistent read timeouts like the following: ALLOW FILTERING should be renamed PROBABLY TIMEOUT in order to properly describe its typical performance. As a general statement, if you have to ALLOW FILTERING, you are probably Doing It Wrong in terms of schema design. A correctly operated cluster is unlikely to need to increase the default timeouts. If you find yourself needing to do so, you are, again, probably Doing It Wrong. =Rob
Cassandra process exiting mysteriously
Hi everyone, For some integration tests, we start up a CassandraDaemon in a separate process (using the Java 7 ProcessBuilder API). All of my integration tests run beautifully on my laptop, but one of them fails on our Jenkins cluster. The failing integration test does around 10k writes to different rows and then 10k reads. After running some number of reads, the job dies with this error: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.10:58209 (com.datastax.driver.core.exceptions.DriverException: Timeout during read)) This error appears to have occurred because the Cassandra process has stopped. The logs for the Cassandra process show some warnings during batch writes (the batches are too big), no activity for a few minutes (I assume this is because all of the read operations were proceeding smoothly), and then look like the following: INFO [StorageServiceShutdownHook] 2014-08-05 19:14:51,903 ThriftServer.java (line 141) Stop listening to thrift clients INFO [StorageServiceShutdownHook] 2014-08-05 19:14:51,920 Server.java (line 182) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-08-05 19:14:51,930 Gossiper.java (line 1279) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-08-05 19:14:53,930 MessagingService.java (line 683) Waiting for messaging service to quiesce INFO [ACCEPT-/127.0.0.10] 2014-08-05 19:14:53,931 MessagingService.java (line 923) MessagingService has terminated the accept() thread Does anyone have any ideas about how to debug this? Looking around on google I found some threads suggesting that this could occur from an OOM error (http://stackoverflow.com/questions/23755040/cassandra-exits-with-no-errors). Wouldn't such an error be logged, however? The test that fails is a test of our MapReduce Hadoop InputFormat and as such it does some pretty big queries across multiple rows (over a range of partitioning key tokens). The default fetch size I believe is 5000 rows, and the values in the rows I am fetching are just simple strings, so I would not think the amount of data in a single read would be too big. FWIW I don't see any log messages about garbage collection for at least 3min before the process shuts down (and no GC messages after the test stops doing writes and starts doing reads). I'd greatly appreciate any help before my team kills me for breaking our Jenkins build so consistently! :) Best regards, Clint
Re: Cassandra process exiting mysteriously
HI Kevin, Thanks for your reply. That is what I assumed, but some of the posts I read on Stack Overflow (e.g., the one that I referenced in my mail) suggested otherwise. I was just curious if others had experienced OOM problems that weren't logged or if there were other common culprits. Best regards, Clint On Tue, Aug 5, 2014 at 9:29 PM, Kevin Burton bur...@spinn3r.com wrote: If there is an oom it will be in the logs. On Aug 5, 2014 8:17 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi everyone, For some integration tests, we start up a CassandraDaemon in a separate process (using the Java 7 ProcessBuilder API). All of my integration tests run beautifully on my laptop, but one of them fails on our Jenkins cluster. The failing integration test does around 10k writes to different rows and then 10k reads. After running some number of reads, the job dies with this error: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.10:58209 (com.datastax.driver.core.exceptions.DriverException: Timeout during read)) This error appears to have occurred because the Cassandra process has stopped. The logs for the Cassandra process show some warnings during batch writes (the batches are too big), no activity for a few minutes (I assume this is because all of the read operations were proceeding smoothly), and then look like the following: INFO [StorageServiceShutdownHook] 2014-08-05 19:14:51,903 ThriftServer.java (line 141) Stop listening to thrift clients INFO [StorageServiceShutdownHook] 2014-08-05 19:14:51,920 Server.java (line 182) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-08-05 19:14:51,930 Gossiper.java (line 1279) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-08-05 19:14:53,930 MessagingService.java (line 683) Waiting for messaging service to quiesce INFO [ACCEPT-/127.0.0.10] 2014-08-05 19:14:53,931 MessagingService.java (line 923) MessagingService has terminated the accept() thread Does anyone have any ideas about how to debug this? Looking around on google I found some threads suggesting that this could occur from an OOM error (http://stackoverflow.com/questions/23755040/cassandra-exits-with-no-errors). Wouldn't such an error be logged, however? The test that fails is a test of our MapReduce Hadoop InputFormat and as such it does some pretty big queries across multiple rows (over a range of partitioning key tokens). The default fetch size I believe is 5000 rows, and the values in the rows I am fetching are just simple strings, so I would not think the amount of data in a single read would be too big. FWIW I don't see any log messages about garbage collection for at least 3min before the process shuts down (and no GC messages after the test stops doing writes and starts doing reads). I'd greatly appreciate any help before my team kills me for breaking our Jenkins build so consistently! :) Best regards, Clint
Re: Occasional read timeouts seen during row scans
Hi all, 1. I saw this issue in an integration test with a single CassandraDaemon running, so I don't think it was a time synchronization issue. 2. I did not look in the log for garbage collection issues, but I was able to reproduce this 100% deterministically, so I think it was an issue having to do with the organization and size of my data. I have been unable to fix this by retrying failed reads (because this behavior, when it occurs, is deterministic). I was looking for some kind of guidance on how to tune Cassandra to increase or decrease this timeout threshold such that I can tolerate a higher timeout in the cluster and so that I can reproduce this in some unit or integration tests. Also if anyone has any ideas on how my particular table layout might lead to these kinds of problems, that would be great. Thanks! Best regards, Clint On Sat, Aug 2, 2014 at 4:40 AM, Jack Krupansky j...@basetechnology.com wrote: Are you seeing garbage collections in the log at around the same time as these occasional timeouts? Can you identify which requests are timing out? And then can you try some of them again and see if they succeed at least sometimes and how long they take then? Do you have a test case that you believe does the worst case for filtering? How long does it take? Can you monitor if the timed-out node is compute bound or I/O bound at the times of failure? Do you see spikes for compute or I/O? Can your app simply retry the timed-out request? Does even a retry typically fail, or does retry get you to 100% success? I would note that even the best distributed systems do not guarantee zero failures for environmental issues, so apps need to tolerate occasional failures. -- Jack Krupansky -Original Message- From: Duncan Sands Sent: Saturday, August 2, 2014 7:04 AM To: user@cassandra.apache.org Subject: Re: Occasional read timeouts seen during row scans Hi Clint, is time correctly synchronized between your nodes? Ciao, Duncan. On 02/08/14 02:12, Clint Kelly wrote: BTW a few other details, sorry for omitting these: * We are using version 2.0.4 of the Java driver * We are running against Cassandra 2.0.9 * I tried messing around with the page size (even reducing it down to a single record) and that didn't seem to help (in the cases where I was observing the timeout) Best regards, Clint On Fri, Aug 1, 2014 at 5:02 PM, Clint Kelly clint.ke...@gmail.com mailto:clint.ke...@gmail.com wrote: Hi everyone, I am seeing occasional read timeouts during multi-row queries, but I'm having difficulty reproducing them or understanding what the problem is. First, some background: Our team wrote a custom MapReduce InputFormat that looks pretty similar to the DataStax InputFormat except that it allows queries that touch multiple CQL tables with the same PRIMARY KEY format (it then assembles together results from multiple tables for the same primary key before sending them back to the user in the RecordReader). During a large batch job in a cluster and during some integration tests, we see errors like the following: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) Our queries look like this: SELECT token(eid_component), eid_component, lg, family, qualifier, version, value FROM kiji_it0.t_foo WHERE lg=? AND family=? AND qualifier=? AND token(eid_component) = ? AND token(eid_component) = ?ALLOW FILTERING; Our tables look like the following: CREATE TABLE kiji_it0.t_foo ( eid_component varchar, lg varchar, family blob, qualifier blob, version bigint, value blob, PRIMARY KEY ((eid_component), lg, family, qualifier, version)) WITH CLUSTERING ORDER BY (lg ASC, family ASC, qualifier ASC, version DESC); with an additional index on the lg column (the lg column is *extremely* low cardinality). (FWIW I realize that having ALLOW FILTERING is potentially a Very Bad Idea, but we are building a framework on top of Cassandra and MapReduce that allows our users to occasionally make queries like this. We don't really mind taking a performance hit since these are batch jobs. We are considering eventually supporting some automatic denormalization, but have not done so yet.) If I change the query above to remove the WHERE clauses, the errors go away. I think I understand the problem here---there are some rows that have huge amounts of data that we have to scan over, and occasionally those scans take so long that there is a timeout. I have a couple of questions: 1. What parameters in my code or in the Cassandra cluster do I need to adjust to get rid of these timeouts? Our table layout is designed
Occasional read timeouts seen during row scans
Hi everyone, I am seeing occasional read timeouts during multi-row queries, but I'm having difficulty reproducing them or understanding what the problem is. First, some background: Our team wrote a custom MapReduce InputFormat that looks pretty similar to the DataStax InputFormat except that it allows queries that touch multiple CQL tables with the same PRIMARY KEY format (it then assembles together results from multiple tables for the same primary key before sending them back to the user in the RecordReader). During a large batch job in a cluster and during some integration tests, we see errors like the following: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) Our queries look like this: SELECT token(eid_component), eid_component, lg, family, qualifier, version, value FROM kiji_it0.t_foo WHERE lg=? AND family=? AND qualifier=? AND token(eid_component) = ? AND token(eid_component) = ?ALLOW FILTERING; Our tables look like the following: CREATE TABLE kiji_it0.t_foo ( eid_component varchar, lg varchar, family blob, qualifier blob, version bigint, value blob, PRIMARY KEY ((eid_component), lg, family, qualifier, version)) WITH CLUSTERING ORDER BY (lg ASC, family ASC, qualifier ASC, version DESC); with an additional index on the lg column (the lg column is *extremely* low cardinality). (FWIW I realize that having ALLOW FILTERING is potentially a Very Bad Idea, but we are building a framework on top of Cassandra and MapReduce that allows our users to occasionally make queries like this. We don't really mind taking a performance hit since these are batch jobs. We are considering eventually supporting some automatic denormalization, but have not done so yet.) If I change the query above to remove the WHERE clauses, the errors go away. I think I understand the problem here---there are some rows that have huge amounts of data that we have to scan over, and occasionally those scans take so long that there is a timeout. I have a couple of questions: 1. What parameters in my code or in the Cassandra cluster do I need to adjust to get rid of these timeouts? Our table layout is designed such that its real-time performance should be pretty good, so I don't mind if the batch queries are a little bit slow. Do I need to change the read_request_timeout_in_ms parameter? Or something else? 2. I have tried to create a test to reproduce this problem, but I have been unable to do so. Any suggestions on how to do this? I tried creating a table similar to that described above and filling in a huge amount of data for some rows to try to increase the amount of space that we'd need to skip over. I also tried reducing read_request_timeout_in_ms from 5000 ms to 50 ms and still no dice. Let me know if anyone has any thoughts or suggestions. At a minimum I'd like to be able to reproduce these read timeout errors in some integration tests. Thanks! Best regards, Clint
Re: Occasional read timeouts seen during row scans
BTW a few other details, sorry for omitting these: - We are using version 2.0.4 of the Java driver - We are running against Cassandra 2.0.9 - I tried messing around with the page size (even reducing it down to a single record) and that didn't seem to help (in the cases where I was observing the timeout) Best regards, Clint On Fri, Aug 1, 2014 at 5:02 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi everyone, I am seeing occasional read timeouts during multi-row queries, but I'm having difficulty reproducing them or understanding what the problem is. First, some background: Our team wrote a custom MapReduce InputFormat that looks pretty similar to the DataStax InputFormat except that it allows queries that touch multiple CQL tables with the same PRIMARY KEY format (it then assembles together results from multiple tables for the same primary key before sending them back to the user in the RecordReader). During a large batch job in a cluster and during some integration tests, we see errors like the following: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) Our queries look like this: SELECT token(eid_component), eid_component, lg, family, qualifier, version, value FROM kiji_it0.t_foo WHERE lg=? AND family=? AND qualifier=? AND token(eid_component) = ? AND token(eid_component) = ?ALLOW FILTERING; Our tables look like the following: CREATE TABLE kiji_it0.t_foo ( eid_component varchar, lg varchar, family blob, qualifier blob, version bigint, value blob, PRIMARY KEY ((eid_component), lg, family, qualifier, version)) WITH CLUSTERING ORDER BY (lg ASC, family ASC, qualifier ASC, version DESC); with an additional index on the lg column (the lg column is *extremely* low cardinality). (FWIW I realize that having ALLOW FILTERING is potentially a Very Bad Idea, but we are building a framework on top of Cassandra and MapReduce that allows our users to occasionally make queries like this. We don't really mind taking a performance hit since these are batch jobs. We are considering eventually supporting some automatic denormalization, but have not done so yet.) If I change the query above to remove the WHERE clauses, the errors go away. I think I understand the problem here---there are some rows that have huge amounts of data that we have to scan over, and occasionally those scans take so long that there is a timeout. I have a couple of questions: 1. What parameters in my code or in the Cassandra cluster do I need to adjust to get rid of these timeouts? Our table layout is designed such that its real-time performance should be pretty good, so I don't mind if the batch queries are a little bit slow. Do I need to change the read_request_timeout_in_ms parameter? Or something else? 2. I have tried to create a test to reproduce this problem, but I have been unable to do so. Any suggestions on how to do this? I tried creating a table similar to that described above and filling in a huge amount of data for some rows to try to increase the amount of space that we'd need to skip over. I also tried reducing read_request_timeout_in_ms from 5000 ms to 50 ms and still no dice. Let me know if anyone has any thoughts or suggestions. At a minimum I'd like to be able to reproduce these read timeout errors in some integration tests. Thanks! Best regards, Clint
Re: Index creation sometimes fails
Hi Tyler, FWIW I was not able to reproduce this problem with a smaller example. I'll go ahead and file the JIRA anyway. Thanks for your help! Best regards, Clint On Thu, Jul 17, 2014 at 3:05 PM, Tyler Hobbs ty...@datastax.com wrote: On Thu, Jul 17, 2014 at 4:59 PM, Clint Kelly clint.ke...@gmail.com wrote: I will post a JIRA, along with directions on how to get this to happen. The tricky thing, though, is that this doesn't always happen, and I cannot reproduce it on my laptop or in a VM. Even if you can't reproduce, just include as many details as you can. C* and driver versions, schemas, logs, etc. BTW you mean the datastax JIRA, correct? Oops! Yes, I meant https://datastax-oss.atlassian.net/browse/JAVA, not the batch statement docs :) -- Tyler Hobbs DataStax http://datastax.com/
Re: Index creation sometimes fails
Hi Tyler, Thanks for replying. This is good to know that I am not going crazy! :) I will post a JIRA, along with directions on how to get this to happen. The tricky thing, though, is that this doesn't always happen, and I cannot reproduce it on my laptop or in a VM. BTW you mean the datastax JIRA, correct? Best regards, Clint On Wed, Jul 16, 2014 at 4:32 PM, Tyler Hobbs ty...@datastax.com wrote: On Tue, Jul 15, 2014 at 1:40 PM, Clint Kelly clint.ke...@gmail.com wrote: Is there some way to get the driver to block until the schema code has propagated everywhere? My currently solution feels rather janky! The driver *should* be blocking until the schema has propagated already. If it's not, that's a bug. I would check the changelog and JIRA for related tickets, and if you don't find anything, open a new ticket with details and steps to repro: http://cassandra.apache.org/doc/cql3/CQL.html#batchStmt -- Tyler Hobbs DataStax
How to maintain the N-most-recent versions of a value?
Hi everyone, I am trying to design a schema that will keep the N-most-recent versions of a value. Currently my table looks like the following: CREATE TABLE foo ( rowkey text, family text, qualifier text, version long, value blob, PRIMARY KEY (rowkey, family, qualifier, version)) WITH CLUSTER ORDER BY (rowkey ASC, family ASC, qualifier ASC, version DESC)); Is there any standard design pattern for updating such a layout such that I keep the N-most-recent (version, value) pairs for every unique (rowkey, family, qualifier)? I can't think of any way to do this without doing a read-modify-write. The best thing I can think of is to use TTL to approximate the desired behavior (which will work if I know how often we are writing new data to the table). I could also use LIMIT N in my queries to limit myself to only N items, but that does not address any of the storage-size issues. In case anyone is curious, this question is related to some work that I am doing translating a system built on HBase (which provides this keep the N-most-recent-version-of-a-cell behavior) to Cassandra while providing the user with as-similar-as-possible an interface. Best regards, Clint
Re: Index creation sometimes fails
FWIW I was able to work around this problem by having my code run the following loop: while (the index doesn't exit and we haven't hit our limit of number of tries): try to create the index read from system.schema_columns to see if the index exists if so, then break if not, pause 5 seconds This loop took three iterations to create the index. Is this expected? This seems really weird! Best regards, Clint On Mon, Jul 14, 2014 at 5:54 PM, Clint Kelly clint.ke...@gmail.com wrote: BTW I have seen this using versions 2.0.1 and 2.0.3 of the java driver on a three-node cluster with DSE 4.5. On Mon, Jul 14, 2014 at 5:51 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi everyone, I have some code that I've been fiddling with today that uses the DataStax Java driver to create a table and then create a secondary index on a column in that table. I've testing this code fairly thoroughly on a single-node Cassandra instance on my laptop and in unit test (using the CassandraDaemon). When running on a three-node cluster, however, I see strange behavior. Although my table always gets created, the secondary index often does not! If I delete the table and then create it again (through the same code that I've written), I've never seen the index fail to appear the second time. Does anyone have any idea what to look for here? I have no experience working on a Cassandra cluster and I wonder if maybe I am doing something dumb (I basically just installed DSE and started up the three nodes and that was it). I don't see anything that looks unusual in OpsCenter for DSE. The only thing I've noticed is that the presence of output like the following from my program after executing the command to create the index is perfectly correlated with successful creation of the index: 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Received event EVENT CREATED kiji_retail2.t_model_repo, scheduling delivery 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: [Control connection] Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [b309518a-35d2-3790-bb66-ea39bb0d188c] If anyone can give me a hand, I would really appreciate it. I am out of ideas! Best regards, Clint
Re: Index creation sometimes fails
Hi DuyHai, Thanks for the response. Is the recommended best practice therefore to do what I describe above? Is there some way to get the driver to block until the schema code has propagated everywhere? My currently solution feels rather janky! Thanks! Best regards, Clint On Tue, Jul 15, 2014 at 11:32 AM, DuyHai Doan doanduy...@gmail.com wrote: As far as I know, schema propagation always takes some times in the cluster. On this mailing list some people in the past faced similar behavior. On Tue, Jul 15, 2014 at 8:20 PM, Clint Kelly clint.ke...@gmail.com wrote: FWIW I was able to work around this problem by having my code run the following loop: while (the index doesn't exit and we haven't hit our limit of number of tries): try to create the index read from system.schema_columns to see if the index exists if so, then break if not, pause 5 seconds This loop took three iterations to create the index. Is this expected? This seems really weird! Best regards, Clint On Mon, Jul 14, 2014 at 5:54 PM, Clint Kelly clint.ke...@gmail.com wrote: BTW I have seen this using versions 2.0.1 and 2.0.3 of the java driver on a three-node cluster with DSE 4.5. On Mon, Jul 14, 2014 at 5:51 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi everyone, I have some code that I've been fiddling with today that uses the DataStax Java driver to create a table and then create a secondary index on a column in that table. I've testing this code fairly thoroughly on a single-node Cassandra instance on my laptop and in unit test (using the CassandraDaemon). When running on a three-node cluster, however, I see strange behavior. Although my table always gets created, the secondary index often does not! If I delete the table and then create it again (through the same code that I've written), I've never seen the index fail to appear the second time. Does anyone have any idea what to look for here? I have no experience working on a Cassandra cluster and I wonder if maybe I am doing something dumb (I basically just installed DSE and started up the three nodes and that was it). I don't see anything that looks unusual in OpsCenter for DSE. The only thing I've noticed is that the presence of output like the following from my program after executing the command to create the index is perfectly correlated with successful creation of the index: 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Received event EVENT CREATED kiji_retail2.t_model_repo, scheduling delivery 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: [Control connection] Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [b309518a-35d2-3790-bb66-ea39bb0d188c] If anyone can give me a hand, I would really appreciate it. I am out of ideas! Best regards, Clint
Index creation sometimes fails
Hi everyone, I have some code that I've been fiddling with today that uses the DataStax Java driver to create a table and then create a secondary index on a column in that table. I've testing this code fairly thoroughly on a single-node Cassandra instance on my laptop and in unit test (using the CassandraDaemon). When running on a three-node cluster, however, I see strange behavior. Although my table always gets created, the secondary index often does not! If I delete the table and then create it again (through the same code that I've written), I've never seen the index fail to appear the second time. Does anyone have any idea what to look for here? I have no experience working on a Cassandra cluster and I wonder if maybe I am doing something dumb (I basically just installed DSE and started up the three nodes and that was it). I don't see anything that looks unusual in OpsCenter for DSE. The only thing I've noticed is that the presence of output like the following from my program after executing the command to create the index is perfectly correlated with successful creation of the index: 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Received event EVENT CREATED kiji_retail2.t_model_repo, scheduling delivery 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: [Control connection] Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [b309518a-35d2-3790-bb66-ea39bb0d188c] If anyone can give me a hand, I would really appreciate it. I am out of ideas! Best regards, Clint
Re: Index creation sometimes fails
BTW I have seen this using versions 2.0.1 and 2.0.3 of the java driver on a three-node cluster with DSE 4.5. On Mon, Jul 14, 2014 at 5:51 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi everyone, I have some code that I've been fiddling with today that uses the DataStax Java driver to create a table and then create a secondary index on a column in that table. I've testing this code fairly thoroughly on a single-node Cassandra instance on my laptop and in unit test (using the CassandraDaemon). When running on a three-node cluster, however, I see strange behavior. Although my table always gets created, the secondary index often does not! If I delete the table and then create it again (through the same code that I've written), I've never seen the index fail to appear the second time. Does anyone have any idea what to look for here? I have no experience working on a Cassandra cluster and I wonder if maybe I am doing something dumb (I basically just installed DSE and started up the three nodes and that was it). I don't see anything that looks unusual in OpsCenter for DSE. The only thing I've noticed is that the presence of output like the following from my program after executing the command to create the index is perfectly correlated with successful creation of the index: 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Received event EVENT CREATED kiji_retail2.t_model_repo, scheduling delivery 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: [Control connection] Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.Cluster: Refreshing schema for kiji_retail2 14/07/14 17:40:01 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [9a8d72f9-e384-3aa8-bc85-185e2c303ade, b309518a-35d2-3790-bb66-ea39bb0d188c] 14/07/14 17:40:02 DEBUG com.datastax.driver.core.ControlConnection: Checking for schema agreement: versions are [b309518a-35d2-3790-bb66-ea39bb0d188c] If anyone can give me a hand, I would really appreciate it. I am out of ideas! Best regards, Clint
Setting up DSE 4.5 for mixed workload with BYOH
Hi everyone, Apologies if this is the incorrect forum for a question like this. I am going to set up a mixed-workload (real-time and analytics) installation of DSE 4.5 using bring-your-own Hadoop (BYOH). We are using CDH 5.0. I was reviewing the installation instructions, and I came across the following comment here: http://www.datastax.com/documentation/datastax_enterprise/4.5/datastax_enterprise/byoh/byohInstall.html 6. Observe workload isolation best practices. Do not enable vnodes. Does this mean that the use of vnodes is not compatible with a mixed-workload installation? Or with BYOH? I am confused why this would be the case. If anyone can clarify, I would greatly appreciate it. Best regards, Clint
Re: Setting up DSE 4.5 for mixed workload with BYOH
Hi Tupshin, Thanks for the quick reply. Is the performance concern from the Hadoop integration needing to set up separate SELECT operations for all of the unique vnode ranges? Best regards, Clint On Wed, Jul 2, 2014 at 6:00 PM, Tupshin Harper tups...@tupshin.com wrote: For performance reasons, you shouldn't enable vnodes on any Cassandra/DSE datacenter that is doing hadoop analytics workloads. Other DCs in the cluster can use vnodes. -Tupshin On Jul 2, 2014 5:50 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi everyone, Apologies if this is the incorrect forum for a question like this. I am going to set up a mixed-workload (real-time and analytics) installation of DSE 4.5 using bring-your-own Hadoop (BYOH). We are using CDH 5.0. I was reviewing the installation instructions, and I came across the following comment here: http://www.datastax.com/documentation/datastax_enterprise/4.5/datastax_enterprise/byoh/byohInstall.html 6. Observe workload isolation best practices. Do not enable vnodes. Does this mean that the use of vnodes is not compatible with a mixed-workload installation? Or with BYOH? I am confused why this would be the case. If anyone can clarify, I would greatly appreciate it. Best regards, Clint
Re: Setting up DSE 4.5 for mixed workload with BYOH
Sorry BTW in case what I wrote below is unclear, is the concern that the Hadoop InputFormat (as an example) will need to have a separate InputSplit (which corresponds to a SELECT foo FROM bar WHERE token(baz) min AND token(baz) max) for every vnode instead of for every token? (I assume this would be an increase of several orders of magnitude in the number of input splits.) Best regards, Clint On Wed, Jul 2, 2014 at 6:04 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi Tupshin, Thanks for the quick reply. Is the performance concern from the Hadoop integration needing to set up separate SELECT operations for all of the unique vnode ranges? Best regards, Clint On Wed, Jul 2, 2014 at 6:00 PM, Tupshin Harper tups...@tupshin.com wrote: For performance reasons, you shouldn't enable vnodes on any Cassandra/DSE datacenter that is doing hadoop analytics workloads. Other DCs in the cluster can use vnodes. -Tupshin On Jul 2, 2014 5:50 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi everyone, Apologies if this is the incorrect forum for a question like this. I am going to set up a mixed-workload (real-time and analytics) installation of DSE 4.5 using bring-your-own Hadoop (BYOH). We are using CDH 5.0. I was reviewing the installation instructions, and I came across the following comment here: http://www.datastax.com/documentation/datastax_enterprise/4.5/datastax_enterprise/byoh/byohInstall.html 6. Observe workload isolation best practices. Do not enable vnodes. Does this mean that the use of vnodes is not compatible with a mixed-workload installation? Or with BYOH? I am confused why this would be the case. If anyone can clarify, I would greatly appreciate it. Best regards, Clint
Re: Is the tarball for a given release in a Maven repository somewhere?
Probably! I'll give that a try, thanks! On Thu, May 22, 2014 at 12:11 PM, Chris Burroughs chris.burrou...@gmail.com wrote: Maven central has bin.tar.gz src.tar.gz downloads for the 'apache-cassandra' artifact. Does that work for your use case? http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22apache-cassandra%22 On 05/20/2014 05:30 PM, Clint Kelly wrote: Hi all, I am using the maven assembly plugin to build a project that contains a development environment for a project that we've built at work on top of Cassandra. I'd like this development environment to include the latest release of Cassandra. Is there a maven repo anywhere that contains an artifact with the Cassandra release in it? I'd like to have the same Cassandra tarball that you can download from the website be a dependency for my project. I can then have the assembly plugin untar it and customize some of the conf files before taring up our entire development environment. That way, anyone using our development environment would have access to the various shell scripts and tools. I poked around online and could not find what I was looking for. Any help would be appreciated! Best regards, Clint
Re: Is the tarball for a given release in a Maven repository somewhere?
Thanks, Lewis. I created a ticket here: https://issues.apache.org/jira/browse/CASSANDRA-7283 For now I just copied the cassandra and cassandra.in.sh scripts into my project, along with custom configuration files. We already have all of the necessary JARs in our project's lib directory, since our code depends on them anyway. Best regards, Clint On Wed, May 21, 2014 at 1:02 PM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Hi Clint, On Wed, May 21, 2014 at 5:29 AM, user-digest-h...@cassandra.apache.org wrote: Is the tarball for a given release in a Maven repository somewhere? Hi all, ...snip I poked around online and could not find what I was looking for. Any help would be appreciated! A Maven repos? No. Currently tar.gz builds are not generated and signed with the Cassandra release process and sent to MCentral. This is an ideal opportunity for someone to intervene and improve this process... log a Jira ticket as it would only benefit people if they can find these artifacts in MCental as well. You can however, ALWAYS find archived releases of Cassandra (as with EVERY Apache release) under archive.apache.org Specifically you will be looking here http://archive.apache.org/dist/cassandra/ It is up to the Cassandra release manager and PMC generally to keep the main distribution server clean and uncluttered so hence the reason non-current releases are automatically pulled down to archive.a.o hth Lewis
Is the tarball for a given release in a Maven repository somewhere?
Hi all, I am using the maven assembly plugin to build a project that contains a development environment for a project that we've built at work on top of Cassandra. I'd like this development environment to include the latest release of Cassandra. Is there a maven repo anywhere that contains an artifact with the Cassandra release in it? I'd like to have the same Cassandra tarball that you can download from the website be a dependency for my project. I can then have the assembly plugin untar it and customize some of the conf files before taring up our entire development environment. That way, anyone using our development environment would have access to the various shell scripts and tools. I poked around online and could not find what I was looking for. Any help would be appreciated! Best regards, Clint
Re: Cassandra token range support for Hadoop (ColumnFamilyInputFormat)
Hi Anton, One approach you could look at is to write a custom InputFormat that allows you to limit the token range of rows that you fetch (if the AbstractColumnFamilyInputFormat does not do what you want). Doing so is not too much work. If you look at the class RowIterator within CqlRecordReader, you can see code in the constructor that creates a query with a certain token range: ResultSet rs = session.execute(cqlQuery, type.compose(type.fromString(split.getStartToken())), type.compose(type.fromString(split.getEndToken())) ); I think you can make a new version of the InputFormat and just tweak this method to achieve what you want. Alternatively, if you just want to get a sample of the data, you might want to change the InputFormat itself such that it chooses to query only a subset of the total input splits (or CfSplits). That might be easier. Best regards, Clint On Wed, May 14, 2014 at 6:29 PM, Anton Brazhnyk anton.brazh...@genesys.com wrote: Greetings, I'm reading data from C* with Spark (via ColumnFamilyInputFormat) and I'd like to read just part of it - something like Spark's sample() function. Cassandra's API seems allow to do it with its ConfigHelper.setInputRange(jobConfiguration, startToken, endToken) method, but it doesn't work. The limit is just ignored and the entire column family is scanned. It seems this kind of feature is just not supported and sources of AbstractColumnFamilyInputFormat.getSplits confirm that (IMO). Questions: 1. Am I right that there is no way to get some data limited by token range with ColumnFamilyInputFormat? 2. Is there other way to limit the amount of data read from Cassandra with Spark and ColumnFamilyInputFormat, so that this amount is predictable (like 5% of entire dataset)? WBR, Anton
Hadoop InputFormat that supports multiple queries
Hi everyone, I couple of months ago I started working on a new Hadoop InputFormat that we needed for something at my work. It is in a semi-working state now so I thought I would post a link in case anyone is interested: https://github.com/wibiclint/cassandra2-hadoop2 At the time I started working on this, I wanted a driver that used the DataStax Java driver. I saw that the most recent Cassandra release uses the Java driver to Hadoop integration, so that somewhat obviates what I was doing, but it has a couple of other features that folks might find useful: * You can combine the results of multiple queries to multiple tables * You can group rows based on the partition key, plus some optional subset of the clustering columns Thanks to the folks on the list who helped answer my questions while I was writing this. Best regards, Clint
Re: Error evicting cold readers when launching an EmbeddedCassandraService for a second time
Hi Duy Hai, I was just trying to be extra-paranoid and to make sure that any screw up in one unit test did not at all affect the environment for my other unit tests. What do you mean by truncating tables BTW? Best regards, Clint On Thu, May 1, 2014 at 11:05 AM, DuyHai Doan doanduy...@gmail.com wrote: Hello Clint Why do you need to remove all SSTables or dropping keyspace between tests ? Truncating tables is not enough to have clean and repeatable tests ? Regards Duy Hai DOAN On Thu, May 1, 2014 at 5:54 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi, I am deleting all of the directories for SSTables, etc. between tests. My goal is for each test to start off with a completely blank-slate Cassandra install. I can more-or-less get what I want by just keeping the same EmbeddedCassandraSession active through *all* of my unit tests and then just creating and dropping keyspaces every test, but I'd like to know how to totally start over if I'd like to. Thanks! Best regards, Clint On Thu, May 1, 2014 at 2:15 AM, DuyHai Doan doanduy...@gmail.com wrote: Hello Clint Just one question, are you sure that nothing from your code remove the SSTables between tests ? I'm using extensively the same infrastructure than the EmbeddedCassandraService with Achilles and I have no such issue so far Regards On Wed, Apr 30, 2014 at 8:43 PM, Clint Kelly clint.ke...@gmail.comwrote: Hi all, I have a unit test framework for a Cassandra project that I'm working on. For every one of my test classes, I delete all of the data file, commit log, and saved cache locations, start an EmbeddedCassandraService, and populate a keyspace and tables from scratch. Currently, the unit tests that run in my first test class work fine, but those in my second class die with this error: java.io.FileNotFoundException: /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db (No such file or directory) This error happens immediately after I call EmbeddedCassandraService.start(); I turned on debugging and traced through the code, and I see this right before the error message: 14/04/30 11:22:47 DEBUG org.apache.cassandra.service.FileCacheService: Evicting cold readers for /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db This seems to happen in a callback when a value (in this case, a file reader) is evicted from a Guava cache. I assume that the problem that I have is something like the following: - There is some kind of reading thread associated with target/cassandra/data/system/local/system-local-jb-5-Data.db - Even after I stop my EmbeddedCassandraService and blow away all of the data file, commit log, and saved cache locations from my first unit test, the information about the reader for the now-deleted data file still exists. - Later when this reference expires in the cache and Cassandra goes to notify the reader, the error occurs because the file no longer exists. Does anyone have any suggestions on how to deal with this? Best regards, Clint
Re: Error evicting cold readers when launching an EmbeddedCassandraService for a second time
Hi, I am deleting all of the directories for SSTables, etc. between tests. My goal is for each test to start off with a completely blank-slate Cassandra install. I can more-or-less get what I want by just keeping the same EmbeddedCassandraSession active through *all* of my unit tests and then just creating and dropping keyspaces every test, but I'd like to know how to totally start over if I'd like to. Thanks! Best regards, Clint On Thu, May 1, 2014 at 2:15 AM, DuyHai Doan doanduy...@gmail.com wrote: Hello Clint Just one question, are you sure that nothing from your code remove the SSTables between tests ? I'm using extensively the same infrastructure than the EmbeddedCassandraService with Achilles and I have no such issue so far Regards On Wed, Apr 30, 2014 at 8:43 PM, Clint Kelly clint.ke...@gmail.comwrote: Hi all, I have a unit test framework for a Cassandra project that I'm working on. For every one of my test classes, I delete all of the data file, commit log, and saved cache locations, start an EmbeddedCassandraService, and populate a keyspace and tables from scratch. Currently, the unit tests that run in my first test class work fine, but those in my second class die with this error: java.io.FileNotFoundException: /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db (No such file or directory) This error happens immediately after I call EmbeddedCassandraService.start(); I turned on debugging and traced through the code, and I see this right before the error message: 14/04/30 11:22:47 DEBUG org.apache.cassandra.service.FileCacheService: Evicting cold readers for /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db This seems to happen in a callback when a value (in this case, a file reader) is evicted from a Guava cache. I assume that the problem that I have is something like the following: - There is some kind of reading thread associated with target/cassandra/data/system/local/system-local-jb-5-Data.db - Even after I stop my EmbeddedCassandraService and blow away all of the data file, commit log, and saved cache locations from my first unit test, the information about the reader for the now-deleted data file still exists. - Later when this reference expires in the cache and Cassandra goes to notify the reader, the error occurs because the file no longer exists. Does anyone have any suggestions on how to deal with this? Best regards, Clint
Error evicting cold readers when launching an EmbeddedCassandraService for a second time
Hi all, I have a unit test framework for a Cassandra project that I'm working on. For every one of my test classes, I delete all of the data file, commit log, and saved cache locations, start an EmbeddedCassandraService, and populate a keyspace and tables from scratch. Currently, the unit tests that run in my first test class work fine, but those in my second class die with this error: java.io.FileNotFoundException: /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db (No such file or directory) This error happens immediately after I call EmbeddedCassandraService.start(); I turned on debugging and traced through the code, and I see this right before the error message: 14/04/30 11:22:47 DEBUG org.apache.cassandra.service.FileCacheService: Evicting cold readers for /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db This seems to happen in a callback when a value (in this case, a file reader) is evicted from a Guava cache. I assume that the problem that I have is something like the following: - There is some kind of reading thread associated with target/cassandra/data/system/local/system-local-jb-5-Data.db - Even after I stop my EmbeddedCassandraService and blow away all of the data file, commit log, and saved cache locations from my first unit test, the information about the reader for the now-deleted data file still exists. - Later when this reference expires in the cache and Cassandra goes to notify the reader, the error occurs because the file no longer exists. Does anyone have any suggestions on how to deal with this? Best regards, Clint
Per-keyspace partitioners?
Hi everyone, Is there a way to change the partitioner on a per-table or per-keyspace basis? We have some tables for which we'd like to enable ordered scans of rows, so we'd like to use the ByteOrdered partitioner for those, but use Murmur3 for everything else in our cluster. Is this possible? Or does the partitioner have to be the same for the entire cluster? Best regards, Clint
Re: Meaning of token column in system.peers and system.local
Hi Theo, Thanks for your response. I understand what you are saying with regard to the load balancing. I posted my question to the DataStax list and one of the folks there answered it. I put his response below (for anyone who may be curious): Sylvain Lebresne sylv...@datastax.com 4:03 AM (4 hours ago) to java-driver-us. The system tables are a bit specific in the sense that they are local to the node that coordinate the query. And by default the java driver round robin the queries over the node of the cluster. The result is that more likely than not, your two system queries (on system.local and system.peers) do not reach the same coordinator, hence what you see. It's possible to enforce that both query goes to the same coordinator by mean of modifying/providing a custom load balancing policy. You could for instance write a wrapper Statement class, that allow to specify which node is supposed to be contacted, and then write a custom load balancing policy that recognize this wrapper class and force the user provided host if there is one (and say fallback on another load balancing policy otherwise). Or, simpler but somewhat less flexible, if all you want is to have 2 requests go to the same coordinator (which is enough to get all tokens of a cluster really), then you can make sure to use TokenAwarePolicy (a good idea anyway), and make sure both query have the same routing key (whatever it is is not all that important, you can use an empty ByteBuffer), see SimpleStatement.setRoutingKey(). Note that I would agree that what's suggested above is slightly involved and could be supported more natively by the driver. And I do plan on exposing the cluster tokens more simply in particular (probably directly from the Host object, it's just a todo not yet done. And I'll probably add the load balancing stuff + Statement wrapper I describe above, because that's probably somewhat generally useful for debugging for instance. Still, it's possible to do currently, just a bit more involved than is probably necessary. -- Sylvain On Mon, Mar 31, 2014 at 3:30 AM, Theo Hultberg t...@iconara.net wrote: your assumption about 256 tokens per node is correct. as for you second question, it seems to me like most of your assumptions are correct, but I'm not sure I understand them correctly. hopefully someone else can answer this better. tokens are a property of the cluster and not the keyspace. the first replica of any token will be the same for all keyspaces, but with different replication factors the other replicas will differ. when you query the system.local and system.peers tables you must make sure that you don't connect to other nodes. I think the inconsistency you think you found is because the first and second queries went to different nodes. the java driver will connect to all nodes and load balance requests by default. T# On Mon, Mar 31, 2014 at 4:06 AM, Clint Kelly clint.ke...@gmail.com wrote: BTW one other thing that I have not been able to debug today that maybe someone can help me with: I am using a three-node Cassandra cluster with Vagrant. The nodes in my cluster are 192.168.200.11, 192.168.200.12, and 192.168.200.13. If I use cqlsh to connect to 192.168.200.11, I see unique sets of tokens when I run the following three commands: select tokens from system.local select tokens from system.peers where peer=192.168.200.12 select tokens from system.peers where peer=192.168.200.13 This is what I expect. However, when I tried making an application with the Java driver that does the following: Create a Session by connecting to 192.168.200.11 From that session, select tokens from system.local From that session, select tokens, peer from system.peers Now I get the exact-same set of tokens from system.local and from the row in system.peers in which peer=192.168.200.13. Anyone have any idea why this would happen? I'm not sure how to debug this. I see the following log from the Java driver: 14/03/30 19:05:24 DEBUG com.datastax.driver.core.Cluster: Starting new cluster with contact points [/192.168.200.11] 14/03/30 19:05:24 INFO com.datastax.driver.core.Cluster: New Cassandra host /192.168.200.13 added 14/03/30 19:05:24 INFO com.datastax.driver.core.Cluster: New Cassandra host /192.168.200.12 added I'm running Cassandra 2.0.6 in the virtual machine and I built my application with version 2.0.1 of the driver. Best regards, Clint On Sun, Mar 30, 2014 at 4:51 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi all, I am working on a Hadoop InputFormat implementation that uses only the native protocol Java driver and not the Thrift API. I am currently trying to replicate some of the behavior of Cassandra.client.describe_ring(myKeyspace) from the Thrift API. I would like to do the following: Get a list of all of the token ranges for a cluster For every token range, determine the replica nodes on which the data in the token range resides Estimate
Re: Cassandra Chef cookbook - weird bug with broadcast_address: 10.0.2.15
Hi Marcin, You are correct that I am using Vagrant. Sorry for not specifying that. OMG you are correct. I spent about an hour over the weekend trying to figure out what was going on. I got confused because listen_address is also set to node[:ipaddress], and listen_address was always set correctly, but that is because it was set directly. Oh my goodness. Thanks for your help, this is really really embarrassing! Best regards, Clint On Mon, Mar 31, 2014 at 8:27 AM, Marcin Cabaj marcin.ca...@datasift.com wrote: Hi Clint, I'm guessing you are using vagrant. The thing is cassandra-chef-cookbook use template cassandra.yaml.erb, where you can find: broadcast_address: %= node[:cassandra][:broadcast_address] % which in turn is equal to node[:ipaddress]. Value of node[:ipaddress] depends on how do you configure networking in vagrant/vbox, with default networking configuration node[:ipaddress] is equal 10.0.2.15 hence your broadcast_address. You can setup networking in different way or setup attribute node[:cassandra][:broadcast_address] manually. On Mon, Mar 31, 2014 at 3:03 AM, Clint Kelly clint.ke...@gmail.com wrote: All, Has anyone used the Cassandra Chef cookbook https://github.com/michaelklishin/cassandra-chef-cookbook and seen broadcast_address: 10.0.2.15 in /etc/cassandra/cassandra.yaml? I looked through the source code for the cookbook and I have no idea how this is happening. I was able to fix this by just commenting out the broadcast_address in the template for /etc/cassandra/cassanrda.yaml and moving on, but this is pretty strange! Best regards, Clint
Meaning of token column in system.peers and system.local
Hi all, I am working on a Hadoop InputFormat implementation that uses only the native protocol Java driver and not the Thrift API. I am currently trying to replicate some of the behavior of *Cassandra.client.describe_ring(myKeyspace)* from the Thrift API. I would like to do the following: - Get a list of all of the token ranges for a cluster - For every token range, determine the replica nodes on which the data in the token range resides - Estimate the number of rows for every range of tokens - Groups ranges of tokens on common replica nodes such that we can create a set of input splits for Hadoop with total estimated line counts that are reasonably close to the requested split size Last week I received some much-appreciated help on this list that pointed me to using the system.peers table to get the list of token ranges for the cluster and the corresponding hosts. Today I created a three-node C* cluster in Vagrant (https://github.com/dholbrook/vagrant-cassandra) and tried inspecting some of the system tables. I have a couple of questions now: 1. *How many total unique tokens should I expect to see in my cluster?* If I have three nodes, and each node has a cassandra.yaml with num_tokens = 256, then should I expect a total of 256*3 = 768 distinct vnodes? 2. *How does the creation of vnodes and their assignment to nodes relate to the replication factor for a given keyspace?* I never thought about this until today, and I tried to reread the documentation on virtual nodes, replication in Cassandra, etc., and now I am sadly still confused. Here is what I think I understand. :) - Given a row with a partition key, any client request for an operation on that row will go to a coordinator node in the cluster. - The coordinator node will compute the token value for the row and from that determine a set of replica nodes for that token. - One of the replica nodes I assume is the node that owns the vnode with the token range that encompasses the token - The identity of the owner of this virtual node is a cross-keyspace property - And the other replicas were originally chosen based on the replica-placement strategy - And therefore the other replicas will be different for each keyspace (because replication factors and replica-placement strategy are properties of a keyspace) 3. What do the values in the token column in system.peers and system.local refer to then? - Since these tables appear to be global, and not per-keyspace properties, I assume that they don't have any information about replication in them, is that correct? - If I have three nodes in my cluster, 256 vnodes per node, and I'm using the Murmur3 partitioner, should I then expect to see the values of tokens in system.peers and system.local be 768 evenly-distributed values between -2^63 and 2^63? 4. Is there any other way, without using Thift, to get as much information as possible about what nodes contain replicas of data for all of the token ranges in a given cluster? I really appreciate any help, thanks! Best regards, Clint
Cassandra Chef cookbook - weird bug with broadcast_address: 10.0.2.15
All, Has anyone used the Cassandra Chef cookbook https://github.com/michaelklishin/cassandra-chef-cookbook and seen broadcast_address: 10.0.2.15 in /etc/cassandra/cassandra.yaml? I looked through the source code for the cookbook and I have no idea how this is happening. I was able to fix this by just commenting out the broadcast_address in the template for /etc/cassandra/cassanrda.yaml and moving on, but this is pretty strange! Best regards, Clint
Re: Meaning of token column in system.peers and system.local
BTW one other thing that I have not been able to debug today that maybe someone can help me with: I am using a three-node Cassandra cluster with Vagrant. The nodes in my cluster are 192.168.200.11, 192.168.200.12, and 192.168.200.13. If I use cqlsh to connect to 192.168.200.11, I see unique sets of tokens when I run the following three commands: select tokens from system.local select tokens from system.peers where peer=192.168.200.12 select tokens from system.peers where peer=192.168.200.13 This is what I expect. However, when I tried making an application with the Java driver that does the following: - Create a Session by connecting to 192.168.200.11 - From that session, select tokens from system.local - From that session, select tokens, peer from system.peers Now I get the exact-same set of tokens from system.local and from the row in system.peers in which peer=192.168.200.13. Anyone have any idea why this would happen? I'm not sure how to debug this. I see the following log from the Java driver: 14/03/30 19:05:24 DEBUG com.datastax.driver.core.Cluster: Starting new cluster with contact points [/192.168.200.11] 14/03/30 19:05:24 INFO com.datastax.driver.core.Cluster: New Cassandra host /192.168.200.13 added 14/03/30 19:05:24 INFO com.datastax.driver.core.Cluster: New Cassandra host /192.168.200.12 added I'm running Cassandra 2.0.6 in the virtual machine and I built my application with version 2.0.1 of the driver. Best regards, Clint On Sun, Mar 30, 2014 at 4:51 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi all, I am working on a Hadoop InputFormat implementation that uses only the native protocol Java driver and not the Thrift API. I am currently trying to replicate some of the behavior of *Cassandra.client.describe_ring(myKeyspace)* from the Thrift API. I would like to do the following: - Get a list of all of the token ranges for a cluster - For every token range, determine the replica nodes on which the data in the token range resides - Estimate the number of rows for every range of tokens - Groups ranges of tokens on common replica nodes such that we can create a set of input splits for Hadoop with total estimated line counts that are reasonably close to the requested split size Last week I received some much-appreciated help on this list that pointed me to using the system.peers table to get the list of token ranges for the cluster and the corresponding hosts. Today I created a three-node C* cluster in Vagrant (https://github.com/dholbrook/vagrant-cassandra) and tried inspecting some of the system tables. I have a couple of questions now: 1. *How many total unique tokens should I expect to see in my cluster?* If I have three nodes, and each node has a cassandra.yaml with num_tokens = 256, then should I expect a total of 256*3 = 768 distinct vnodes? 2. *How does the creation of vnodes and their assignment to nodes relate to the replication factor for a given keyspace?* I never thought about this until today, and I tried to reread the documentation on virtual nodes, replication in Cassandra, etc., and now I am sadly still confused. Here is what I think I understand. :) - Given a row with a partition key, any client request for an operation on that row will go to a coordinator node in the cluster. - The coordinator node will compute the token value for the row and from that determine a set of replica nodes for that token. - One of the replica nodes I assume is the node that owns the vnode with the token range that encompasses the token - The identity of the owner of this virtual node is a cross-keyspace property - And the other replicas were originally chosen based on the replica-placement strategy - And therefore the other replicas will be different for each keyspace (because replication factors and replica-placement strategy are properties of a keyspace) 3. What do the values in the token column in system.peers and system.local refer to then? - Since these tables appear to be global, and not per-keyspace properties, I assume that they don't have any information about replication in them, is that correct? - If I have three nodes in my cluster, 256 vnodes per node, and I'm using the Murmur3 partitioner, should I then expect to see the values of tokens in system.peers and system.local be 768 evenly-distributed values between -2^63 and 2^63? 4. Is there any other way, without using Thift, to get as much information as possible about what nodes contain replicas of data for all of the token ranges in a given cluster? I really appreciate any help, thanks! Best regards, Clint
How to tear down an EmbeddedCassandraService in unit tests?
All, I have a question about how to use the EmbeddedCassandraService in unit tests. I wrote a short collection of unit tests here: https://github.com/wibiclint/cassandra-java-driver-keyspaces I'm trying to start up a new EmbeddedCassandraService for each unit test. I looked at the Cassandra source code to try to see how that happens there and replicated it as well as I could here. My first unit test works great, but in subsequent unit tests I get this error: java.lang.RuntimeException: java.io.FileNotFoundException: target/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-2-Data.db (No such file or directory) I assume that this is because I am not shutting down the EmbeddedCassandraService in the first unit test correctly (I do not have any @After method). Does anyone have any advice on how to clean up the EmbeddedCassandraService between unit tests? I can instead create the EmbeddedCassandraService in a static @BeforeClass method and then have every unit test use a different keyspace, but that strikes me as somewhat sloppy and I'd rather understand what I'm doing well enough to be able to have one service per test if necessary. Thanks! Best regards, Clint
Building a maven project that depends on Cassandra 2.0.6 or 2.1 beta?
Folks, Can anyone instruct me about how to set up a maven project that depends on either 2.0.6 or 2.1? I am interested in using some of the new features (e.g., static columns) in my current project. Being able to just install one of these versions in my local maven repository would be good enough for now (until either of these becomes an official release). Usually I can do this myself pretty easily, but I've become a little bit lost with Ant and Cassandra and given up after a few hours. :) Alternatively, if there is a repo from which I can grab 2.0.6 somewhere, that would be great. Thanks! Best regards, Clint
Re: CQL: Any way to have inequalities on multiple clustering columns in a WHERE clause?
Yes, thank you!
On Thu, Feb 27, 2014 at 10:26 PM, DuyHai Doan doanduy...@gmail.com wrote:
Clint, what you want is this :
https://issues.apache.org/jira/browse/CASSANDRA-4851
select * from foo where key=something and fam = 'Info' and (qual,version)
('A',2013) and qual 'D' ALLOW FILTERING
On Fri, Feb 28, 2014 at 6:57 AM, Clint Kelly clint.ke...@gmail.comwrote:
All,
Is there any way to have inequalities comparisons on multiple clustering
columns in a WHERE clause in CQL? For example, I'd like to do:
select * from foo where fam = 'Info' and qual 'A' and qual 'D' and
version 2013 ALLOW FILTERING;
I get an error:
Bad Request: PRIMARY KEY part version cannot be restricted (preceding
part qual is either not restricted or by a non-EQ relation)
when I try this. Is there any way to make a query like this work? I
understand that this query will not be a nice, continuous scan of data, but
I'd rather have a slower query than have to do all of this filtering on the
client side. Any other suggestions?
BTW my table looks like this:
CREATE TABLE foo (
key text,
fam text,
qual text,
version int,
val text,
PRIMARY KEY (key, fam, qual, version)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
Best regards,
Clint
Re: Getting the most-recent version from time-series data
Hi Tupshin,
Thanks for your help once again, I really appreciate it. Quick question
regarding the issue of token-aware routing, etc. Let's say that I am using
the table described earlier:
CREATE TABLE time_series_stuff (
key text,
family text,
version int,
val text,
PRIMARY KEY (key, family, version)
) WITH CLUSTERING ORDER BY (family ASC, version DESC)
I want to retrieve values for the most-recent version of every family for a
given key, doing something like:
SELECT * from time_series_stuff where key='mykey'
but getting only one version per family.
All of this data should live on the same node (or set of replica nodes),
correct? I am specifying the partition key here, and I thought that only
the partition key determined on what physical nodes data exists.
Therefore, I would think that all of the results from this query would come
from a single replica node (or set of replica nodes, if the consistency
level is greater than 1).
Would you mind clarifying? Thanks a lot!
Best regards,
Clint
On Wed, Feb 26, 2014 at 4:56 AM, Tupshin Harper tups...@tupshin.com wrote:
And one last clarification. Where I said stored procedure earlier, I
meant prepared statement. Sorry for the confusion. Too much typing while
tired.
-Tupshin
On Tue, Feb 25, 2014 at 10:36 PM, Tupshin Harper tups...@tupshin.comwrote:
I failed to address the matter of not knowing the families in advance.
I can't really recommend any solution to that other than storing the list
of families in another structure that is readily queryable. I don't know
how many families you are thinking, but if it is in the millions or more,
You might consider constructing another table such as:
CREATE TABLE families (
key int,
family text,
PRIMARY KEY (key, family)
);
store your families there, with a knowable set of keys (I suggest
something like the last 3 digits of the md5 hash of the family). So then
you could retrieve your families in nice sized batches
SELECT family FROM id WHERE key=0;
and then do the fan-out selects that I described previously.
-Tupshin
On Tue, Feb 25, 2014 at 10:15 PM, Tupshin Harper tups...@tupshin.comwrote:
Hi Clint,
What you are describing could actually be accomplished with the Thrift
API and a multiget_slice with a slicerange having a count of 1. Initially I
was thinking that this was an important feature gap between Thrift and CQL,
and was going to suggest that it should be implemented (possible syntax is
in https://issues.apache.org/jira/browse/CASSANDRA-6167 which is almost
a superset of this feature).
But then I was convinced by some colleagues, that with a modern CQL
driver that is token aware, you are actually better off (in terms of
latency, throughput, and reliability), by doing each query separately on
the client.
The reasoning is that if you did this with a single query, it would
necessarily be sent to a coordinator that wouldn't own most of the data
that you are looking for. That coordinator would then need to fan out the
read to all the nodes owning the partitions you are looking for.
Far better to just do it directly on the client. The token aware client
will send each request for a row straight to a node that owns it. With a
separate connection open to each node, this is done in parallel from the
get-go. Fewer hops. Less load on the coordinator. No bottlenecks. And with
a stored procedure, very very little additional overhead to the client,
server, or network.
-Tupshin
On Tue, Feb 25, 2014 at 7:48 PM, Clint Kelly clint.ke...@gmail.comwrote:
Hi everyone,
Let's say that I have a table that looks like the following:
CREATE TABLE time_series_stuff (
key text,
family text,
version int,
val text,
PRIMARY KEY (key, family, version)
) WITH CLUSTERING ORDER BY (family ASC, version DESC) AND
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
cqlsh:fiddle select * from time_series_stuff ;
key| family | version | val
+-+-+
monday | revenue | 3 | $$
monday | revenue | 2 |$$$
monday | revenue | 1 | $$
monday | revenue | 0 | $
monday | traffic | 2 | medium
monday | traffic | 1 | light
monday | traffic | 0 | heavy
(7 rows)
Now let's say that I'd like to perform a query that gets me the most
recent N versions of revenue and traffic.
Is there a CQL query to do this? Let's say that N=1. Then I know that
I can do:
cqlsh:fiddle select
Re: Combine multiple SELECT statements into one RPC?
Hi Sylvain, Thanks for your response. I am writing code to allow users to query a table that looks something like this: CREATE TABLE time_series_stuff ( key text, family text, qualifier text, version long, val blob, PRIMARY KEY (key, family, qualifier, version) ) WITH CLUSTERING ORDER BY (family ASC, qualifier ASC, version DESC) To a user, working with this table looks something like working with a nested map (from family - qualifier - version - value). A user of our system can create a single data request that indicates that he or she would like to fetch data for a given (family, qualifier, version) value, but a user can also do things like: - Specify only a (family, qualifier) - fetch all of the versions - Specify (family, qualifier, min and max versions) - fetch all of the versions in a range - Specify (family) - fetch all qualifiers, all versions in a family - Specify (family, *, min and max versions) - fetch all values with version range for any qualifier etc. So some of the data requests from users can translate into multiple C* SELECT statements. I was just looking for a way to combine them into a single client / server transaction, if possible. At the moment I'm just issuing multiple SELECT statements, which is fine, but I was wondering if there was a way to combine them together somehow. Thanks for your help! Best regards, Clint On Thu, Feb 27, 2014 at 1:05 AM, Sylvain Lebresne sylv...@datastax.comwrote: On Thu, Feb 27, 2014 at 1:00 AM, Clint Kelly clint.ke...@gmail.comwrote: Hi all, Is there any way to use the DataStax Java driver to combine multiple SELECT statements into a single RPC? I assume not (I could not find anything about this in the documentation), but I just wanted to check. The short answer is no. The slightly longer answer is that the DataStax Java driver uses the so-called native protocol. And that protocol does not allow to have multiple SELECT into a single protocol message (the protocol is not really RPC-based strictly speaking so I'll assume you meant one client-server message here), and it follows that the driver can't either. But I'll note that the reason why the protocol doesn't have such a thing is that it's generally a better idea to parallelize your SELECT client side, though since you haven't provided much context for you question I'd rather not go into too much details here since that might be off-topic. -- Sylvain
Re: Getting the most-recent version from time-series data
Hi Tupshin, BTW, you asked earlier about the number of different distinct family values. There could easily be millions of different families, each with many different values. Right now I see two options: 1. Query the table once just to get all of the distinct families, then do separate queries for each family to get the most-recent version. 2. Read back all of the versions of all of the families and then filter on the client side. Neither one of these are great solutions, although once we have users reading back millions of values in a single query, they will have to indicate (to our software that sits on top of C*) that they are going to use paging, and then we are going to be doing multiple client / server operations anyway. I'd just like to minimize them. :) Best regards, Clint On Fri, Feb 28, 2014 at 9:47 AM, Clint Kelly clint.ke...@gmail.com wrote: Hi Tupshin, Thanks for your help once again, I really appreciate it. Quick question regarding the issue of token-aware routing, etc. Let's say that I am using the table described earlier: CREATE TABLE time_series_stuff ( key text, family text, version int, val text, PRIMARY KEY (key, family, version) ) WITH CLUSTERING ORDER BY (family ASC, version DESC) I want to retrieve values for the most-recent version of every family for a given key, doing something like: SELECT * from time_series_stuff where key='mykey' but getting only one version per family. All of this data should live on the same node (or set of replica nodes), correct? I am specifying the partition key here, and I thought that only the partition key determined on what physical nodes data exists. Therefore, I would think that all of the results from this query would come from a single replica node (or set of replica nodes, if the consistency level is greater than 1). Would you mind clarifying? Thanks a lot! Best regards, Clint On Wed, Feb 26, 2014 at 4:56 AM, Tupshin Harper tups...@tupshin.comwrote: And one last clarification. Where I said stored procedure earlier, I meant prepared statement. Sorry for the confusion. Too much typing while tired. -Tupshin On Tue, Feb 25, 2014 at 10:36 PM, Tupshin Harper tups...@tupshin.comwrote: I failed to address the matter of not knowing the families in advance. I can't really recommend any solution to that other than storing the list of families in another structure that is readily queryable. I don't know how many families you are thinking, but if it is in the millions or more, You might consider constructing another table such as: CREATE TABLE families ( key int, family text, PRIMARY KEY (key, family) ); store your families there, with a knowable set of keys (I suggest something like the last 3 digits of the md5 hash of the family). So then you could retrieve your families in nice sized batches SELECT family FROM id WHERE key=0; and then do the fan-out selects that I described previously. -Tupshin On Tue, Feb 25, 2014 at 10:15 PM, Tupshin Harper tups...@tupshin.comwrote: Hi Clint, What you are describing could actually be accomplished with the Thrift API and a multiget_slice with a slicerange having a count of 1. Initially I was thinking that this was an important feature gap between Thrift and CQL, and was going to suggest that it should be implemented (possible syntax is in https://issues.apache.org/jira/browse/CASSANDRA-6167 which is almost a superset of this feature). But then I was convinced by some colleagues, that with a modern CQL driver that is token aware, you are actually better off (in terms of latency, throughput, and reliability), by doing each query separately on the client. The reasoning is that if you did this with a single query, it would necessarily be sent to a coordinator that wouldn't own most of the data that you are looking for. That coordinator would then need to fan out the read to all the nodes owning the partitions you are looking for. Far better to just do it directly on the client. The token aware client will send each request for a row straight to a node that owns it. With a separate connection open to each node, this is done in parallel from the get-go. Fewer hops. Less load on the coordinator. No bottlenecks. And with a stored procedure, very very little additional overhead to the client, server, or network. -Tupshin On Tue, Feb 25, 2014 at 7:48 PM, Clint Kelly clint.ke...@gmail.comwrote: Hi everyone, Let's say that I have a table that looks like the following: CREATE TABLE time_series_stuff ( key text, family text, version int, val text, PRIMARY KEY (key, family, version) ) WITH CLUSTERING ORDER BY (family ASC, version DESC) AND bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND comment='' AND dclocal_read_repair_chance=0.00 AND gc_grace_seconds=864000 AND index_interval=128 AND read_repair_chance=0.10
Any way to get a list of per-node token ranges using the DataStax Java driver?
Hi everyone, I've been working on a rewrite of the Cassandra InputFormat for Hadoop 2 using the DataStax Java driver instead of the Thrift API. I have a prototype working now, but there is one bit of code that I have not been able to replace with code for the Java driver. In the InputFormat#getSplits method, the old code has a call like the following: map = client.describe_ring(ConfigHelper.getInputKeyspace(conf)); This gets a list of the distinct token ranges for the Cassandra cluster. The rest of getSplits then takes these key ranges, breaks them up into subranges (to match the user-specified input split size), and then gets the replica nodes for the various token ranges (as the locations for the splits). Does anyone know how I can do the following with the native protocol? - Get the distinct token ranges for the C* cluster - Get the set of replica nodes for a given range of tokens? I tried looking around in Cluster and Metadata, among other places, in the API docs, but I didn't see anything that looked like it would do what I want. Thanks! Best regards, Clint
Re: Resetting a counter in CQL
Great, thanks! On Fri, Feb 28, 2014 at 4:38 PM, Tyler Hobbs ty...@datastax.com wrote: On Fri, Feb 28, 2014 at 6:32 PM, Clint Kelly clint.ke...@gmail.comwrote: What is the best known method for resetting a counter in CQL? Is it best to read the counter and then increment it by a negative amount? Do this. Or to delete the row and then increment it by zero? Don't do this. When you delete a counter, you are basically saying I will never use this counter again. If you try to use it again, the behavior is undefined. It's one of the documented limitations of counters. -- Tyler Hobbs DataStax http://datastax.com/
Re: Naming variables in a prepared statement in the DataStax Java driver
Ah never mind, I see, currently you can refer to the ?'s by name by using
the name of the column to which the ? refers. And this works as long as
each column is present only one in the statement.
Sorry for the extra list traffic!
On Thu, Feb 27, 2014 at 7:33 PM, Clint Kelly clint.ke...@gmail.com wrote:
Folks,
Is there a way to name the variables in a prepared statement when using
the DataStax Java driver?
For example, instead of doing:
ByteBuffer byteBuffer = ... // some application logic
String query = SELECT * FROM foo WHERE bar = ?;
PreparedStatement preparedStatement = session.prepare(query);
BoundStatement boundStatement = preparedStatement.bind(byteBuffer);
I'd like to be able to be able to name the fields indicated by the ?s,
e.g.,:
ByteBuffer byteBuffer = ... // some application logic
String query = SELECT * FROM foo WHERE bar = ?bar; // I just made up
this syntax
PreparedStatement preparedStatement = session.prepare(query);
BoundStatement boundStatement = preparedStatement.bind(bar, byteBuffer);
Looking at the DataStax API docs, it seems like there should be a way to
be able to do this, but I can't tell for sure.
This would be particularly useful when I have some application logic in
which I have very long queries with lots of bound variables and then
sometimes extend them with different clauses. Right now this code gets
very verbose, because I cannot figure out how to break up my bind
statements to bind different values to a bound statement in separate
statements. In other words, I'd like to be able to do something like:
BoundStatement boundStatement = // Create from a prepared statement
boundStatement = boundStatement.bind( ... ); // Bind all of the values
that I use in every flavor of this query
if ( ... ) {
boundStatement = boundStatement.bind(some field, someVal);
else {
boundStatement = boundStatement.bind(other field, otherVal);
Thanks!
Best regards,
Clint
CQL: Any way to have inequalities on multiple clustering columns in a WHERE clause?
All,
Is there any way to have inequalities comparisons on multiple clustering
columns in a WHERE clause in CQL? For example, I'd like to do:
select * from foo where fam = 'Info' and qual 'A' and qual 'D' and
version 2013 ALLOW FILTERING;
I get an error:
Bad Request: PRIMARY KEY part version cannot be restricted (preceding
part qual is either not restricted or by a non-EQ relation)
when I try this. Is there any way to make a query like this work? I
understand that this query will not be a nice, continuous scan of data, but
I'd rather have a slower query than have to do all of this filtering on the
client side. Any other suggestions?
BTW my table looks like this:
CREATE TABLE foo (
key text,
fam text,
qual text,
version int,
val text,
PRIMARY KEY (key, fam, qual, version)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
Best regards,
Clint
Re: Update multiple rows in a CQL lightweight transaction
Thanks for your help everyone.
Sylvain, as I understand it, the scenario I described above is not resolved
by CASSANDRA-6561, correct?
(This scenario may not matter to most folks, which is totally fine, I just
want to make sure that I understand.)
Should I instead look into using the Thrift API to address this?
Best regards,
Clint
On Tue, Feb 25, 2014 at 11:30 PM, Sylvain Lebresne sylv...@datastax.comwrote:
Sorry to interfere again here but CASSANDRA-5633 will not be picked up
because pretty much everything it was set to fix is fixed by
CASSANDRA-6561, this is *not* a syntax problem anymore.
On Wed, Feb 26, 2014 at 3:18 AM, Tupshin Harper tups...@tupshin.comwrote:
Unfortunately there is no option to vote for a resolved ticket, but if
you can propose a better syntax that people agree on, you could probably
get some fresh traction on it.
-Tupshin
On Feb 25, 2014 7:20 PM, Clint Kelly clint.ke...@gmail.com wrote:
Hi Tupshin,
Thanks for your help! Unfortunately in my case, I will need to do a
compare and set in which the compare is against a value in a dynamic column.
In general, I need to be able to do the following:
- Check whether a given value exists in a dynamic column
- If so, perform some number of insertions / deletions for dynamic
columns in the same row (i.e., with the same partition key as the dynamic
column used for the compare)
I think you are correct that I need
https://issues.apache.org/jira/browse/CASSANDRA-5633 to be
implemented. Is there any way to vote for that to get picked up again? :)
Best regards,
Clint
On Mon, Feb 24, 2014 at 2:32 PM, Tupshin Harper tups...@tupshin.comwrote:
Hi Clint,
That does appear to be an omission in CQL3. It would be possible to
simulate it by doing
BEGIN BATCH
UPDATE foo SET z = 10 WHERE x = 'a' AND y = 1 IF t= 2 AND z=10;
UPDATE foo SET t = 5,z=6 where x = 'a' AND y = 4
APPLY BATCH;
However, this does a redundant write to the first row if the condition
holds, and I certainly wouldn't recommend doing that routinely.
Alternatively, depending on your needs, you might be able to use a
static column (coming with 2.0.6) as your conditional flag, as that column
is shared by all rows in the partition.
-Tupshin
On Mon, Feb 24, 2014 at 3:57 PM, Clint Kelly clint.ke...@gmail.comwrote:
Hi Tupshin,
Thanks for your help; I appreciate it.
Could I do something like the following?
Given the same table you started with:
x | y | t | z
---+---+---+
a | 1 | 2 | 10
a | 2 | 2 | 20
I'd like to write a compare-and-set that does something like:
If there is a row with (x,y,t,z) = (a,1,2,10), then update/insert a
row with (x,y,t,z) = (a,3,4,5) and update/insert a row with (x,y,t,z)
= (a,4,5,6).
I don't see how I could do this with what you outlined above---just
curious. It seems like what I describe above under the hood would be
a compare-and-(batch)-set on a single wide row, so it maybe is
possible with the Thrift API (I have to check).
Thanks again!
Best regards,
Clint
On Sat, Feb 22, 2014 at 11:38 AM, Tupshin Harper tups...@tupshin.com
wrote:
#5633 was actually closed because the static columns feature
(https://issues.apache.org/jira/browse/CASSANDRA-6561) which has
been
checked in to the 2.0 branch but is not yet part of a release (it
will be in
2.0.6).
That feature will let you update multiple rows within a single
partition by
doing a CAS write based on a static column shared by all rows within
the
partition.
Example extracted from the ticket:
CREATE TABLE foo (
x text,
y bigint,
t bigint static,
z bigint,
PRIMARY KEY (x, y) );
insert into foo (x,y,t, z) values ('a', 1, 1, 10);
insert into foo (x,y,t, z) values ('a', 2, 2, 20);
select * from foo;
x | y | t | z
---+---+---+
a | 1 | 2 | 10
a | 2 | 2 | 20
(Note that both values of t are 2 because it is static)
begin batch update foo set z = 1 where x = 'a' and y = 1; update
foo set z
= 2 where x = 'a' and y = 2 if t = 4; apply batch;
[applied] | x | y| t
---+---+--+---
False | a | null | 2
(Both updates failed to apply because there was an unmet conditional
on one
of them)
select * from foo;
x | y | t | z
---+---+---+
a | 1 | 2 | 10
a | 2 | 2 | 20
begin batch update foo set z = 1 where x = 'a' and y = 1; update foo
set z =
2 where x = 'a' and y = 2 if t = 2; apply batch;
[applied]
---
True
(both updates succeeded because the check on t succeeded)
select * from foo;
x | y | t | z
---+---+---+---
a | 1 | 2 | 1
a | 2 | 2 | 2
Hope this helps.
-Tupshin
On Fri, Feb 21, 2014 at 6:05 PM, DuyHai Doan doanduy...@gmail.com
wrote:
Hello Clint
The Resolution status of the JIRA is set to Later, probably the
implementation is not done yet. The JIRA was opened to discuss
about impl
strategy
Combine multiple SELECT statements into one RPC?
Hi all, Is there any way to use the DataStax Java driver to combine multiple SELECT statements into a single RPC? I assume not (I could not find anything about this in the documentation), but I just wanted to check. Thanks! Best regards, Clint
Re: Update multiple rows in a CQL lightweight transaction
Hi Tupshin,
Thanks for your help! Unfortunately in my case, I will need to do a
compare and set in which the compare is against a value in a dynamic column.
In general, I need to be able to do the following:
- Check whether a given value exists in a dynamic column
- If so, perform some number of insertions / deletions for dynamic
columns in the same row (i.e., with the same partition key as the dynamic
column used for the compare)
I think you are correct that I need
https://issues.apache.org/jira/browse/CASSANDRA-5633 to be implemented. Is
there any way to vote for that to get picked up again? :)
Best regards,
Clint
On Mon, Feb 24, 2014 at 2:32 PM, Tupshin Harper tups...@tupshin.com wrote:
Hi Clint,
That does appear to be an omission in CQL3. It would be possible to
simulate it by doing
BEGIN BATCH
UPDATE foo SET z = 10 WHERE x = 'a' AND y = 1 IF t= 2 AND z=10;
UPDATE foo SET t = 5,z=6 where x = 'a' AND y = 4
APPLY BATCH;
However, this does a redundant write to the first row if the condition
holds, and I certainly wouldn't recommend doing that routinely.
Alternatively, depending on your needs, you might be able to use a static
column (coming with 2.0.6) as your conditional flag, as that column is
shared by all rows in the partition.
-Tupshin
On Mon, Feb 24, 2014 at 3:57 PM, Clint Kelly clint.ke...@gmail.comwrote:
Hi Tupshin,
Thanks for your help; I appreciate it.
Could I do something like the following?
Given the same table you started with:
x | y | t | z
---+---+---+
a | 1 | 2 | 10
a | 2 | 2 | 20
I'd like to write a compare-and-set that does something like:
If there is a row with (x,y,t,z) = (a,1,2,10), then update/insert a
row with (x,y,t,z) = (a,3,4,5) and update/insert a row with (x,y,t,z)
= (a,4,5,6).
I don't see how I could do this with what you outlined above---just
curious. It seems like what I describe above under the hood would be
a compare-and-(batch)-set on a single wide row, so it maybe is
possible with the Thrift API (I have to check).
Thanks again!
Best regards,
Clint
On Sat, Feb 22, 2014 at 11:38 AM, Tupshin Harper tups...@tupshin.com
wrote:
#5633 was actually closed because the static columns feature
(https://issues.apache.org/jira/browse/CASSANDRA-6561) which has been
checked in to the 2.0 branch but is not yet part of a release (it will
be in
2.0.6).
That feature will let you update multiple rows within a single
partition by
doing a CAS write based on a static column shared by all rows within the
partition.
Example extracted from the ticket:
CREATE TABLE foo (
x text,
y bigint,
t bigint static,
z bigint,
PRIMARY KEY (x, y) );
insert into foo (x,y,t, z) values ('a', 1, 1, 10);
insert into foo (x,y,t, z) values ('a', 2, 2, 20);
select * from foo;
x | y | t | z
---+---+---+
a | 1 | 2 | 10
a | 2 | 2 | 20
(Note that both values of t are 2 because it is static)
begin batch update foo set z = 1 where x = 'a' and y = 1; update foo
set z
= 2 where x = 'a' and y = 2 if t = 4; apply batch;
[applied] | x | y| t
---+---+--+---
False | a | null | 2
(Both updates failed to apply because there was an unmet conditional on
one
of them)
select * from foo;
x | y | t | z
---+---+---+
a | 1 | 2 | 10
a | 2 | 2 | 20
begin batch update foo set z = 1 where x = 'a' and y = 1; update foo
set z =
2 where x = 'a' and y = 2 if t = 2; apply batch;
[applied]
---
True
(both updates succeeded because the check on t succeeded)
select * from foo;
x | y | t | z
---+---+---+---
a | 1 | 2 | 1
a | 2 | 2 | 2
Hope this helps.
-Tupshin
On Fri, Feb 21, 2014 at 6:05 PM, DuyHai Doan doanduy...@gmail.com
wrote:
Hello Clint
The Resolution status of the JIRA is set to Later, probably the
implementation is not done yet. The JIRA was opened to discuss about
impl
strategy but nothing has been coded so far I guess.
On Sat, Feb 22, 2014 at 12:02 AM, Clint Kelly clint.ke...@gmail.com
wrote:
Folks,
Does anyone know how I can modify multiple rows at once in a
lightweight transaction in CQL3?
I saw the following ticket:
https://issues.apache.org/jira/browse/CASSANDRA-5633
but it was not obvious to me from the comments how (or whether) this
got resolved. I also couldn't find anything in the DataStax
documentation about how to perform these operations.
I'm in particular interested in how to perform a compare-and-set
operation that modifies multiple rows (with the same partition key)
using the DataStax Java driver.
Thanks!
Best regards,
Clint
Re: Getting the most-recent version from time-series data
Hi Jonathan,
Thanks for the suggestion! I see a couple of problems with this approach:
1. I do not know a priori all of the family names (so I still would not
know what value to use for LIMIT).
2. The versions here are similar to timestamps, so one family may get
updated far more often than the other. Hence, if I order all of my data by
version, then the first 1000 rows in version order could all be from the
same family---I want to just get the most recent value (or N-most recent
values) for each unique family.
I don't think there is a way to do this without performing some client-side
filtering, but I thought I'd see if anyone has any ideas. I'm translating
a framework that was originally designed on top of HBase, so offering this
kind of functionality (by using HBases timestamp dimension) was
previously easy. :)
Best regards,
Clint
On Tue, Feb 25, 2014 at 4:51 PM, Jonathan Lacefield jlacefi...@datastax.com
wrote:
Clint
One approach would be to create a copy of this table and switch the
clustering columns around so version precedes family. This way you
could easily grab the 1st, 2nd, N version rows. Would this help you
in your situation?
Jonathan
On Feb 25, 2014, at 7:49 PM, Clint Kelly clint.ke...@gmail.com wrote:
Hi everyone,
Let's say that I have a table that looks like the following:
CREATE TABLE time_series_stuff (
key text,
family text,
version int,
val text,
PRIMARY KEY (key, family, version)
) WITH CLUSTERING ORDER BY (family ASC, version DESC) AND
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
cqlsh:fiddle select * from time_series_stuff ;
key| family | version | val
+-+-+
monday | revenue | 3 | $$
monday | revenue | 2 |$$$
monday | revenue | 1 | $$
monday | revenue | 0 | $
monday | traffic | 2 | medium
monday | traffic | 1 | light
monday | traffic | 0 | heavy
(7 rows)
Now let's say that I'd like to perform a query that gets me the most
recent N versions of revenue and traffic.
Is there a CQL query to do this? Let's say that N=1. Then I know that
I can do:
cqlsh:fiddle select * from time_series_stuff where key='monday' and
family='revenue' limit 1;
key| family | version | val
+-+-+
monday | revenue | 3 | $$
(1 rows)
cqlsh:fiddle select * from time_series_stuff where key='monday' and
family='traffic' limit 1;
key| family | version | val
+-+-+
monday | traffic | 2 | medium
(1 rows)
But what if I have lots of families and I want to get the most recent
N versions of all of them in a single CQL statement. Is that possible?
Unfortunately I am working on something where the family names and the
number of most-recent versions are not known a priori (I am porting some
code that was designed for HBase).
Best regards,
Clint
Re: Update multiple rows in a CQL lightweight transaction
Hi Tupshin,
Thanks for your help; I appreciate it.
Could I do something like the following?
Given the same table you started with:
x | y | t | z
---+---+---+
a | 1 | 2 | 10
a | 2 | 2 | 20
I'd like to write a compare-and-set that does something like:
If there is a row with (x,y,t,z) = (a,1,2,10), then update/insert a
row with (x,y,t,z) = (a,3,4,5) and update/insert a row with (x,y,t,z)
= (a,4,5,6).
I don't see how I could do this with what you outlined above---just
curious. It seems like what I describe above under the hood would be
a compare-and-(batch)-set on a single wide row, so it maybe is
possible with the Thrift API (I have to check).
Thanks again!
Best regards,
Clint
On Sat, Feb 22, 2014 at 11:38 AM, Tupshin Harper tups...@tupshin.com wrote:
#5633 was actually closed because the static columns feature
(https://issues.apache.org/jira/browse/CASSANDRA-6561) which has been
checked in to the 2.0 branch but is not yet part of a release (it will be in
2.0.6).
That feature will let you update multiple rows within a single partition by
doing a CAS write based on a static column shared by all rows within the
partition.
Example extracted from the ticket:
CREATE TABLE foo (
x text,
y bigint,
t bigint static,
z bigint,
PRIMARY KEY (x, y) );
insert into foo (x,y,t, z) values ('a', 1, 1, 10);
insert into foo (x,y,t, z) values ('a', 2, 2, 20);
select * from foo;
x | y | t | z
---+---+---+
a | 1 | 2 | 10
a | 2 | 2 | 20
(Note that both values of t are 2 because it is static)
begin batch update foo set z = 1 where x = 'a' and y = 1; update foo set z
= 2 where x = 'a' and y = 2 if t = 4; apply batch;
[applied] | x | y| t
---+---+--+---
False | a | null | 2
(Both updates failed to apply because there was an unmet conditional on one
of them)
select * from foo;
x | y | t | z
---+---+---+
a | 1 | 2 | 10
a | 2 | 2 | 20
begin batch update foo set z = 1 where x = 'a' and y = 1; update foo set z =
2 where x = 'a' and y = 2 if t = 2; apply batch;
[applied]
---
True
(both updates succeeded because the check on t succeeded)
select * from foo;
x | y | t | z
---+---+---+---
a | 1 | 2 | 1
a | 2 | 2 | 2
Hope this helps.
-Tupshin
On Fri, Feb 21, 2014 at 6:05 PM, DuyHai Doan doanduy...@gmail.com wrote:
Hello Clint
The Resolution status of the JIRA is set to Later, probably the
implementation is not done yet. The JIRA was opened to discuss about impl
strategy but nothing has been coded so far I guess.
On Sat, Feb 22, 2014 at 12:02 AM, Clint Kelly clint.ke...@gmail.com
wrote:
Folks,
Does anyone know how I can modify multiple rows at once in a
lightweight transaction in CQL3?
I saw the following ticket:
https://issues.apache.org/jira/browse/CASSANDRA-5633
but it was not obvious to me from the comments how (or whether) this
got resolved. I also couldn't find anything in the DataStax
documentation about how to perform these operations.
I'm in particular interested in how to perform a compare-and-set
operation that modifies multiple rows (with the same partition key)
using the DataStax Java driver.
Thanks!
Best regards,
Clint
Update multiple rows in a CQL lightweight transaction
Folks, Does anyone know how I can modify multiple rows at once in a lightweight transaction in CQL3? I saw the following ticket: https://issues.apache.org/jira/browse/CASSANDRA-5633 but it was not obvious to me from the comments how (or whether) this got resolved. I also couldn't find anything in the DataStax documentation about how to perform these operations. I'm in particular interested in how to perform a compare-and-set operation that modifies multiple rows (with the same partition key) using the DataStax Java driver. Thanks! Best regards, Clint
Buffering for lots of INSERT or UPDATE calls with DataStax Java driver?
Folks, Is there a recommended way to perform lots of INSERT operations in a row when using the DataStax Java driver? I notice that the RecordWriter for the CQL3 Hadoop implementation in Cassandra does some per-data-node buffering of CQL3 queries. The DataStax Java driver, on the other hand, supports asynchronous query execution. In a situation in which we are doing lots of and lots of writes (like a RecordWriter for Hadoop), do we need to do any buffering or can we just fire away session.executeAsync(...) calls as soon as they are ready? Thanks! Best regards, Clint
Re: Buffering for lots of INSERT or UPDATE calls with DataStax Java driver?
Ah yes, thanks! I had forgotten about the use of batches for this purpose. Appreciate the help, cheers! Best regards, Clint On Sat, Feb 8, 2014 at 1:24 PM, DuyHai Doan doanduy...@gmail.com wrote: Is there a recommended way to perform lots of INSERT operations in a row when using the DataStax Java driver? -- Yes, use UNLOGGED batches. More info here: http://www.datastax.com/documentation/cql/3.0/webhelp/index.html#cql/cql_reference/batch_r.html On Sat, Feb 8, 2014 at 10:19 PM, Clint Kelly clint.ke...@gmail.com wrote: Folks, Is there a recommended way to perform lots of INSERT operations in a row when using the DataStax Java driver? I notice that the RecordWriter for the CQL3 Hadoop implementation in Cassandra does some per-data-node buffering of CQL3 queries. The DataStax Java driver, on the other hand, supports asynchronous query execution. In a situation in which we are doing lots of and lots of writes (like a RecordWriter for Hadoop), do we need to do any buffering or can we just fire away session.executeAsync(...) calls as soon as they are ready? Thanks! Best regards, Clint
CQL3 delete using or ?
Folks,
Is there any way to perform a delete in CQL of all rows where a
particular columns (that is part of the primary key) is less than a
certain value? I believe that the corresponding SELECT statement
works, as in this example:
cqlsh:fiddle describe table foo;
CREATE TABLE foo (
key text,
fam text,
qual text,
version int,
val text,
PRIMARY KEY (key, fam, qual, version)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
cqlsh:fiddle select * from foo;
key | fam | qual | version | val
---+--+--+-+
Clint | info | hair | 5 | brown
Clint | info | hair | 10 | blonde
(2 rows)
cqlsh:fiddle select val from foo where key='Clint' and fam='info' and
qual='hair' and version = 5;
val
---
brown
(1 rows)
cqlsh:fiddle delete val from foo where key='Clint' and fam='info' and
qual='hair' and version = 5;
Bad Request: Invalid operator LTE for PRIMARY KEY part version
I assume the only way I can implement the functionality I'm looking
for is to do a select, get a list of all of the elements that I want
to delete, and then delete them in a big batch statement. Is that
correct?
Thanks!
Best regards,
Clint
Re: Cassandra 2.0 with Hadoop 2.x?
Thunder, Cyril, thanks for your responses. FWIW I started working on writing a set of InputFormat and OutputFormat classes that use the DataStax Java driver instead of the thrift client. That seems to be going pretty well. I am not sure if I can eliminate all of the thrift code, but at least the RecordReader was pretty easy to implement using only the DataStax driver. I can post a link if anyone is curious (once it is done). The DataStax driver has automatic paging and a few other nice features that made the code more concise. Best regards, Clint On Tue, Feb 4, 2014 at 3:36 PM, Thunder Stumpges thunder.stump...@gmail.com wrote: Hello Clint, Yes I was able to get it working after a bit of work. I have pushed the branch with the fix (which is currently quite a ways behind latest). You can compare to yours I suppose. Let me know if you have any questions. https://github.com/VerticalSearchWorks/cassandra/tree/Cassandra2-CDH4 regards, Thunder On Tue, Feb 4, 2014 at 1:40 PM, Cyril Scetbon cyril.scet...@free.fr wrote: Hi, Look for posts from Thunder Stumpges in this mailing list. I know he has succeeded to make it Hadoop 2.x work with Cassandra 2.x For those who are interested in using it with Cassandra 1.2.13 you can use the patch https://github.com/cscetbon/cassandra/commit/88d694362d8d6bc09b3eeceb6baad7b3cc068ad3.patch It uses Cloudera CDH4 repository for Hadoop Classes but you can use others. Regards -- Cyril SCETBON On 03 Feb 2014, at 19:10, Clint Kelly clint.ke...@gmail.com wrote: Folks, Has anyone out there used Cassandra 2.0 with Hadoop 2.x? I saw this discussion on the Cassandra JIRA: https://issues.apache.org/jira/browse/CASSANDRA-5201 but the fix referenced (https://github.com/michaelsembwever/cassandra-hadoop) is for Cassandra 1.2. I put together a similar patch for Cassandra 2.0 for anyone who is interested: https://github.com/wibiclint/cassandra2-hadoop2 but I'm wondering if there is a more official solution to this problem. Any help would be appreciated. Thanks! Best regards, Clint
Cassandra 2.0 with Hadoop 2.x?
Folks, Has anyone out there used Cassandra 2.0 with Hadoop 2.x? I saw this discussion on the Cassandra JIRA: https://issues.apache.org/jira/browse/CASSANDRA-5201 but the fix referenced (https://github.com/michaelsembwever/cassandra-hadoop) is for Cassandra 1.2. I put together a similar patch for Cassandra 2.0 for anyone who is interested: https://github.com/wibiclint/cassandra2-hadoop2 but I'm wondering if there is a more official solution to this problem. Any help would be appreciated. Thanks! Best regards, Clint
