Re: Cassandra backup via snapshots in production
True Delete in CQL just create tombstone so from the storage engine pov it's just adding some physical columns Truncate does trigger snapshot creation though Le 21 nov. 2014 19:29, Robert Coli rc...@eventbrite.com a écrit : On Fri, Nov 21, 2014 at 8:40 AM, Jens Rantil jens.ran...@tink.se wrote: The main purpose is to protect us from human errors (eg. unexpected manipulations: delete, drop tables, …). If that is the main purpose, having auto_snapshot: true” in cassandra.yaml will be enough to protect you. OP includes delete in their list of unexpected manipulations, and auto_snapshot: true will not protect you in any way from DELETE. =Rob http://twitter.com/rcolidba
Re: Compaction Strategy guidance
Hi Andrei, Hi Nicolai, Which version of C* are you using ? There are some recommendations about the max storage per node : http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to handle 10x (3-5TB). I have the feeling that those recommendations are sensitive according many criteria such as : - your hardware - the compaction strategy - ... It looks that LCS lower those limitations. Increasing the size of sstables might help if you have enough CPU and you can put more load on your I/O system (@Andrei, I am interested by the results of your experimentation about large sstable files) From my point of view, there are some usage patterns where it is better to have many small servers than a few large servers. Probably, it is better to have many small servers if you need LCS for large tables. Just my 2 cents. Jean-Armel 2014-11-24 19:56 GMT+01:00 Robert Coli rc...@eventbrite.com: On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev ngrigor...@gmail.com wrote: One of the obvious recommendations I have received was to run more than one instance of C* per host. Makes sense - it will reduce the amount of data per node and will make better use of the resources. This is usually a Bad Idea to do in production. =Rob
Re: Cassandra backup via snapshots in production
Truncate does trigger snapshot creation though Doesn’t it? With “auto_snapshot: true” it should. ——— Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter On Tue, Nov 25, 2014 at 9:21 AM, DuyHai Doan doanduy...@gmail.com wrote: True Delete in CQL just create tombstone so from the storage engine pov it's just adding some physical columns Truncate does trigger snapshot creation though Le 21 nov. 2014 19:29, Robert Coli rc...@eventbrite.com a écrit : On Fri, Nov 21, 2014 at 8:40 AM, Jens Rantil jens.ran...@tink.se wrote: The main purpose is to protect us from human errors (eg. unexpected manipulations: delete, drop tables, …). If that is the main purpose, having auto_snapshot: true” in cassandra.yaml will be enough to protect you. OP includes delete in their list of unexpected manipulations, and auto_snapshot: true will not protect you in any way from DELETE. =Rob http://twitter.com/rcolidba
Fwd: Issues in moving data from cassandra to elasticsearch in java.
Hi, I am working on a java plugin which moves data from cassandra to elasticsearch. This plugin must run in the server for every 5 seconds. The data is getting moved, but the issue is that every time the plugin runs(ie after every 5 seconds) all the data, including data which has been already moved into elasticsearch in the previous iteration is moving to it. So we are having duplicate values in the elastic search. How to avoid this problem. We are using this plugin to manage logs which are generated during any online transaction. So we will be having millions of transactions. Following is the table schema. CREATE TABLE logs ( txn_id text, logged_at timestamp, des text, key_name text, params text, PRIMARY KEY (txn_id, logged_at) ) The txn_id is a 16 digit number and is not unique. It is a combination of 6 random numbers generated using a random function, followed by the epoch timestamp in millisec(10 digits). I want to move only the data which has been generated after the previous transaction and not the data which was already moved in the previous transaction. I tried to do it with static values, counter variables, comparing the write_time of each row and order by. Still its not working . Please suggest me any ideas. Thanks and regards vinod joseph 8050136948
Cassandra schema migrator
Hi, Anyone who is using, or could recommend, a tool for versioning schemas/migrating in Cassandra? My list of requirements is: * Support for adding tables. * Support for versioning of table properties. All our tables are to be defaulted to LeveledCompactionStrategy. * Support for adding non-existing columns. * Optional: Support for removing columns. * Optional: Support for removing tables. We are preferably a Java shop, but could potentially integrate something non-Java. I understand I could write a tool that would make these decisions using system.schema_columnfamilies and system.schema_columns, but as always reusing a proven tool would be preferable. So far I only know of Spring Data Cassandra that handles creating tables and adding columns. However, it does not handle table properties in any way. Thanks, Jens ——— Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter
Re: Compaction Strategy guidance
Hi Jean-Armel, Nikolai, 1. Increasing sstable size doesn't work (well, I think, unless we overscale - add more nodes than really necessary, which is prohibitive for us in a way). Essentially there is no change. I gave up and will go for STCS;-( 2. We use 2.0.11 as of now 3. We are running on EC2 c3.8xlarge instances with EBS volumes for data (GP SSD) Jean-Armel, I believe that what you say about many small instances is absolutely true. But, is not good in our case - we write a lot and almost never read what we've written. That is, we want to be able to read everything, but in reality we hardly read 1%, I think. This implies that smaller instances are of no use in terms of read performance for us. And generally nstances/cpu/ram is more expensive than storage. So, we really would like to have instances with large storage. Andrei. On Tue, Nov 25, 2014 at 11:23 AM, Jean-Armel Luce jaluc...@gmail.com wrote: Hi Andrei, Hi Nicolai, Which version of C* are you using ? There are some recommendations about the max storage per node : http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to handle 10x (3-5TB). I have the feeling that those recommendations are sensitive according many criteria such as : - your hardware - the compaction strategy - ... It looks that LCS lower those limitations. Increasing the size of sstables might help if you have enough CPU and you can put more load on your I/O system (@Andrei, I am interested by the results of your experimentation about large sstable files) From my point of view, there are some usage patterns where it is better to have many small servers than a few large servers. Probably, it is better to have many small servers if you need LCS for large tables. Just my 2 cents. Jean-Armel 2014-11-24 19:56 GMT+01:00 Robert Coli rc...@eventbrite.com: On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev ngrigor...@gmail.com wrote: One of the obvious recommendations I have received was to run more than one instance of C* per host. Makes sense - it will reduce the amount of data per node and will make better use of the resources. This is usually a Bad Idea to do in production. =Rob
Re: Compaction Strategy guidance
If you are that write-heavy you should definitely go with STCS, LCS optimizes for reads by doing more compactions /Marcus On Tue, Nov 25, 2014 at 11:22 AM, Andrei Ivanov aiva...@iponweb.net wrote: Hi Jean-Armel, Nikolai, 1. Increasing sstable size doesn't work (well, I think, unless we overscale - add more nodes than really necessary, which is prohibitive for us in a way). Essentially there is no change. I gave up and will go for STCS;-( 2. We use 2.0.11 as of now 3. We are running on EC2 c3.8xlarge instances with EBS volumes for data (GP SSD) Jean-Armel, I believe that what you say about many small instances is absolutely true. But, is not good in our case - we write a lot and almost never read what we've written. That is, we want to be able to read everything, but in reality we hardly read 1%, I think. This implies that smaller instances are of no use in terms of read performance for us. And generally nstances/cpu/ram is more expensive than storage. So, we really would like to have instances with large storage. Andrei. On Tue, Nov 25, 2014 at 11:23 AM, Jean-Armel Luce jaluc...@gmail.com wrote: Hi Andrei, Hi Nicolai, Which version of C* are you using ? There are some recommendations about the max storage per node : http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to handle 10x (3-5TB). I have the feeling that those recommendations are sensitive according many criteria such as : - your hardware - the compaction strategy - ... It looks that LCS lower those limitations. Increasing the size of sstables might help if you have enough CPU and you can put more load on your I/O system (@Andrei, I am interested by the results of your experimentation about large sstable files) From my point of view, there are some usage patterns where it is better to have many small servers than a few large servers. Probably, it is better to have many small servers if you need LCS for large tables. Just my 2 cents. Jean-Armel 2014-11-24 19:56 GMT+01:00 Robert Coli rc...@eventbrite.com: On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev ngrigor...@gmail.com wrote: One of the obvious recommendations I have received was to run more than one instance of C* per host. Makes sense - it will reduce the amount of data per node and will make better use of the resources. This is usually a Bad Idea to do in production. =Rob
Re: Cassandra schema migrator
Hi Jens, maybe you should have a look at mutagen for cassandra: https://github.com/toddfast/mutagen-cassandra It is a litte quiet around this for some months, but maybe still worth it. Cheers, Jan Am 25.11.2014 um 10:22 schrieb Jens Rantil: Hi, Anyone who is using, or could recommend, a tool for versioning schemas/migrating in Cassandra? My list of requirements is: * Support for adding tables. * Support for versioning of table properties. All our tables are to be defaulted to LeveledCompactionStrategy. * Support for adding non-existing columns. * Optional: Support for removing columns. * Optional: Support for removing tables. We are preferably a Java shop, but could potentially integrate something non-Java. I understand I could write a tool that would make these decisions using system.schema_columnfamilies and system.schema_columns, but as always reusing a proven tool would be preferable. So far I only know of Spring Data Cassandra that handles creating tables and adding columns. However, it does not handle table properties in any way. Thanks, Jens ——— Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter
Re: Compaction Strategy guidance
Yep, Marcus, I know. It's mainly a question of cost of those extra x2 disks, you know. Our final setup will be more like 30TB, so doubling it is still some cost. But i guess, we will have to live with it On Tue, Nov 25, 2014 at 1:26 PM, Marcus Eriksson krum...@gmail.com wrote: If you are that write-heavy you should definitely go with STCS, LCS optimizes for reads by doing more compactions /Marcus On Tue, Nov 25, 2014 at 11:22 AM, Andrei Ivanov aiva...@iponweb.net wrote: Hi Jean-Armel, Nikolai, 1. Increasing sstable size doesn't work (well, I think, unless we overscale - add more nodes than really necessary, which is prohibitive for us in a way). Essentially there is no change. I gave up and will go for STCS;-( 2. We use 2.0.11 as of now 3. We are running on EC2 c3.8xlarge instances with EBS volumes for data (GP SSD) Jean-Armel, I believe that what you say about many small instances is absolutely true. But, is not good in our case - we write a lot and almost never read what we've written. That is, we want to be able to read everything, but in reality we hardly read 1%, I think. This implies that smaller instances are of no use in terms of read performance for us. And generally nstances/cpu/ram is more expensive than storage. So, we really would like to have instances with large storage. Andrei. On Tue, Nov 25, 2014 at 11:23 AM, Jean-Armel Luce jaluc...@gmail.com wrote: Hi Andrei, Hi Nicolai, Which version of C* are you using ? There are some recommendations about the max storage per node : http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to handle 10x (3-5TB). I have the feeling that those recommendations are sensitive according many criteria such as : - your hardware - the compaction strategy - ... It looks that LCS lower those limitations. Increasing the size of sstables might help if you have enough CPU and you can put more load on your I/O system (@Andrei, I am interested by the results of your experimentation about large sstable files) From my point of view, there are some usage patterns where it is better to have many small servers than a few large servers. Probably, it is better to have many small servers if you need LCS for large tables. Just my 2 cents. Jean-Armel 2014-11-24 19:56 GMT+01:00 Robert Coli rc...@eventbrite.com: On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev ngrigor...@gmail.com wrote: One of the obvious recommendations I have received was to run more than one instance of C* per host. Makes sense - it will reduce the amount of data per node and will make better use of the resources. This is usually a Bad Idea to do in production. =Rob
Re: Cassandra schema migrator
On 25.11.2014 10:22, Jens Rantil wrote: Anyone who is using, or could recommend, a tool for versioning schemas/migrating in Cassandra? I've recently written a tool to solve schema migration at our company which may be useful: https://github.com/advancedtelematic/cql-migrate My list of requirements is: * Support for adding tables. Yep, this works. * Support for versioning of table properties. All our tables are to be defaulted to LeveledCompactionStrategy. My parser/statement splitter doesn't currently recognise the WITH ... syntax on CREATE tables and ALTER TABLE, but it could easily be added. Do you need to modify the properties after the table has been created? If so the changes would need to be expressed as ALTER TABLE statements. * Support for adding non-existing columns. Supported. cql-migrate works by getting the user to express the upgrade as an ALTER TABLE ... ADD column statement. * Optional: Support for removing columns. * Optional: Support for removing tables. Not supported, although it wouldn't be hard to add. We are preferably a Java shop, but could potentially integrate something non-Java. I understand I could write a tool that would make these decisions using system.schema_columnfamilies and system.schema_columns, but as always reusing a proven tool would be preferable. It is implemented in python, and can be built into a Debian package for deployment. Rather than try and fully parse the CQL, cql-migrate works by splitting a .CQL script into individual statements, executing them in turn and ignoring errors that relate to tables etc already exiting. Doing it that way means there is very little chance that the schema created will differ from what running the same script through cqlsh would have produced. Let me know if you need a hand getting going with it. Cheers, Phil
Re: Getting the counters with the highest values
We have too many documents per day to materialize in memory, so querying per day and aggregating the results isn’t really possible. You don't really need to, that's part of the point. You can paginate across a partition with most client drivers, and materializing this view is just copying data from one table to another with a different layout. So you end up just having to read then write a few thousand records at a shot. doc_id as the partitioning key and day as the clustering key means that you have to iterate over documents from some outside knowledge (a definitive source on what the set of doc_id's is), and reading so many separate partitions (one per doc_id) will produce memory pressure in your cluster. Compared against ((day), doc_id) where you can SELECT * WHERE day=?. Your approach would give you a very nice time series view of views per document over time, which itself might be useful elsewhere in your application. That said, there are physical and practical limits on the number of columns you can have in a partition (2 billion physical, and depending on the data sometimes just on the order of a few hundred thousand practical without causing some troubles in areas such as compaction, repair, and bootstrap). However, I still suspect you may benefit by keying the counters table primarily by date, but maybe add another key rotator in there, like ((day, subpartition), doc_id). Compute your sub partition deterministically but in an evenly distributed manner from your doc_id (eg, doc_id mod 16, or md5(doc_id).take(2), etc., depending on what the data type is for your doc_id). This will split your single logical partition up across n physical partitions, opens you up to parallel processing of those partitions (materializing can get a lot faster as you can very easily have a single worker working on each physical partition without reduced hotspotting). Each worker can be assigned a day and subpartition and work exclusively on that data set for materialization (or a single worker can iterate the subpartitions for the same effect). Now to make a too-long email even longer, if you're approaching practical limits on using doc_id as part of a clustering key, your materialized view is going to have similar issues. So you may have to either only materialize documents with a view count over a certain threshold, or engage in a similar sub partitioning scheme there. On Mon Nov 24 2014 at 10:33:50 AM Robert Wille rwi...@fold3.com wrote: We do get a large number of documents getting counts each day, which is why I’m thinking the running totals table be ((doc_id), day) rather than ((day), doc_id). We have too many documents per day to materialize in memory, so querying per day and aggregating the results isn’t really possible. I’m planning on bucketing the materialized ordering because we get enough unique document views per day that the rows will be quite large. Not so large as to be unmanageable, but pushing the limits. If we were so lucky as to get a significant increase in traffic, I might start having issues. I didn’t include bucketing in my post because I didn’t want to complicate my question. I hadn’t considered that I could bucket by hour and then use a local midnight instead of a global midnight. Interesting idea. Thanks for your response. Robert On Nov 24, 2014, at 9:40 AM, Eric Stevens migh...@gmail.com wrote: You're right that there's no way to use the counter data type to materialize a view ordered by the counter. Computing this post hoc is the way to go if your needs allow for it (if not, something like Summingbird or vanilla Storm may be necessary). I might suggest that you make your primary key for your running totals by day table be ((day), doc_id) because it will make it easy to compute the materialized ordered view (SELECT doc_id, count FROM running_totals WHERE day=?) unless you expect to have a really large number of documents getting counts each day. For your materialized ordering, I'd suggest a primary key of ((day), count) as then for a given day you'll be able to select top by count (SELECT count, doc_id FROM doc_counts WHERE day=? ORDER BY count DESC). One more thing to consider if your users are not all in a single timezone is having your time bucket be hour instead of day so that you can answer by day goal posted by local midnight (except the handful of locations that use half hour timezone offsets) instead of a single global midnight. You can then either include either just each hour in each row (and aggregate at read time), or you can make each row a rolling 24 hours (aggregating at write time), depending on which use case fits your needs better. On Sun Nov 23 2014 at 8:42:11 AM Robert Wille rwi...@fold3.com wrote: I’m working on moving a bunch of counters out of our relational database to Cassandra. For the most part, Cassandra is a very nice fit, except for one feature on our website. We manage a time series of
Re: Cassandra version 1.0.10 Data Loss upon restart
Rob, The JIRA https://issues.apache.org/jira/browse/CASSANDRA-4446 refers to the problem after we've invoked drain. However, we did not invoke drain or flush. We are running one node cassandra within one data center and it is being replicated with another node in another data center. We are using the thrift API in java application to retrieve and modify the data from the primary node only. We are not using the other node in the other data center for any operations. The data loss well exceeds the commit log sync period. Thanks, Ankit On Mon, Nov 24, 2014 at 8:53 PM, Robert Coli rc...@eventbrite.com wrote: On Mon, Nov 24, 2014 at 5:51 PM, Robert Coli rc...@eventbrite.com wrote: What is your replication factor? What CL are you using to read? Ah, I see from OP that RF is 1. As a general statement, RF=1 is an edge case which very, very few people have ever operated in production. It is relatively likely that there are some undiscovered edge cases which relate to it. That said, this would be a particularly glaring one, which I would expect to be discovered in other contexts. =Rob
Re: Compaction Strategy guidance
Hi Jean-Armel, I am using latest and greatest DSE 4.5.2 (4.5.3 in another cluster but there are no relevant changes between 4.5.2 and 4.5.3) - thus, Cassandra 2.0.10. I have about 1,8Tb of data per node now in total, which falls into that range. As I said, it is really a problem with large amount of data in a single CF, not total amount of data. Quite often the nodes are idle yet having quite a bit of pending compactions. I have discussed it with other members of C* community and DataStax guys and, they have confirmed my observation. I believe that increasing the sstable size won't help at all and probably will make the things worse - everything else being equal, of course. But I would like to hear from Andrei when he is done with his test. Regarding the last statement - yes, C* clearly likes many small servers more than fewer large ones. But it is all relative - and can be all recalculated to $$$ :) C* is all about partitioning of everything - storage, traffic...Less data per node and more nodes give you lower latency, lower heap usage etc, etc. I think I have learned this with my project. Somewhat hard way but still, nothing is better than the personal experience :) On Tue, Nov 25, 2014 at 3:23 AM, Jean-Armel Luce jaluc...@gmail.com wrote: Hi Andrei, Hi Nicolai, Which version of C* are you using ? There are some recommendations about the max storage per node : http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to handle 10x (3-5TB). I have the feeling that those recommendations are sensitive according many criteria such as : - your hardware - the compaction strategy - ... It looks that LCS lower those limitations. Increasing the size of sstables might help if you have enough CPU and you can put more load on your I/O system (@Andrei, I am interested by the results of your experimentation about large sstable files) From my point of view, there are some usage patterns where it is better to have many small servers than a few large servers. Probably, it is better to have many small servers if you need LCS for large tables. Just my 2 cents. Jean-Armel 2014-11-24 19:56 GMT+01:00 Robert Coli rc...@eventbrite.com: On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev ngrigor...@gmail.com wrote: One of the obvious recommendations I have received was to run more than one instance of C* per host. Makes sense - it will reduce the amount of data per node and will make better use of the resources. This is usually a Bad Idea to do in production. =Rob -- Nikolai Grigoriev (514) 772-5178
Data synchronization between 2 running clusters on different availability zone
Hello! I have the following scenario: 1. For ensuring high availability I would like to install one Cassandra cluster on one availability zone (on Amazon EC2 US-east) and one Cassandra cluster on other AZ (Amazon EC2 US-west). 2.I have pipeline that is running on Amazon EC2-EAST and is feeding the Cassandra installed on this AZ. Here are my questions: 1. Is this scenario feasible? 2. Is the architecture correct regarding the availability of Cassandra? 3. If the architecture is fine, how do you keep data synchronized between the two instances? I look forward for your answers. Regards, Florin
Re: Issues in moving data from cassandra to elasticsearch in java.
Consider adding log_bucket timestamp, and then indexing that column. Your data loader can SELECT * FROM logs WHERE log_bucket = ?. The value you supply there would be the timestamp log bucket you're processing - in your case logged_at % 5. However, I'll caution against writing data to Cassandra and then trying to reliably read it back immediately after. You're likely to miss values this way due to eventual consistency unless you read at CL_ALL. But then your data loader will break whenever you have any node offline. Writing then immediately reading data is a typical antipattern in any eventually consistent system. If using DataStax Java Driver you can use CL_ALL with DowngradingConsistencyRetryPolicy and you would at least strike a nice balance between reasonably strong consistency and loss of resiliency from CL_ALL (but when you have a node offline, your load process may get significantly slower). This would mitigate but not eliminate the antipattern. On Tue Nov 25 2014 at 2:11:36 AM Vinod Joseph vinodjosep...@gmail.com wrote: Hi, I am working on a java plugin which moves data from cassandra to elasticsearch. This plugin must run in the server for every 5 seconds. The data is getting moved, but the issue is that every time the plugin runs(ie after every 5 seconds) all the data, including data which has been already moved into elasticsearch in the previous iteration is moving to it. So we are having duplicate values in the elastic search. How to avoid this problem. We are using this plugin to manage logs which are generated during any online transaction. So we will be having millions of transactions. Following is the table schema. CREATE TABLE logs ( txn_id text, logged_at timestamp, des text, key_name text, params text, PRIMARY KEY (txn_id, logged_at) ) The txn_id is a 16 digit number and is not unique. It is a combination of 6 random numbers generated using a random function, followed by the epoch timestamp in millisec(10 digits). I want to move only the data which has been generated after the previous transaction and not the data which was already moved in the previous transaction. I tried to do it with static values, counter variables, comparing the write_time of each row and order by. Still its not working . Please suggest me any ideas. Thanks and regards vinod joseph 8050136948
Re: Rule of thumb for concurrent asynchronous queries?
Great question. The safe answer is to do a proof of concept implementation and try various rates to determine where the bottleneck is. It will also depend on the row size. Hard to say if you will be limited by the cluster load or network bandwidth. Is there only one client talking to your cluster? Or are you asking what each of, say, one million clients can be simultaneously requesting? The rate of requests will matter as well, particularly if the cluster has a non-trivial load. My ultimate rule of thumb is simple: Moderation. Not too many threads, not too frequent request rate. It would be nice if we had a way to calculate this number (both numbers) for you so that a client (driver) could ping for it from the cluster, as well as for the cluster to return a suggested wait interval before sending another request based on actual load. -- Jack Krupansky -Original Message- From: Robert Wille Sent: Tuesday, November 25, 2014 10:57 AM To: user@cassandra.apache.org Subject: Rule of thumb for concurrent asynchronous queries? Suppose I have the primary keys for 10,000 rows and I want them all. Is there a rule of thumb for the maximum number of concurrent asynchronous queries I should execute?=
Re: Rule of thumb for concurrent asynchronous queries?
I think it all depends on how many machines will be involved in the query (read consistency is also a factor) and how long is a typical response in bytes. Large responses will put more pressure on the GC, which will result in more time spent in GC and possibly long(er) GC pauses. Cassandra can tolerate many things - but at the cost for other queries and all the way up to the heal of the individual node. From the original question it is not clear if all these rows are coming from the same or few nodes (token range) or these are really 10K primary keys - so they are spread more or less evenly across the cluster. Also the node disk I/O may be a concern - especially if the data is not in OS cache (or row cache if applicable). I think it is a tough question to get a precise answer. If I had such a problem I would try to determine the peak speed I can achieve first. I.e. find the limiting factor (CPU or disk I/O most likely), then shoot as many requests in as many threads as practical for the client app. Measure the load to prove that you've determined the limiting factor correctly (either CPU or I/O, I doubt it will be network). Then measure the latency and decide what kind of latency you can tolerate for your use case. And then go down from that peak load you've created by certain factor (i.e. limit yourself to XX% of the peak load you have achieved). On Tue, Nov 25, 2014 at 11:34 AM, Jack Krupansky j...@basetechnology.com wrote: Great question. The safe answer is to do a proof of concept implementation and try various rates to determine where the bottleneck is. It will also depend on the row size. Hard to say if you will be limited by the cluster load or network bandwidth. Is there only one client talking to your cluster? Or are you asking what each of, say, one million clients can be simultaneously requesting? The rate of requests will matter as well, particularly if the cluster has a non-trivial load. My ultimate rule of thumb is simple: Moderation. Not too many threads, not too frequent request rate. It would be nice if we had a way to calculate this number (both numbers) for you so that a client (driver) could ping for it from the cluster, as well as for the cluster to return a suggested wait interval before sending another request based on actual load. -- Jack Krupansky -Original Message- From: Robert Wille Sent: Tuesday, November 25, 2014 10:57 AM To: user@cassandra.apache.org Subject: Rule of thumb for concurrent asynchronous queries? Suppose I have the primary keys for 10,000 rows and I want them all. Is there a rule of thumb for the maximum number of concurrent asynchronous queries I should execute?= -- Nikolai Grigoriev (514) 772-5178
Keyspace and table/cf limits
What's the latest on the maximum number of keyspaces and/or tables that one can have in Cassandra 2.1.x? -Raj
Re: max ttl for column
Mark / Philip - Thanks a lot. This is really helpful.
BTW It was my bad, i was mistaken that ttl was in miliseconds rather than
seconds..
Regards
Rajanish GJ
apigee | rajan...@apigee.com
On Fri, Nov 21, 2014 at 9:42 AM, Philip Thompson
philip.thomp...@datastax.com wrote:
With the newest versions of Cassandra, cql is not hanging, but returns the
same Invalid Query Exception you are seeing through hector. I would assume
from the exception that 63072 is in fact that largest TTL you can use.
What are you doing that you need to set a TTL of approximately 30 years?
On Fri, Nov 21, 2014 at 11:29 AM, Rajanish GJ rajan...@apigee.com wrote:
Does hector or cassandra imposes a limit on max ttl value for a column?
I am trying to insert record into one of the column family and seeing the
following error..
Cassandra version : 1.1.12
Hector : 1.1-4
Any pointers appreciated.
me.prettyprint.hector.api.exceptions.HInvalidRequestException:
InvalidRequestException(why:ttl is too large. requested (951027277) maximum
(63072))
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:52)
~[hector-core-1.1-4.jar:na]
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:260)
~[hector-core-1.1-4.jar:na]
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:113)
~[hector-core-1.1-4.jar:na]
at
me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
~[hector-core-1.1-4.jar:na]
at
me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeBatch(AbstractColumnFamilyTemplate.java:115)
~[hector-core-1.1-4.jar:na]
at
me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeIfNotBatched(AbstractColumnFamilyTemplate.java:163)
~[hector-core-1.1-4.jar:na]
at
me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.update(ColumnFamilyTemplate.java:69)
~[hector-core-1.1-4.jar:na]
=
*Also tried using cql, and it seems to hangs and not responding.. trying
again with few combinations*
*INSERT INTO users (key,id) values ('test6','951027277 secs') USING
TTL 951027277 ; *
Regards
Rajanish GJ
apigee | rajan...@apigee.com
Re: Compaction Strategy guidance
Nikolai, Just in case you've missed my comment in the thread (guess you have) - increasing sstable size does nothing (in our case at least). That is, it's not worse but the load pattern is still the same - doing nothing most of the time. So, I switched to STCS and we will have to live with extra storage cost - storage is way cheaper than cpu etc anyhow:-) On Tue, Nov 25, 2014 at 5:53 PM, Nikolai Grigoriev ngrigor...@gmail.com wrote: Hi Jean-Armel, I am using latest and greatest DSE 4.5.2 (4.5.3 in another cluster but there are no relevant changes between 4.5.2 and 4.5.3) - thus, Cassandra 2.0.10. I have about 1,8Tb of data per node now in total, which falls into that range. As I said, it is really a problem with large amount of data in a single CF, not total amount of data. Quite often the nodes are idle yet having quite a bit of pending compactions. I have discussed it with other members of C* community and DataStax guys and, they have confirmed my observation. I believe that increasing the sstable size won't help at all and probably will make the things worse - everything else being equal, of course. But I would like to hear from Andrei when he is done with his test. Regarding the last statement - yes, C* clearly likes many small servers more than fewer large ones. But it is all relative - and can be all recalculated to $$$ :) C* is all about partitioning of everything - storage, traffic...Less data per node and more nodes give you lower latency, lower heap usage etc, etc. I think I have learned this with my project. Somewhat hard way but still, nothing is better than the personal experience :) On Tue, Nov 25, 2014 at 3:23 AM, Jean-Armel Luce jaluc...@gmail.com wrote: Hi Andrei, Hi Nicolai, Which version of C* are you using ? There are some recommendations about the max storage per node : http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to handle 10x (3-5TB). I have the feeling that those recommendations are sensitive according many criteria such as : - your hardware - the compaction strategy - ... It looks that LCS lower those limitations. Increasing the size of sstables might help if you have enough CPU and you can put more load on your I/O system (@Andrei, I am interested by the results of your experimentation about large sstable files) From my point of view, there are some usage patterns where it is better to have many small servers than a few large servers. Probably, it is better to have many small servers if you need LCS for large tables. Just my 2 cents. Jean-Armel 2014-11-24 19:56 GMT+01:00 Robert Coli rc...@eventbrite.com: On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev ngrigor...@gmail.com wrote: One of the obvious recommendations I have received was to run more than one instance of C* per host. Makes sense - it will reduce the amount of data per node and will make better use of the resources. This is usually a Bad Idea to do in production. =Rob -- Nikolai Grigoriev (514) 772-5178
Re: large range read in Cassandra
Thanks Rob. To be clear, I expect this range query to take a long time and perform relatively heavy I/O. What I expected Cassandra to do was use auto-paging ( https://issues.apache.org/jira/browse/CASSANDRA-4415, http://stackoverflow.com/questions/17664438/iterating-through-cassandra-wide-row-with-cql3) so that we aren't literally pulling the entire thing in. Am I misunderstanding this use case? Could you clarify why exactly it would slow way down? It seems like with each read it should be doing a simple range read from one or two sstables. If this won't work then it may me we need to start using Hive/Spark/Pig etc. sooner, or page it manually using LIMIT and WHERE [the last returned result]. On Mon, Nov 24, 2014 at 5:49 PM, Robert Coli rc...@eventbrite.com wrote: On Mon, Nov 24, 2014 at 4:26 PM, Dan Kinder dkin...@turnitin.com wrote: We have a web crawler project currently based on Cassandra ( https://github.com/iParadigms/walker, written in Go and using the gocql driver), with the following relevant usage pattern: - Big range reads over a CF to grab potentially millions of rows and dispatch new links to crawl If you really mean millions of storage rows, this is just about the worst case for Cassandra. The problem you're having is probably that you shouldn't try to do this in Cassandra. Your timeouts are either from the read actually taking longer than the timeout or from the reads provoking heap pressure and resulting GC. =Rob
Re: Compaction Strategy guidance
Andrei, Oh, yes, I have scanned the top of your previous email but overlooked the last part. I am using SSDs so I prefer to put extra work to keep my system performing and save expensive disk space. So far I've been able to size the system more or less correctly so these LCS limitations do not cause too much troubles. But I do keep the CF sharding option as backup - for me it will be relatively easy to implement it. On Tue, Nov 25, 2014 at 1:25 PM, Andrei Ivanov aiva...@iponweb.net wrote: Nikolai, Just in case you've missed my comment in the thread (guess you have) - increasing sstable size does nothing (in our case at least). That is, it's not worse but the load pattern is still the same - doing nothing most of the time. So, I switched to STCS and we will have to live with extra storage cost - storage is way cheaper than cpu etc anyhow:-) On Tue, Nov 25, 2014 at 5:53 PM, Nikolai Grigoriev ngrigor...@gmail.com wrote: Hi Jean-Armel, I am using latest and greatest DSE 4.5.2 (4.5.3 in another cluster but there are no relevant changes between 4.5.2 and 4.5.3) - thus, Cassandra 2.0.10. I have about 1,8Tb of data per node now in total, which falls into that range. As I said, it is really a problem with large amount of data in a single CF, not total amount of data. Quite often the nodes are idle yet having quite a bit of pending compactions. I have discussed it with other members of C* community and DataStax guys and, they have confirmed my observation. I believe that increasing the sstable size won't help at all and probably will make the things worse - everything else being equal, of course. But I would like to hear from Andrei when he is done with his test. Regarding the last statement - yes, C* clearly likes many small servers more than fewer large ones. But it is all relative - and can be all recalculated to $$$ :) C* is all about partitioning of everything - storage, traffic...Less data per node and more nodes give you lower latency, lower heap usage etc, etc. I think I have learned this with my project. Somewhat hard way but still, nothing is better than the personal experience :) On Tue, Nov 25, 2014 at 3:23 AM, Jean-Armel Luce jaluc...@gmail.com wrote: Hi Andrei, Hi Nicolai, Which version of C* are you using ? There are some recommendations about the max storage per node : http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to handle 10x (3-5TB). I have the feeling that those recommendations are sensitive according many criteria such as : - your hardware - the compaction strategy - ... It looks that LCS lower those limitations. Increasing the size of sstables might help if you have enough CPU and you can put more load on your I/O system (@Andrei, I am interested by the results of your experimentation about large sstable files) From my point of view, there are some usage patterns where it is better to have many small servers than a few large servers. Probably, it is better to have many small servers if you need LCS for large tables. Just my 2 cents. Jean-Armel 2014-11-24 19:56 GMT+01:00 Robert Coli rc...@eventbrite.com: On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev ngrigor...@gmail.com wrote: One of the obvious recommendations I have received was to run more than one instance of C* per host. Makes sense - it will reduce the amount of data per node and will make better use of the resources. This is usually a Bad Idea to do in production. =Rob -- Nikolai Grigoriev (514) 772-5178 -- Nikolai Grigoriev (514) 772-5178
Re: Compaction Strategy guidance
Ah, clear then. SSD usage imposes a different bias in terms of costs;-) On Tue, Nov 25, 2014 at 9:48 PM, Nikolai Grigoriev ngrigor...@gmail.com wrote: Andrei, Oh, yes, I have scanned the top of your previous email but overlooked the last part. I am using SSDs so I prefer to put extra work to keep my system performing and save expensive disk space. So far I've been able to size the system more or less correctly so these LCS limitations do not cause too much troubles. But I do keep the CF sharding option as backup - for me it will be relatively easy to implement it. On Tue, Nov 25, 2014 at 1:25 PM, Andrei Ivanov aiva...@iponweb.net wrote: Nikolai, Just in case you've missed my comment in the thread (guess you have) - increasing sstable size does nothing (in our case at least). That is, it's not worse but the load pattern is still the same - doing nothing most of the time. So, I switched to STCS and we will have to live with extra storage cost - storage is way cheaper than cpu etc anyhow:-) On Tue, Nov 25, 2014 at 5:53 PM, Nikolai Grigoriev ngrigor...@gmail.com wrote: Hi Jean-Armel, I am using latest and greatest DSE 4.5.2 (4.5.3 in another cluster but there are no relevant changes between 4.5.2 and 4.5.3) - thus, Cassandra 2.0.10. I have about 1,8Tb of data per node now in total, which falls into that range. As I said, it is really a problem with large amount of data in a single CF, not total amount of data. Quite often the nodes are idle yet having quite a bit of pending compactions. I have discussed it with other members of C* community and DataStax guys and, they have confirmed my observation. I believe that increasing the sstable size won't help at all and probably will make the things worse - everything else being equal, of course. But I would like to hear from Andrei when he is done with his test. Regarding the last statement - yes, C* clearly likes many small servers more than fewer large ones. But it is all relative - and can be all recalculated to $$$ :) C* is all about partitioning of everything - storage, traffic...Less data per node and more nodes give you lower latency, lower heap usage etc, etc. I think I have learned this with my project. Somewhat hard way but still, nothing is better than the personal experience :) On Tue, Nov 25, 2014 at 3:23 AM, Jean-Armel Luce jaluc...@gmail.com wrote: Hi Andrei, Hi Nicolai, Which version of C* are you using ? There are some recommendations about the max storage per node : http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to handle 10x (3-5TB). I have the feeling that those recommendations are sensitive according many criteria such as : - your hardware - the compaction strategy - ... It looks that LCS lower those limitations. Increasing the size of sstables might help if you have enough CPU and you can put more load on your I/O system (@Andrei, I am interested by the results of your experimentation about large sstable files) From my point of view, there are some usage patterns where it is better to have many small servers than a few large servers. Probably, it is better to have many small servers if you need LCS for large tables. Just my 2 cents. Jean-Armel 2014-11-24 19:56 GMT+01:00 Robert Coli rc...@eventbrite.com: On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev ngrigor...@gmail.com wrote: One of the obvious recommendations I have received was to run more than one instance of C* per host. Makes sense - it will reduce the amount of data per node and will make better use of the resources. This is usually a Bad Idea to do in production. =Rob -- Nikolai Grigoriev (514) 772-5178 -- Nikolai Grigoriev (514) 772-5178
Re: large range read in Cassandra
On Tue, Nov 25, 2014 at 10:45 AM, Dan Kinder dkin...@turnitin.com wrote: To be clear, I expect this range query to take a long time and perform relatively heavy I/O. What I expected Cassandra to do was use auto-paging ( https://issues.apache.org/jira/browse/CASSANDRA-4415, http://stackoverflow.com/questions/17664438/iterating-through-cassandra-wide-row-with-cql3) so that we aren't literally pulling the entire thing in. Am I misunderstanding this use case? Could you clarify why exactly it would slow way down? It seems like with each read it should be doing a simple range read from one or two sstables. If you're paging through a single partition, that's likely to be fine. When you said range reads ... over rows my impression was you were talking about attempting to page through millions of partitions. With that confusion cleared up, the likely explanation for lack of availability in your case is heap pressure/GC time. Look for GCs around that time. Also, if you're using authentication, make sure that your authentication keyspace has a replication factor greater than 1. =Rob
Unsubscribe
Only BlueCat IPAM delivers true network intelligence, a smarter way to manage your network and devices. Read more at www.bluecatnetworks.com/networkintelligence http://www.bluecatnetworks.com/networkintelligence. This e-mail and any attachments are for the sole use of the intended recipient and may contain confidential information.
Re: Cassandra version 1.0.10 Data Loss upon restart
On Tue, Nov 25, 2014 at 6:40 AM, Ankit Patel ankit7...@gmail.com wrote: The JIRA https://issues.apache.org/jira/browse/CASSANDRA-4446 refers to the problem after we've invoked drain. However, we did not invoke drain or flush. We are running one node cassandra within one data center and it is being replicated with another node in another data center. We are using the thrift API in java application to retrieve and modify the data from the primary node only. We are not using the other node in the other data center for any operations. The data loss well exceeds the commit log sync period. Right, what I meant by the era of CASSANDRA-4446 is that around that time there were reports of weird behavior around the commit log. Not your particular weird behavior, but similar enough that I am not *overly* surprised to hear it. Unfortunately no one is likely to want to fix your by now quite old version, especially as a re-write of the commit log occurred in (IIRC) 1.1.x series. I recommend upgrading at least to the HEAD of 1.1.x ASAP. =Rob PS - I understand and sympathize, the above is not a terribly satisfying answer in the face of data loss. :/
Re: Keyspace and table/cf limits
On Tue, Nov 25, 2014 at 9:07 AM, Raj N raj.cassan...@gmail.com wrote: What's the latest on the maximum number of keyspaces and/or tables that one can have in Cassandra 2.1.x? Most relevant changes lately would be : https://issues.apache.org/jira/browse/CASSANDRA-6689 and https://issues.apache.org/jira/browse/CASSANDRA-6694 Which should meaningfully reduce the amount of heap memtables consume. That heap can then be used to support more heap-persistent structures associated with many CFs. I have no idea how to estimate the scale of the improvement. As a general/meta statement, Cassandra is very multi-threaded, and consumes file handles like crazy. How many different query cases do you really want to put on one cluster/node? ;D =Rob
Re: Data synchronization between 2 running clusters on different availability zone
On Tue, Nov 25, 2014 at 7:09 AM, Spico Florin spicoflo...@gmail.com wrote: 1. For ensuring high availability I would like to install one Cassandra cluster on one availability zone (on Amazon EC2 US-east) and one Cassandra cluster on other AZ (Amazon EC2 US-west). One cluster, replication factor of 2, cluster configured with a rack aware snitch is how this is usually done. Well, more accurately, people usually deploy with at least RF=3 and across 3 AZs. A RF of at least 3 is also required to use QUORUM Consistency Level. If you will always operate only out of EC2, you probably want to look into the EC2Snitch. If you plan to ultimately go multi-region, EC2MultiRegionSnitch. If maybe hybrid in the future, GossipingPropertyFileSnitch. http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureSnitchEC2_t.html http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureSnitchEC2MultiRegion_c.html http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureSnitchGossipPF_c.html For some good meta on the internals here : https://issues.apache.org/jira/browse/CASSANDRA-3810 =Rob http://twitter.com/rcolidba
Re: large range read in Cassandra
Thanks, very helpful Rob, I'll watch for that. On Tue, Nov 25, 2014 at 11:45 AM, Robert Coli rc...@eventbrite.com wrote: On Tue, Nov 25, 2014 at 10:45 AM, Dan Kinder dkin...@turnitin.com wrote: To be clear, I expect this range query to take a long time and perform relatively heavy I/O. What I expected Cassandra to do was use auto-paging ( https://issues.apache.org/jira/browse/CASSANDRA-4415, http://stackoverflow.com/questions/17664438/iterating-through-cassandra-wide-row-with-cql3) so that we aren't literally pulling the entire thing in. Am I misunderstanding this use case? Could you clarify why exactly it would slow way down? It seems like with each read it should be doing a simple range read from one or two sstables. If you're paging through a single partition, that's likely to be fine. When you said range reads ... over rows my impression was you were talking about attempting to page through millions of partitions. With that confusion cleared up, the likely explanation for lack of availability in your case is heap pressure/GC time. Look for GCs around that time. Also, if you're using authentication, make sure that your authentication keyspace has a replication factor greater than 1. =Rob -- Dan Kinder Senior Software Engineer Turnitin – www.turnitin.com dkin...@turnitin.com
RAM vs SSD for real world performance?
The new SSDs that we have (as well as Fusion IO) in theory can saturate the gigabit ethernet port. The 4k random read and write IOs they’re doing now can easily add up quick and they’re faster than gigabit and even two gigabit. However, not all of that 4k is actually used. I suspect that on average half is wasted. But the question is how much. Of course YMMV. I’m thinking of getting our servers with a moderate amount of RAM. Say 24GB. Then allocate 8GB to Cassandra, another 8GB to random daemons we run, then another 8GB to page cache. Curious what other people have seen here in practice. Are they getting comparable performance to RAM in practice? Latencies would be higher of course but we’re fine with that. -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: RAM vs SSD for real world performance?
On Tue, Nov 25, 2014 at 5:31 PM, Kevin Burton bur...@spinn3r.com wrote: Curious what other people have seen here in practice. Are they getting comparable performance to RAM in practice? Latencies would be higher of course but we’re fine with that. My understanding is that when one runs Cassandra with SSDs, one replaces the typical i/o bound with a CPU bound. Cassandra also has various internal assumptions that do not make best use of the spare i/o available; SSD+Cassandra has only been deployed at scale for a few years, so this makes sense. =Rob
High cpu usage segfaulting
We are using v2.0.11 and have seen several instances in our 24 node cluster
where the node becomes unresponsive, when we look into it we find that
there is a cassandra process chewing up a lot of CPU. There are no other
indications in logs or anything as to what might be happening, however if
we strace the process that is chewing up CPU we see a segmental fault:
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigreturn(0x7fd61110f862)= 30618997712
futex(0x7fd614844054, FUTEX_WAIT_PRIVATE, 27333, NULL) = -1 EAGAIN
(Resource temporarily unavailable)
futex(0x7fd614844028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fd6148e2e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd6148e2e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7fd6148e2e28, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7fd614844054, FUTEX_WAIT_PRIVATE, 27335, NULL) = 0
futex(0x7fd614844028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fd6148e2e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd6148e2e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7fd6148e2e28, FUTEX_WAKE_PRIVATE, 1) = 1
And this happens over and over again while running strafe.
Has anyone seen this? Does anyone have any ideas what might be happening,
or how we could debug it further?
Thanks for your help,
Stan
Re: RAM vs SSD for real world performance?
I imagine I’d generally be happy if we were CPU bound :-) … as long as the number of transactions per second is generally reasonable. On Tue, Nov 25, 2014 at 7:35 PM, Robert Coli rc...@eventbrite.com wrote: On Tue, Nov 25, 2014 at 5:31 PM, Kevin Burton bur...@spinn3r.com wrote: Curious what other people have seen here in practice. Are they getting comparable performance to RAM in practice? Latencies would be higher of course but we’re fine with that. My understanding is that when one runs Cassandra with SSDs, one replaces the typical i/o bound with a CPU bound. Cassandra also has various internal assumptions that do not make best use of the spare i/o available; SSD+Cassandra has only been deployed at scale for a few years, so this makes sense. =Rob -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: High cpu usage segfaulting
On Tue, Nov 25, 2014 at 8:07 PM, Stan Lemon sle...@salesforce.com wrote: We are using v2.0.11 and have seen several instances in our 24 node cluster where the node becomes unresponsive, when we look into it we find that there is a cassandra process chewing up a lot of CPU. There are no other indications in logs or anything as to what might be happening, however if we strace the process that is chewing up CPU we see a segmental fault: Has anyone seen this? Does anyone have any ideas what might be happening, or how we could debug it further? Does it go away when you restart the node? First, you should do the standard checks for if this is GC pre-fail, which looks like a flattop on heap consumption graphs combined with a spike in GC duration. If you don't find that or OOM log messages, your version is new enough that I would file a JIRA at http://issues.apache.org =Rob
Re: High cpu usage segfaulting
Hi Stan,
Put some monitoring on this. The first thing I think of when I hear
chewing up CPU for Java apps is GC. In SPM http://sematext.com/spm/
you can easily see individual JVM memory pools and see if any of them are
at (close to) 100%. You can typically correlate that to increased GC times
and counts. I'd look at that before looking at strace and such.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr Elasticsearch Support * http://sematext.com/
On Tue, Nov 25, 2014 at 11:07 PM, Stan Lemon sle...@salesforce.com wrote:
We are using v2.0.11 and have seen several instances in our 24 node
cluster where the node becomes unresponsive, when we look into it we find
that there is a cassandra process chewing up a lot of CPU. There are no
other indications in logs or anything as to what might be happening,
however if we strace the process that is chewing up CPU we see a segmental
fault:
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigreturn(0x7fd61110f862)= 30618997712
futex(0x7fd614844054, FUTEX_WAIT_PRIVATE, 27333, NULL) = -1 EAGAIN
(Resource temporarily unavailable)
futex(0x7fd614844028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fd6148e2e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd6148e2e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7fd6148e2e28, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7fd614844054, FUTEX_WAIT_PRIVATE, 27335, NULL) = 0
futex(0x7fd614844028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fd6148e2e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd6148e2e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7fd6148e2e28, FUTEX_WAKE_PRIVATE, 1) = 1
And this happens over and over again while running strafe.
Has anyone seen this? Does anyone have any ideas what might be happening,
or how we could debug it further?
Thanks for your help,
Stan
Partial replication to a DC
Is it possible to replicate a subset of the keyspaces to a data center? For example, if I want to run reports without impacting my production nodes, can I put the relevant column families in a keyspace and create a DC for reporting that replicates only that keyspace? Robert
Re: Partial replication to a DC
On Tue, Nov 25, 2014 at 8:52 PM, Robert Wille rwi...@fold3.com wrote: Is it possible to replicate a subset of the keyspaces to a data center? For example, if I want to run reports without impacting my production nodes, can I put the relevant column families in a keyspace and create a DC for reporting that replicates only that keyspace? Yes, that's one of the things that Datastax DSE does to enable the SOLR special sauce. Be sure to have a version with LOCAL_X CLs, and use them. =Rob http://twitter.com/rcolidba
Re: mysql based columnar DB to Cassandra DB - Migration
Hello Folks, I have one mysql based columnar DB, i want to migrate it to Cassandra. How its possible ? Best Regards Akshay Ballarpure Tata Consultancy Services Cell:- 9985084075 Mailto: akshay.ballarp...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting From: Akshay Ballarpure/HYD/TCS To: user@cassandra.apache.org Date: 11/18/2014 09:00 PM Subject:mysql based columnar DB to Cassandra DB - Migration I have one mysql based columnar DB, i want to migrate it to Cassandra. How its possible ? Best Regards Akshay Ballarpure Tata Consultancy Services Cell:- 9985084075 Mailto: akshay.ballarp...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
RE: mysql based columnar DB to Cassandra DB - Migration
Hi Akshay, this heavily depends on your data model. There is no general way to do it. It includes several steps: 1) Migration of applications using Mysql to Cassandra 2) Migration of the Mysql Database to Cassandra itself Keep in mind that there are no such things like relations or joins in Cassandra. In general Actually Datastax published some documents that could be of interest for you: http://www.datastax.com/2012/03/how-to-move-data-from-relational-databases-to-datastax-enterprise-cassandra-using-sqoop http://www.datastax.com/resources/whitepapers/mysql-to-cassandra Also I found this from EbayTech: http://www.ebaytechblog.com/2012/07/16/cassandra-data-modeling-best-practices-part-1/#.VHV2BlXF_Eg Regards Andi From: Akshay Ballarpure [akshay.ballarp...@tcs.com] Sent: 26 November 2014 07:15 To: user@cassandra.apache.org Subject: Re: mysql based columnar DB to Cassandra DB - Migration Hello Folks, I have one mysql based columnar DB, i want to migrate it to Cassandra. How its possible ? Best Regards Akshay Ballarpure Tata Consultancy Services Cell:- 9985084075 Mailto: akshay.ballarp...@tcs.com Website: http://www.tcs.comhttp://www.tcs.com/ Experience certainty.IT Services Business Solutions Consulting From:Akshay Ballarpure/HYD/TCS To:user@cassandra.apache.org Date:11/18/2014 09:00 PM Subject:mysql based columnar DB to Cassandra DB - Migration I have one mysql based columnar DB, i want to migrate it to Cassandra. How its possible ? Best Regards Akshay Ballarpure Tata Consultancy Services Cell:- 9985084075 Mailto: akshay.ballarp...@tcs.com Website: http://www.tcs.comhttp://www.tcs.com/ Experience certainty. IT Services Business Solutions Consulting =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
RE: mysql based columnar DB to Cassandra DB - Migration
Thanks Andy for quick reply. I will have a look at the below link and get back. Best Regards Akshay Ballarpure Tata Consultancy Services Cell:- 9985084075 Mailto: akshay.ballarp...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting From: Andreas Finke andreas.fi...@solvians.com To: user@cassandra.apache.org user@cassandra.apache.org Date: 11/26/2014 12:12 PM Subject:RE: mysql based columnar DB to Cassandra DB - Migration Hi Akshay, this heavily depends on your data model. There is no general way to do it. It includes several steps: 1) Migration of applications using Mysql to Cassandra 2) Migration of the Mysql Database to Cassandra itself Keep in mind that there are no such things like relations or joins in Cassandra. In general Actually Datastax published some documents that could be of interest for you: http://www.datastax.com/2012/03/how-to-move-data-from-relational-databases-to-datastax-enterprise-cassandra-using-sqoop http://www.datastax.com/resources/whitepapers/mysql-to-cassandra Also I found this from EbayTech: http://www.ebaytechblog.com/2012/07/16/cassandra-data-modeling-best-practices-part-1/#.VHV2BlXF_Eg Regards Andi From: Akshay Ballarpure [akshay.ballarp...@tcs.com] Sent: 26 November 2014 07:15 To: user@cassandra.apache.org Subject: Re: mysql based columnar DB to Cassandra DB - Migration Hello Folks, I have one mysql based columnar DB, i want to migrate it to Cassandra. How its possible ? Best Regards Akshay Ballarpure Tata Consultancy Services Cell:- 9985084075 Mailto: akshay.ballarp...@tcs.com Website: http://www.tcs.com Experience certainty.IT Services Business Solutions Consulting From:Akshay Ballarpure/HYD/TCS To:user@cassandra.apache.org Date:11/18/2014 09:00 PM Subject:mysql based columnar DB to Cassandra DB - Migration I have one mysql based columnar DB, i want to migrate it to Cassandra. How its possible ? Best Regards Akshay Ballarpure Tata Consultancy Services Cell:- 9985084075 Mailto: akshay.ballarp...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
