Using Cassandra for session tokens
We're considering switching from using Redis to Cassandra to store short lived (~1 hour) session tokens, in order to reduce the number of data storage engines we have to manage. Can anyone foresee any problems with the following approach: 1) Use the TTL functionality in Cassandra to remove old tokens. 2) Store the tokens in a table like: CREATE TABLE tokens ( id uuid, username text, // (other session information) PRIMARY KEY (id) ); 3) Perform ~100 writes/sec like: INSERT INTO tokens (id, username ) VALUES (468e0d69-1ebe-4477-8565-00a4cb6fa9f2, 'bob') USING TTL 3600; 4) Perform ~1000 reads/sec like: SELECT * FROM tokens WHERE ID=468e0d69-1ebe-4477-8565-00a4cb6fa9f2 ; The tokens will be about 100 bytes each, and we will grant 100 per second on a small 3 node cluster. Therefore there will be about 360k tokens alive at any time, with a total size of 36MB before database overhead. My biggest worry at the moment is that this kind of workload will stress compaction in an unusual way. Are there any metrics I should keep an eye on to make sure it is working fine? I read over the following links, but they mostly talk about DELETE-ing and tombstones. Am I right in thinking that as soon as a node performs a compaction then the rows with an expired TTL will be thrown away, regardless of gc_grace_seconds? https://issues.apache.org/jira/browse/CASSANDRA-7534 http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets https://issues.apache.org/jira/browse/CASSANDRA-6654 Thank you Phil
Re: does safe cassandra shutdown require disable binary?
Personally I believe that you do not have to do these steps just to perform the restart. I know the node will start faster if drained before shutdown but according to my experience these steps make the restart process slightly longer (I mean stop + start phase, total). So if it is really about rolling restart to apply some JVM or C* settings I would simply kill it and start. On Mon, Dec 1, 2014 at 12:02 AM, Kevin Burton bur...@spinn3r.com wrote: I’m trying to figure out a safe way to do a rolling restart. http://devblog.michalski.im/2012/11/25/safe-cassandra-shutdown-and-restart/ It has the following command which make sense: root@cssa01:~# nodetool -h cssa01.michalski.im disablegossiproot@cssa01:~# nodetool -h cssa01.michalski.im disablethriftroot@cssa01:~# nodetool -h cssa01.michalski.im drain … but I don’t think this takes into consideration CQL. So you would first disablethrift, then disablebinary anything else needed in modern Cassandra ? -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com -- Nikolai Grigoriev (514) 772-5178
Re: Using Cassandra for session tokens
This sounds like a good use case for http://www.datastax.com/dev/blog/datetieredcompactionstrategy http://www.datastax.com/dev/blog/datetieredcompactionstrategy On Dec 1, 2014, at 3:07 AM, Phil Wise p...@advancedtelematic.com wrote: We're considering switching from using Redis to Cassandra to store short lived (~1 hour) session tokens, in order to reduce the number of data storage engines we have to manage. Can anyone foresee any problems with the following approach: 1) Use the TTL functionality in Cassandra to remove old tokens. 2) Store the tokens in a table like: CREATE TABLE tokens ( id uuid, username text, // (other session information) PRIMARY KEY (id) ); 3) Perform ~100 writes/sec like: INSERT INTO tokens (id, username ) VALUES (468e0d69-1ebe-4477-8565-00a4cb6fa9f2, 'bob') USING TTL 3600; 4) Perform ~1000 reads/sec like: SELECT * FROM tokens WHERE ID=468e0d69-1ebe-4477-8565-00a4cb6fa9f2 ; The tokens will be about 100 bytes each, and we will grant 100 per second on a small 3 node cluster. Therefore there will be about 360k tokens alive at any time, with a total size of 36MB before database overhead. My biggest worry at the moment is that this kind of workload will stress compaction in an unusual way. Are there any metrics I should keep an eye on to make sure it is working fine? I read over the following links, but they mostly talk about DELETE-ing and tombstones. Am I right in thinking that as soon as a node performs a compaction then the rows with an expired TTL will be thrown away, regardless of gc_grace_seconds? https://issues.apache.org/jira/browse/CASSANDRA-7534 http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets https://issues.apache.org/jira/browse/CASSANDRA-6654 Thank you Phil
Re: Using Cassandra for session tokens
I don't think DateTiered will help here, since there's no clustering key defined. This is a pretty straightforward workload, I've done something similar. Are you overwriting the session on every request? Or just writing it once? On Mon Dec 01 2014 at 6:45:14 AM Matt Brown m...@mattnworb.com wrote: This sounds like a good use case for http://www.datastax.com/dev/blog/datetieredcompactionstrategy On Dec 1, 2014, at 3:07 AM, Phil Wise p...@advancedtelematic.com wrote: We're considering switching from using Redis to Cassandra to store short lived (~1 hour) session tokens, in order to reduce the number of data storage engines we have to manage. Can anyone foresee any problems with the following approach: 1) Use the TTL functionality in Cassandra to remove old tokens. 2) Store the tokens in a table like: CREATE TABLE tokens ( id uuid, username text, // (other session information) PRIMARY KEY (id) ); 3) Perform ~100 writes/sec like: INSERT INTO tokens (id, username ) VALUES (468e0d69-1ebe-4477-8565-00a4cb6fa9f2, 'bob') USING TTL 3600; 4) Perform ~1000 reads/sec like: SELECT * FROM tokens WHERE ID=468e0d69-1ebe-4477-8565-00a4cb6fa9f2 ; The tokens will be about 100 bytes each, and we will grant 100 per second on a small 3 node cluster. Therefore there will be about 360k tokens alive at any time, with a total size of 36MB before database overhead. My biggest worry at the moment is that this kind of workload will stress compaction in an unusual way. Are there any metrics I should keep an eye on to make sure it is working fine? I read over the following links, but they mostly talk about DELETE-ing and tombstones. Am I right in thinking that as soon as a node performs a compaction then the rows with an expired TTL will be thrown away, regardless of gc_grace_seconds? https://issues.apache.org/jira/browse/CASSANDRA-7534 http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets https://issues.apache.org/jira/browse/CASSANDRA-6654 Thank you Phil
Re: Using Cassandra for session tokens
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The session will be written once at create time, and never modified after that. Will that affect things? Thank you - -Phil On 01.12.2014 15:58, Jonathan Haddad wrote: I don't think DateTiered will help here, since there's no clustering key defined. This is a pretty straightforward workload, I've done something similar. Are you overwriting the session on every request? Or just writing it once? On Mon Dec 01 2014 at 6:45:14 AM Matt Brown m...@mattnworb.com wrote: This sounds like a good use case for http://www.datastax.com/dev/blog/datetieredcompactionstrategy On Dec 1, 2014, at 3:07 AM, Phil Wise p...@advancedtelematic.com wrote: We're considering switching from using Redis to Cassandra to store short lived (~1 hour) session tokens, in order to reduce the number of data storage engines we have to manage. Can anyone foresee any problems with the following approach: 1) Use the TTL functionality in Cassandra to remove old tokens. 2) Store the tokens in a table like: CREATE TABLE tokens ( id uuid, username text, // (other session information) PRIMARY KEY (id) ); 3) Perform ~100 writes/sec like: INSERT INTO tokens (id, username ) VALUES (468e0d69-1ebe-4477-8565-00a4cb6fa9f2, 'bob') USING TTL 3600; 4) Perform ~1000 reads/sec like: SELECT * FROM tokens WHERE ID=468e0d69-1ebe-4477-8565-00a4cb6fa9f2 ; The tokens will be about 100 bytes each, and we will grant 100 per second on a small 3 node cluster. Therefore there will be about 360k tokens alive at any time, with a total size of 36MB before database overhead. My biggest worry at the moment is that this kind of workload will stress compaction in an unusual way. Are there any metrics I should keep an eye on to make sure it is working fine? I read over the following links, but they mostly talk about DELETE-ing and tombstones. Am I right in thinking that as soon as a node performs a compaction then the rows with an expired TTL will be thrown away, regardless of gc_grace_seconds? https://issues.apache.org/jira/browse/CASSANDRA-7534 http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets https://issues.apache.org/jira/browse/CASSANDRA-6654 Thank you Phil -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBAgAGBQJUfIR1AAoJEAvGtrO88FBAnpAP/0RCdwCy4Wi0ogz24SRKpCu0 c/i6O2HBTinl2RXLoH9xMOT8kXJ82P9tVDeKjLQAZYnBgRwF7Fcbvd40GPf+5aaj aU1TkU4jLnDCeFTwG/vx+TIfZEE27nppsECLtfmnzJEl/4yZwAG3Dy+VkuqBurMu J6If9bMnseEgvF1onmA7ZLygJq44tlgOGyHT0WdYRX7CwAE6HeyxMC38ArarRU37 dfGhsttBMqdxHreKE0CqRZZ67iT+KixGoUeCvZUnTvOLTsrEWO17yTezQDamAee0 jIsVfgKqqhoiKeAj99J75rcsIT3WAbS23MV1s92AQXYkpR1KmHTB6KvUjH2AQBew 9xwdDSg/eVsdQNkGbtSJ2cNPnFuBBZv2kzW5PVyQ625bMHNAF2GE9rLIKddMUbNQ LiwOPAJDWBJeZnJYj3cncdfC2Jw1H4rlV0k6BHwdzZUrEdbvUKlHtyl8/ZsZnJHs SrPsiYQa0NI6C+faAFqzBEyLhsWdJL3ygNZTo4CW3I8z+yYEyzZtmKPDmHdVzK/M M8GlaRYw1t7OY81VBXKcmPyr5Omti7wtkffC6bhopsPCm7ATSq2r46z8OFlkUdJl wcTMJM0E6gZtiMIr3D+WbOTzI5kPX6x4UB3ec3xq6+GIObPwioVAJf3ADmIK4iHT G106NwdUnag5XlnbwgMX =6zXb -END PGP SIGNATURE-
Re: Using Cassandra for session tokens
Since the session tokens are random, perhaps computing a shard from each one and using it as the partition key would be a good idea. I would also use uuid v1 to get ordering. With such a small amount of data, only a few shards would be needed. On Mon, Dec 1, 2014 at 10:08 AM, Phil Wise p...@advancedtelematic.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The session will be written once at create time, and never modified after that. Will that affect things? Thank you - -Phil On 01.12.2014 15:58, Jonathan Haddad wrote: I don't think DateTiered will help here, since there's no clustering key defined. This is a pretty straightforward workload, I've done something similar. Are you overwriting the session on every request? Or just writing it once? On Mon Dec 01 2014 at 6:45:14 AM Matt Brown m...@mattnworb.com wrote: This sounds like a good use case for http://www.datastax.com/dev/blog/datetieredcompactionstrategy On Dec 1, 2014, at 3:07 AM, Phil Wise p...@advancedtelematic.com wrote: We're considering switching from using Redis to Cassandra to store short lived (~1 hour) session tokens, in order to reduce the number of data storage engines we have to manage. Can anyone foresee any problems with the following approach: 1) Use the TTL functionality in Cassandra to remove old tokens. 2) Store the tokens in a table like: CREATE TABLE tokens ( id uuid, username text, // (other session information) PRIMARY KEY (id) ); 3) Perform ~100 writes/sec like: INSERT INTO tokens (id, username ) VALUES (468e0d69-1ebe-4477-8565-00a4cb6fa9f2, 'bob') USING TTL 3600; 4) Perform ~1000 reads/sec like: SELECT * FROM tokens WHERE ID=468e0d69-1ebe-4477-8565-00a4cb6fa9f2 ; The tokens will be about 100 bytes each, and we will grant 100 per second on a small 3 node cluster. Therefore there will be about 360k tokens alive at any time, with a total size of 36MB before database overhead. My biggest worry at the moment is that this kind of workload will stress compaction in an unusual way. Are there any metrics I should keep an eye on to make sure it is working fine? I read over the following links, but they mostly talk about DELETE-ing and tombstones. Am I right in thinking that as soon as a node performs a compaction then the rows with an expired TTL will be thrown away, regardless of gc_grace_seconds? https://issues.apache.org/jira/browse/CASSANDRA-7534 http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets https://issues.apache.org/jira/browse/CASSANDRA-6654 Thank you Phil -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBAgAGBQJUfIR1AAoJEAvGtrO88FBAnpAP/0RCdwCy4Wi0ogz24SRKpCu0 c/i6O2HBTinl2RXLoH9xMOT8kXJ82P9tVDeKjLQAZYnBgRwF7Fcbvd40GPf+5aaj aU1TkU4jLnDCeFTwG/vx+TIfZEE27nppsECLtfmnzJEl/4yZwAG3Dy+VkuqBurMu J6If9bMnseEgvF1onmA7ZLygJq44tlgOGyHT0WdYRX7CwAE6HeyxMC38ArarRU37 dfGhsttBMqdxHreKE0CqRZZ67iT+KixGoUeCvZUnTvOLTsrEWO17yTezQDamAee0 jIsVfgKqqhoiKeAj99J75rcsIT3WAbS23MV1s92AQXYkpR1KmHTB6KvUjH2AQBew 9xwdDSg/eVsdQNkGbtSJ2cNPnFuBBZv2kzW5PVyQ625bMHNAF2GE9rLIKddMUbNQ LiwOPAJDWBJeZnJYj3cncdfC2Jw1H4rlV0k6BHwdzZUrEdbvUKlHtyl8/ZsZnJHs SrPsiYQa0NI6C+faAFqzBEyLhsWdJL3ygNZTo4CW3I8z+yYEyzZtmKPDmHdVzK/M M8GlaRYw1t7OY81VBXKcmPyr5Omti7wtkffC6bhopsPCm7ATSq2r46z8OFlkUdJl wcTMJM0E6gZtiMIr3D+WbOTzI5kPX6x4UB3ec3xq6+GIObPwioVAJf3ADmIK4iHT G106NwdUnag5XlnbwgMX =6zXb -END PGP SIGNATURE-
Re: does safe cassandra shutdown require disable binary?
On Sun, Nov 30, 2014 at 9:02 PM, Kevin Burton bur...@spinn3r.com wrote: It has the following command which make sense: root@cssa01:~# nodetool -h cssa01.michalski.im disablegossiproot@cssa01:~# nodetool -h cssa01.michalski.im disablethriftroot@cssa01:~# nodetool -h cssa01.michalski.im drain … but I don’t think this takes into consideration CQL. Nothing but the drain there really shuts off writes to the node. https://issues.apache.org/jira/browse/CASSANDRA-4162 If I wanted to do anything but stop a node on clean shutdown, I would limit my actions to drain. And usually I wouldn't bother. =Rob
Re: Repair hanging with C* 2.1.2
On Fri, Nov 28, 2014 at 6:46 AM, Reynald BOURTEMBOURG reynald.bourtembo...@esrf.fr wrote: We have a three nodes cluster running Cassandra 2.1.2 on Linux Debian 7. More than 2 hours later, I executed nodetool repair on one of the nodes (cass2). It started to repair the keyspace we created (RF=3) and it got stuck there. The nodetool repair command didn't return yet. Yes, repair has historically not really worked, and still hangs sometimes. Search the archives for tons of posts where I link various JIRA tickets with background. On the plus side, so many people have had such a negative experience for so long that the squeaky wheel is finally getting the grease. Significant improvements have been made in the reliability and transparency of repair in recent versions. Is there anything that you could suggest me to do before to restart any node in order to try to better understand the origin of this problem, if this is an unknown issue? You could use the JMX endpoint to stop repair, but if you have vnodes it's probably easier to just restart the affected nodes. https://issues.apache.org/jira/browse/CASSANDRA-3486 =Rob
Re: Data synchronization between 2 running clusters on different availability zone
On Thu, Nov 27, 2014 at 1:24 AM, Spico Florin spicoflo...@gmail.com wrote: I have another question. What about the following scenario: two Cassandra instances installed on different cloud providers (EC2, Flexiant)? How do you synchronize them? Can you use some internal tools or do I have to implement my own mechanism? That's what I meant by if maybe hybrid in the future, use GPFS : http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureSnitchGossipPF_c.html hybrid in this case means AWS-and-not-AWS. =Rob
Re: Cassandra add a node and remove a node
On Sun, Nov 30, 2014 at 10:15 PM, Neha Trivedi nehajtriv...@gmail.com wrote: I need to Add new Node and remove existing node. What is the purpose of this action? Is the old node defective, and being replaced 1:1 with the new node? =Rob
Re: Data synchronization between 2 running clusters on different availability zone
Here's a snitch we use for this situation - it uses a property file if it exists, but falls back to EC2 autodiscovery if it is missing. https://github.com/barchart/cassandra-plugins/blob/master/src/main/java/com/barchart/cassandra/plugins/snitch/GossipingPropertyFileWithEC2FallbackSnitch.java On Mon, Dec 1, 2014 at 12:33 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Nov 27, 2014 at 1:24 AM, Spico Florin spicoflo...@gmail.com wrote: I have another question. What about the following scenario: two Cassandra instances installed on different cloud providers (EC2, Flexiant)? How do you synchronize them? Can you use some internal tools or do I have to implement my own mechanism? That's what I meant by if maybe hybrid in the future, use GPFS : http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureSnitchGossipPF_c.html hybrid in this case means AWS-and-not-AWS. =Rob
Re: Using Cassandra for session tokens
I don't know what the advantage would be of using this sharding system. I would recommend just going with a simple k-v table as the OP suggested. On Mon Dec 01 2014 at 7:18:51 AM Laing, Michael michael.la...@nytimes.com wrote: Since the session tokens are random, perhaps computing a shard from each one and using it as the partition key would be a good idea. I would also use uuid v1 to get ordering. With such a small amount of data, only a few shards would be needed. On Mon, Dec 1, 2014 at 10:08 AM, Phil Wise p...@advancedtelematic.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The session will be written once at create time, and never modified after that. Will that affect things? Thank you - -Phil On 01.12.2014 15:58, Jonathan Haddad wrote: I don't think DateTiered will help here, since there's no clustering key defined. This is a pretty straightforward workload, I've done something similar. Are you overwriting the session on every request? Or just writing it once? On Mon Dec 01 2014 at 6:45:14 AM Matt Brown m...@mattnworb.com wrote: This sounds like a good use case for http://www.datastax.com/dev/blog/datetieredcompactionstrategy On Dec 1, 2014, at 3:07 AM, Phil Wise p...@advancedtelematic.com wrote: We're considering switching from using Redis to Cassandra to store short lived (~1 hour) session tokens, in order to reduce the number of data storage engines we have to manage. Can anyone foresee any problems with the following approach: 1) Use the TTL functionality in Cassandra to remove old tokens. 2) Store the tokens in a table like: CREATE TABLE tokens ( id uuid, username text, // (other session information) PRIMARY KEY (id) ); 3) Perform ~100 writes/sec like: INSERT INTO tokens (id, username ) VALUES (468e0d69-1ebe-4477-8565-00a4cb6fa9f2, 'bob') USING TTL 3600; 4) Perform ~1000 reads/sec like: SELECT * FROM tokens WHERE ID=468e0d69-1ebe-4477-8565-00a4cb6fa9f2 ; The tokens will be about 100 bytes each, and we will grant 100 per second on a small 3 node cluster. Therefore there will be about 360k tokens alive at any time, with a total size of 36MB before database overhead. My biggest worry at the moment is that this kind of workload will stress compaction in an unusual way. Are there any metrics I should keep an eye on to make sure it is working fine? I read over the following links, but they mostly talk about DELETE-ing and tombstones. Am I right in thinking that as soon as a node performs a compaction then the rows with an expired TTL will be thrown away, regardless of gc_grace_seconds? https://issues.apache.org/jira/browse/CASSANDRA-7534 http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets https://issues.apache.org/jira/browse/CASSANDRA-6654 Thank you Phil -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBAgAGBQJUfIR1AAoJEAvGtrO88FBAnpAP/0RCdwCy4Wi0ogz24SRKpCu0 c/i6O2HBTinl2RXLoH9xMOT8kXJ82P9tVDeKjLQAZYnBgRwF7Fcbvd40GPf+5aaj aU1TkU4jLnDCeFTwG/vx+TIfZEE27nppsECLtfmnzJEl/4yZwAG3Dy+VkuqBurMu J6If9bMnseEgvF1onmA7ZLygJq44tlgOGyHT0WdYRX7CwAE6HeyxMC38ArarRU37 dfGhsttBMqdxHreKE0CqRZZ67iT+KixGoUeCvZUnTvOLTsrEWO17yTezQDamAee0 jIsVfgKqqhoiKeAj99J75rcsIT3WAbS23MV1s92AQXYkpR1KmHTB6KvUjH2AQBew 9xwdDSg/eVsdQNkGbtSJ2cNPnFuBBZv2kzW5PVyQ625bMHNAF2GE9rLIKddMUbNQ LiwOPAJDWBJeZnJYj3cncdfC2Jw1H4rlV0k6BHwdzZUrEdbvUKlHtyl8/ZsZnJHs SrPsiYQa0NI6C+faAFqzBEyLhsWdJL3ygNZTo4CW3I8z+yYEyzZtmKPDmHdVzK/M M8GlaRYw1t7OY81VBXKcmPyr5Omti7wtkffC6bhopsPCm7ATSq2r46z8OFlkUdJl wcTMJM0E6gZtiMIr3D+WbOTzI5kPX6x4UB3ec3xq6+GIObPwioVAJf3ADmIK4iHT G106NwdUnag5XlnbwgMX =6zXb -END PGP SIGNATURE-
Re: Using Cassandra for session tokens
Sharding lets you use the row cache effectively in 2.1. But like everything, one should test :) On Mon, Dec 1, 2014 at 1:56 PM, Jonathan Haddad j...@jonhaddad.com wrote: I don't know what the advantage would be of using this sharding system. I would recommend just going with a simple k-v table as the OP suggested. On Mon Dec 01 2014 at 7:18:51 AM Laing, Michael michael.la...@nytimes.com wrote: Since the session tokens are random, perhaps computing a shard from each one and using it as the partition key would be a good idea. I would also use uuid v1 to get ordering. With such a small amount of data, only a few shards would be needed. On Mon, Dec 1, 2014 at 10:08 AM, Phil Wise p...@advancedtelematic.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The session will be written once at create time, and never modified after that. Will that affect things? Thank you - -Phil On 01.12.2014 15:58, Jonathan Haddad wrote: I don't think DateTiered will help here, since there's no clustering key defined. This is a pretty straightforward workload, I've done something similar. Are you overwriting the session on every request? Or just writing it once? On Mon Dec 01 2014 at 6:45:14 AM Matt Brown m...@mattnworb.com wrote: This sounds like a good use case for http://www.datastax.com/dev/blog/datetieredcompactionstrategy On Dec 1, 2014, at 3:07 AM, Phil Wise p...@advancedtelematic.com wrote: We're considering switching from using Redis to Cassandra to store short lived (~1 hour) session tokens, in order to reduce the number of data storage engines we have to manage. Can anyone foresee any problems with the following approach: 1) Use the TTL functionality in Cassandra to remove old tokens. 2) Store the tokens in a table like: CREATE TABLE tokens ( id uuid, username text, // (other session information) PRIMARY KEY (id) ); 3) Perform ~100 writes/sec like: INSERT INTO tokens (id, username ) VALUES (468e0d69-1ebe-4477-8565-00a4cb6fa9f2, 'bob') USING TTL 3600; 4) Perform ~1000 reads/sec like: SELECT * FROM tokens WHERE ID=468e0d69-1ebe-4477-8565-00a4cb6fa9f2 ; The tokens will be about 100 bytes each, and we will grant 100 per second on a small 3 node cluster. Therefore there will be about 360k tokens alive at any time, with a total size of 36MB before database overhead. My biggest worry at the moment is that this kind of workload will stress compaction in an unusual way. Are there any metrics I should keep an eye on to make sure it is working fine? I read over the following links, but they mostly talk about DELETE-ing and tombstones. Am I right in thinking that as soon as a node performs a compaction then the rows with an expired TTL will be thrown away, regardless of gc_grace_seconds? https://issues.apache.org/jira/browse/CASSANDRA-7534 http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets https://issues.apache.org/jira/browse/CASSANDRA-6654 Thank you Phil -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBAgAGBQJUfIR1AAoJEAvGtrO88FBAnpAP/0RCdwCy4Wi0ogz24SRKpCu0 c/i6O2HBTinl2RXLoH9xMOT8kXJ82P9tVDeKjLQAZYnBgRwF7Fcbvd40GPf+5aaj aU1TkU4jLnDCeFTwG/vx+TIfZEE27nppsECLtfmnzJEl/4yZwAG3Dy+VkuqBurMu J6If9bMnseEgvF1onmA7ZLygJq44tlgOGyHT0WdYRX7CwAE6HeyxMC38ArarRU37 dfGhsttBMqdxHreKE0CqRZZ67iT+KixGoUeCvZUnTvOLTsrEWO17yTezQDamAee0 jIsVfgKqqhoiKeAj99J75rcsIT3WAbS23MV1s92AQXYkpR1KmHTB6KvUjH2AQBew 9xwdDSg/eVsdQNkGbtSJ2cNPnFuBBZv2kzW5PVyQ625bMHNAF2GE9rLIKddMUbNQ LiwOPAJDWBJeZnJYj3cncdfC2Jw1H4rlV0k6BHwdzZUrEdbvUKlHtyl8/ZsZnJHs SrPsiYQa0NI6C+faAFqzBEyLhsWdJL3ygNZTo4CW3I8z+yYEyzZtmKPDmHdVzK/M M8GlaRYw1t7OY81VBXKcmPyr5Omti7wtkffC6bhopsPCm7ATSq2r46z8OFlkUdJl wcTMJM0E6gZtiMIr3D+WbOTzI5kPX6x4UB3ec3xq6+GIObPwioVAJf3ADmIK4iHT G106NwdUnag5XlnbwgMX =6zXb -END PGP SIGNATURE-
Re: Repair completes successfully but data is still inconsistent
On Thu, Nov 27, 2014 at 2:38 AM, André Cruz andre.c...@co.sapo.pt wrote: On 26 Nov 2014, at 19:07, Robert Coli rc...@eventbrite.com wrote: Yes. Do you know if 5748 was created as a result of compaction or via a flush from a memtable? It was the result of a compaction: Ok, so in theory if you had the input SSTables to that compaction, you could use them to diagnose this bug, by seeing what values they had for that row before compaction. This is why something like tablesnap can be useful in practice... =Rob
Re: Cassandra backup via snapshots in production
On Thu, Nov 27, 2014 at 2:34 AM, Jens Rantil jens.ran...@tink.se wrote: Late answer; You can find my backup script here: https://gist.github.com/JensRantil/a8150e998250edfcd1a3 Why not use the much more robustly designed and maintained community based project, tablesnap? https://github.com/JeremyGrosser/tablesnap =Rob
Re: Performance Difference between Batch Insert and Bulk Load
On Sun, Nov 30, 2014 at 8:44 PM, Dong Dai daidon...@gmail.com wrote: The question is can I expect a better performance using the BulkLoader this way comparing with using Batch insert? You just asked if writing once (via streaming) is likely to be significantly more efficient than writing twice (once to the commit log, and then once at flush time). Yes. =Rob
Re: Nodes get stuck in crazy GC loop after some time, leading to timeouts
On Fri, Nov 28, 2014 at 12:55 PM, Paulo Ricardo Motta Gomes paulo.mo...@chaordicsystems.com wrote: We restart the whole cluster every 1 or 2 months, to avoid machines getting into this crazy state. We tried tuning GC size and parameters, different cassandra versions (1.1, 1.2, 2.0), but this behavior keeps happening. More recently, during black friday, we received about 5x our normal load, and some machines started presenting this behavior. Once again, we restart the nodes an the GC behaves normal again. ... You can clearly notice some memory is actually reclaimed during GC in healthy nodes, while in sick machines very little memory is reclaimed. Also, since GC is executed more frequently in sick machines, it uses about 2x more CPU than non-sick nodes. Have you ever observed this behavior in your cluster? Could this be related to heap fragmentation? Would using the G1 collector help in this case? Any GC tuning or monitoring advice to troubleshoot this issue? The specific combo of symptoms does in fact sound like a combination of being close to heap exhaustion with working set and then fragmentation putting you over the top. I would probably start by increasing your heap, which will help avoid the pre-fail condition from your working set. But for tuning, examine the contents of each generation when the JVM gets into this state. You are probably exhausting permanent generation, but depending on what that says, you could change the relatively sizing of the generations. =Rob
Re: Performance Difference between Batch Insert and Bulk Load
Thanks Rob, I guess you mean that BulkLoader is done by streaming whole SSTable to remote servers, so it is faster? The documentation says that all the rows in the SSTable will be inserted into the new cluster conforming to the replication strategy of that cluster. This gives me a felling that the BulkLoader was done by calling insertion after being transmitted to coordinators. I have this question because I tried batch insertion. It is too fast and makes me think that BulkLoader can not beat it. thanks, - Dong On Dec 1, 2014, at 1:37 PM, Robert Coli rc...@eventbrite.com wrote: On Sun, Nov 30, 2014 at 8:44 PM, Dong Dai daidon...@gmail.com mailto:daidon...@gmail.com wrote: The question is can I expect a better performance using the BulkLoader this way comparing with using Batch insert? You just asked if writing once (via streaming) is likely to be significantly more efficient than writing twice (once to the commit log, and then once at flush time). Yes. =Rob
What happens at server side when using SSTableLoader
Is there a page explaining what happens at server side when using SSTableLoader? I'm seeking the answers of the following questions: 1. What's about the existing data in the table? From my test, the data in sstable files will be applied to the existing data. Am I right? - The new data row or columns in sstable will be created - The existing columns will be updated. - The deleted row/column are also applied. 2. What the impact to read operations when you are bulk loading data? Thanks.
Re: Performance Difference between Batch Insert and Bulk Load
On Mon, Dec 1, 2014 at 12:10 PM, Dong Dai daidon...@gmail.com wrote: I guess you mean that BulkLoader is done by streaming whole SSTable to remote servers, so it is faster? Well, it's not exactly whole SSTable but yes, that's the sort of statement I'm making. [1] The documentation says that all the rows in the SSTable will be inserted into the new cluster conforming to the replication strategy of that cluster. This gives me a felling that the BulkLoader was done by calling insertion after being transmitted to coordinators. A good slide-deck from pgorla, here : http://www.slideshare.net/DataStax/bulk-loading-data-into-cassandra General background. http://www.palominodb.com/blog/2012/09/25/bulk-loading-options-cassandra But briefly, no. It uses the streaming interface, not the client interface. The streaming interface results in avoiding the whole commitlog/memtable process. I have this question because I tried batch insertion. It is too fast and makes me think that BulkLoader can not beat it. Turn of writes to the commitlog with durable_writes:false and you can simulate how much faster it would be without the double-write to the commitlog. That said, the double-write to the commitlog is one of the most significant overheads of doing a write from the client, but it is far from the only overhead. =Rob [1] http://www.datastax.com/dev/blog/streaming-in-cassandra-2-0
Error when dropping keyspaces; One row required, 0 found
I'm running Cassandra 2.1.0. I was attempting to drop two keyspaces via cqlsh and encountered an error in the CLI as well as the appearance of losing all my keyspaces. Below is the output from my cqlsh session: $ cqlsh Connected to Production Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 2.1.0 | CQL spec 3.2.0 | Native protocol v3] Use HELP for help. cqlsh desc keyspaces; contacts_index contacts_testing contacts system OpsCenter system_traces cqlsh drop keyspace contacts_index; cqlsh drop keyspace contacts; ErrorMessage code= [Server error] message=java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NullPointerException cqlsh drop keyspace contacts; ErrorMessage code= [Server error] message=java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: One row required, 0 found *cqlsh desc keyspaces;empty* -- OH SHIT -- After it appeared that I had lost all my keyspaces, I looked at the system.log and found this: (full log attached) ERROR [MigrationStage:1] 2014-12-01 23:54:05,622 CassandraDaemon.java:166 - Exception in thread Thread[MigrationStage:1,5,main] java.lang.IllegalStateException: One row required, 0 found at org.apache.cassandra.cql3.UntypedResultSet$FromResultSet.one(UntypedResultSet.java:78) ~[apache-cassandra-2.1.0.jar:2.1.0] at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:275) ~[apache-cassandra-2.1.0.jar:2.1.0] at org.apache.cassandra.db.DefsTables.mergeKeyspaces(DefsTables.java:230) ~[apache-cassandra-2.1.0.jar:2.1.0] at org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:186) ~[apache-cassandra-2.1.0.jar:2.1.0] at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:164) ~[apache-cassandra-2.1.0.jar:2.1.0] at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:49) ~[apache-cassandra-2.1.0.jar:2.1.0] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0.jar:2.1.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_65] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] At this point I wasn't sure quite what to do about this and did a rolling restart of the entire ring. After which, the keyspaces that were not attempted to be deleted returned when running 'desc keyspaces' and my intended keyspaces to be deleted had been removed as expected. Strangely enough, because we run OpsCenter, we lost the dashboards we had configured. Not a total deal breaker, but concerning that data loss occurred here assuming it's related. Anyone run into something like this before? system.log Description: Binary data
Re: Cassandra add a node and remove a node
No the old node is not defective. We Just want to separate out that Server for testing. And add a new node. (Present cluster has two Nodes and RF=2) thanks On Tue, Dec 2, 2014 at 12:04 AM, Robert Coli rc...@eventbrite.com wrote: On Sun, Nov 30, 2014 at 10:15 PM, Neha Trivedi nehajtriv...@gmail.com wrote: I need to Add new Node and remove existing node. What is the purpose of this action? Is the old node defective, and being replaced 1:1 with the new node? =Rob
Re: opscenter with community cassandra
Hi Tim, We have happy SPM for Cassandra users who are using SPM for monitoring, alerting, and anomaly detection for their Cassandra clusters. SPM agents phone home by definition if you are using the Cloud version. The On Premise / AWS AMI versions do not phone home. See: http://sematext.com/spm http://blog.sematext.com/tag/cassandra/ http://blog.sematext.com/2014/06/02/announcement-cassandra-performance-monitoring-in-spm/ Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Tue, Oct 28, 2014 at 12:03 PM, Tim Dunphy bluethu...@gmail.com wrote: Furthermore, people ask questions about monitoring and management utilities for Cassandra all the time--this is in the same vein. Speaking of which. Are there any viable alternatives to opscenter that people also like? On Tue, Oct 28, 2014 at 11:56 AM, Redmumba redmu...@gmail.com wrote: Furthermore, people ask questions about monitoring and management utilities for Cassandra all the time--this is in the same vein. On Tue, Oct 28, 2014 at 8:52 AM, Ken Hancock ken.hanc...@schange.com wrote: Your criteria for what is appropriate for production may differ from others, but it's equally incorrect of you to make a blanket statement that OpsCenter isn't suitable for production. A number of people use it in production. On Tue, Oct 28, 2014 at 11:48 AM, Colin co...@clark.ws wrote: No, actually, you cant Tyler. If you mean the useless information it provides outside of licence, fine, if you mean the components outside, then same argument. Last time i checked, this forumn was about apache and not about datastax. Maybe a separate group should be deducated to provider specific offerings. -- *Colin Clark* +1-320-221-9531 On Oct 28, 2014, at 10:41 AM, Tyler Hobbs ty...@datastax.com wrote: On Tue, Oct 28, 2014 at 10:08 AM, Colin colpcl...@gmail.com wrote: It is a mistake to call a proprietary piece of software community when you cant use it in production. You can use OpsCenter community in production (however you'd like). -- Tyler Hobbs DataStax http://datastax.com/ -- *Ken Hancock *| System Architect, Advanced Advertising SeaChange International 50 Nagog Park Acton, Massachusetts 01720 ken.hanc...@schange.com | www.schange.com | NASDAQ:SEAC http://www.schange.com/en-US/Company/InvestorRelations.aspx Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hanc...@schange.com | [image: Skype:]hancockks | [image: Yahoo IM:] hancockks[image: LinkedIn] http://www.linkedin.com/in/kenhancock [image: SeaChange International] http://www.schange.com/This e-mail and any attachments may contain information which is SeaChange International confidential. The information enclosed is intended only for the addressees herein and may not be copied or forwarded without permission from SeaChange International. -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
Re: Nodes get stuck in crazy GC loop after some time, leading to timeouts
Hi Rob, any recommended documentation on describing explanation/configuration of the JVM heap and permanent generation ? We stucked in this same situation too. :( Jason On Tue, Dec 2, 2014 at 3:42 AM, Robert Coli rc...@eventbrite.com wrote: On Fri, Nov 28, 2014 at 12:55 PM, Paulo Ricardo Motta Gomes paulo.mo...@chaordicsystems.com wrote: We restart the whole cluster every 1 or 2 months, to avoid machines getting into this crazy state. We tried tuning GC size and parameters, different cassandra versions (1.1, 1.2, 2.0), but this behavior keeps happening. More recently, during black friday, we received about 5x our normal load, and some machines started presenting this behavior. Once again, we restart the nodes an the GC behaves normal again. ... You can clearly notice some memory is actually reclaimed during GC in healthy nodes, while in sick machines very little memory is reclaimed. Also, since GC is executed more frequently in sick machines, it uses about 2x more CPU than non-sick nodes. Have you ever observed this behavior in your cluster? Could this be related to heap fragmentation? Would using the G1 collector help in this case? Any GC tuning or monitoring advice to troubleshoot this issue? The specific combo of symptoms does in fact sound like a combination of being close to heap exhaustion with working set and then fragmentation putting you over the top. I would probably start by increasing your heap, which will help avoid the pre-fail condition from your working set. But for tuning, examine the contents of each generation when the JVM gets into this state. You are probably exhausting permanent generation, but depending on what that says, you could change the relatively sizing of the generations. =Rob
Re: Cassandra backup via snapshots in production
On Mon, Dec 1, 2014 at 8:39 PM, Robert Coli rc...@eventbrite.com wrote: Why not use the much more robustly designed and maintained community based project, tablesnap? For two reasons: - Because I am tired of the deployment model of Python apps which require me to set up virtual environments. - Because it did, AFAIK, not support (asymmetric) encryption before uploading. -- Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se Phone: +46 708 84 18 32 Web: www.tink.se Facebook https://www.facebook.com/#!/tink.se Linkedin http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_phototrkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary Twitter https://twitter.com/tink
