Using mySQL to emulate Cassandra
As our launch date approaches, I am getting increasingly nervous about Cassandra tuning. It is a mysterious black art that I haven't mastered even at the low usages that we have now. I know of a few more things I can do to improve things, but how will I know if it is enough? All this is particularly ironic since - as we are just starting out - we don't have scalability problems yet, though we hope to! Luckily, I have completely wrapped Cassandra in an entity mapper, so that I can easily trade in something else, perhaps temporarily, until we really need Cassandra's scalability. So, I'm thinking of emulating Cassandra with mySQL. I would use mySQL either as a simple key-value store, without joins, or map Cassandra supercolumns to mySQL columns, probably of type CLOB. Does anyone want to talk me out of this?
Re: Using mySQL to emulate Cassandra
If you want to do this, don´t use autoincrementing integers! ;o) Use UUID instead. (or implement an stand alone id-server that handles autoincrement savely) 2010/11/28 David Boxenhorn da...@lookin2.com As our launch date approaches, I am getting increasingly nervous about Cassandra tuning. It is a mysterious black art that I haven't mastered even at the low usages that we have now. I know of a few more things I can do to improve things, but how will I know if it is enough? All this is particularly ironic since - as we are just starting out - we don't have scalability problems yet, though we hope to! Luckily, I have completely wrapped Cassandra in an entity mapper, so that I can easily trade in something else, perhaps temporarily, until we really need Cassandra's scalability. So, I'm thinking of emulating Cassandra with mySQL. I would use mySQL either as a simple key-value store, without joins, or map Cassandra supercolumns to mySQL columns, probably of type CLOB. Does anyone want to talk me out of this?
Taking down a node in a 3-node cluster, RF=2
For the vast majority of my data usage eventual consistency is fine (i.e. CL=ONE) but I have a small amount of critical data for which I read and write using CL=QUORUM. If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean that a value can be read from or written to any 2 nodes, or does it have to be the particular 2 nodes that store the data? If it is the particular 2 nodes that store the data, that means that I can't even take down one node, since it will be the mandatory 2nd node for 1/3 of my data...
Re: Taking down a node in a 3-node cluster, RF=2
to me it makes sense that if hinted handoff is off then cassandra cannot satisfy 2 out of every 3rd writes writes when one of the nodes is down since this node is the designated node of 2/3 writes. But I don't remember reading this somewhere. Does hinted handoff affect David's situation? (David, did you disable HH in your storage-config? HintedHandoffEnabledfalse/HintedHandoffEnabled) On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn da...@lookin2.com wrote: For the vast majority of my data usage eventual consistency is fine (i.e. CL=ONE) but I have a small amount of critical data for which I read and write using CL=QUORUM. If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean that a value can be read from or written to any 2 nodes, or does it have to be the particular 2 nodes that store the data? If it is the particular 2 nodes that store the data, that means that I can't even take down one node, since it will be the mandatory 2nd node for 1/3 of my data... -- /Ran
Re: Taking down a node in a 3-node cluster, RF=2
Actually you can't. As explained in the wiki page linked: A hinted write does NOT count towards ConsistencyLevel requirements of ONE, QUORUM, or ALL For CL.QUORUM, you do need QUORUM *replicas* to be alive to answer the query. At RF=2, QUORUM=2 so no, you cannot take down any node down or (some) quorum writes/reads will result in UnavailableException. And this is not related to the number of node you have, only to the replication factor. If you want to support having a node down, you need to have RF=3. For that very reason, this is the minimum replication factor I would advise for a production cluster. -- Sylvain On Sun, Nov 28, 2010 at 6:11 PM, Jake Luciani jak...@gmail.com wrote: If you read/write data with quorum then you can safely take a node down in this scenario. Subsequent writes will use hinted handoff to be passed to the node when it comes back up. More info is here: http://wiki.apache.org/cassandra/HintedHandoff Does that answer your question? -Jake On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory ran...@gmail.com wrote: to me it makes sense that if hinted handoff is off then cassandra cannot satisfy 2 out of every 3rd writes writes when one of the nodes is down since this node is the designated node of 2/3 writes. But I don't remember reading this somewhere. Does hinted handoff affect David's situation? (David, did you disable HH in your storage-config? HintedHandoffEnabledfalse/HintedHandoffEnabled) On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn da...@lookin2.com wrote: For the vast majority of my data usage eventual consistency is fine (i.e. CL=ONE) but I have a small amount of critical data for which I read and write using CL=QUORUM. If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean that a value can be read from or written to any 2 nodes, or does it have to be the particular 2 nodes that store the data? If it is the particular 2 nodes that store the data, that means that I can't even take down one node, since it will be the mandatory 2nd node for 1/3 of my data... -- /Ran
Re: Taking down a node in a 3-node cluster, RF=2
Thank you, Jake. It does... except that in another context you told me: Hints only happen when a node is unavailable and you are writing with CL.ANY If you never write with CL.ANY then you can turn off hinted handoff. How do I reconcile this? On Sun, Nov 28, 2010 at 7:11 PM, Jake Luciani jak...@gmail.com wrote: If you read/write data with quorum then you can safely take a node down in this scenario. Subsequent writes will use hinted handoff to be passed to the node when it comes back up. More info is here: http://wiki.apache.org/cassandra/HintedHandoff Does that answer your question? -Jake On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory ran...@gmail.com wrote: to me it makes sense that if hinted handoff is off then cassandra cannot satisfy 2 out of every 3rd writes writes when one of the nodes is down since this node is the designated node of 2/3 writes. But I don't remember reading this somewhere. Does hinted handoff affect David's situation? (David, did you disable HH in your storage-config? HintedHandoffEnabledfalse/HintedHandoffEnabled) On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn da...@lookin2.comwrote: For the vast majority of my data usage eventual consistency is fine (i.e. CL=ONE) but I have a small amount of critical data for which I read and write using CL=QUORUM. If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean that a value can be read from or written to any 2 nodes, or does it have to be the particular 2 nodes that store the data? If it is the particular 2 nodes that store the data, that means that I can't even take down one node, since it will be the mandatory 2nd node for 1/3 of my data... -- /Ran
Re: Taking down a node in a 3-node cluster, RF=2
On Sun, Nov 28, 2010 at 6:26 PM, David Boxenhorn da...@lookin2.com wrote: Hints only happen when a node is unavailable and you are writing with CL.ANY If you never write with CL.ANY then you can turn off hinted handoff. Hints happens at any consistency level. But they never helps to reach the consistency level requirements, except at CL.ANY. To say this otherwise, HH are (only) useful to make dead nodes 'catch up' quicker when they are back up (that is, without waiting for a read-repair or a nodetool repair to happen). But they doesn't help a write to work where it wouldn't have without HH. Except at CL.ANY. At CL.ANY, even if all the replicas for the data you are writting are dead, the write is still accepted and a hint will be used. -- Sylvain How do I reconcile this? On Sun, Nov 28, 2010 at 7:11 PM, Jake Luciani jak...@gmail.com wrote: If you read/write data with quorum then you can safely take a node down in this scenario. Subsequent writes will use hinted handoff to be passed to the node when it comes back up. More info is here: http://wiki.apache.org/cassandra/HintedHandoff Does that answer your question? -Jake On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory ran...@gmail.com wrote: to me it makes sense that if hinted handoff is off then cassandra cannot satisfy 2 out of every 3rd writes writes when one of the nodes is down since this node is the designated node of 2/3 writes. But I don't remember reading this somewhere. Does hinted handoff affect David's situation? (David, did you disable HH in your storage-config? HintedHandoffEnabledfalse/HintedHandoffEnabled) On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn da...@lookin2.com wrote: For the vast majority of my data usage eventual consistency is fine (i.e. CL=ONE) but I have a small amount of critical data for which I read and write using CL=QUORUM. If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean that a value can be read from or written to any 2 nodes, or does it have to be the particular 2 nodes that store the data? If it is the particular 2 nodes that store the data, that means that I can't even take down one node, since it will be the mandatory 2nd node for 1/3 of my data... -- /Ran
Re: Taking down a node in a 3-node cluster, RF=2
I was wrong on this scenario and I'll explain where I was incorrect. Hints are stored for a downed node but they don't count towards meeting a consistency level. Let's take 2 scenarios: RF=6, Nodes=10 If you READ/WRITE with CL.QUORUM you will need 4 alive nodes if one is down it will still have 4 active replicas to write to, one of these will store a hint and update the downed node when it comes back. RF=2, Nodes=3 If you READ/WRITE with CL.QUORUM you need 2 live nodes. If one of these 2 are down you can't meet the QUORUM level so the write will fail. In your scenario your best bet is to update to RF=3, then any two nodes will accept QUORUM Sorry for the confusion, -Jake On Sun, Nov 28, 2010 at 12:26 PM, David Boxenhorn da...@lookin2.com wrote: Thank you, Jake. It does... except that in another context you told me: Hints only happen when a node is unavailable and you are writing with CL.ANY If you never write with CL.ANY then you can turn off hinted handoff. How do I reconcile this? On Sun, Nov 28, 2010 at 7:11 PM, Jake Luciani jak...@gmail.com wrote: If you read/write data with quorum then you can safely take a node down in this scenario. Subsequent writes will use hinted handoff to be passed to the node when it comes back up. More info is here: http://wiki.apache.org/cassandra/HintedHandoff Does that answer your question? -Jake On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory ran...@gmail.com wrote: to me it makes sense that if hinted handoff is off then cassandra cannot satisfy 2 out of every 3rd writes writes when one of the nodes is down since this node is the designated node of 2/3 writes. But I don't remember reading this somewhere. Does hinted handoff affect David's situation? (David, did you disable HH in your storage-config? HintedHandoffEnabledfalse/HintedHandoffEnabled) On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn da...@lookin2.comwrote: For the vast majority of my data usage eventual consistency is fine (i.e. CL=ONE) but I have a small amount of critical data for which I read and write using CL=QUORUM. If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean that a value can be read from or written to any 2 nodes, or does it have to be the particular 2 nodes that store the data? If it is the particular 2 nodes that store the data, that means that I can't even take down one node, since it will be the mandatory 2nd node for 1/3 of my data... -- /Ran
Re: Using mySQL to emulate Cassandra
On Sun, Nov 28, 2010 at 11:35 AM, Tom Melendez t...@supertom.com wrote: On Sun, Nov 28, 2010 at 12:28 AM, David Boxenhorn da...@lookin2.com wrote: As our launch date approaches, I am getting increasingly nervous about Cassandra tuning. It is a mysterious black art that I haven't mastered even at the low usages that we have now. I know of a few more things I can do to improve things, but how will I know if it is enough? All this is particularly ironic since - as we are just starting out - we don't have scalability problems yet, though we hope to! How are your load tests looking? Of course, there's nothing like going live, but I expect you'll be able to simulate 2x-3x your initial launch traffic. Luckily, I have completely wrapped Cassandra in an entity mapper, so that I can easily trade in something else, perhaps temporarily, until we really need Cassandra's scalability. So, I'm thinking of emulating Cassandra with mySQL. I would use mySQL either as a simple key-value store, without joins, or map Cassandra supercolumns to mySQL columns, probably of type CLOB. Does anyone want to talk me out of this? As you said, I think you just have some cold feet. My feeling is that you did some original research and decided on Cassandra for various reasons. I think if you put the MySQL solution in now, you won't go back to the Cassandra solution, because once its live, it will be much riskier to switch. And if you feel you made a mistake in your original assessment, then great, at least you found out before launch. Whatever you choose, I would flesh out my my fears with as much detail as possible. Invest in load tests and develop contingency plans. I talked about this in 2009 a little bit here - see slide 22, we call these Defcon Levels. http://www.slideshare.net/supertom/building-configurable-applications-for-the-web The idea is prioritizing what REALLY is important if the shit hits the fan (watch out, biz folks think everything is always important) and having processes to implemen and knobs to turn and levers to pull should you get slashdotted (or facebooked, tweeted, oprahed, techcrunched or whatever we call it these days). Good luck with your launch. Thanks, Tom You should always worry about everything, but you should also have confidence in your decisions. If your worry is how your cluster will perform under load, then you should find a way to test under load. Tweeks and tunes do not make scalability (they help), hardware does. If you want to be ready to be 'slashdotted' you better have a rack of servers idling. If you just need a key-value store you may not need Cassandra. Cassandra is scalable in a different way then MySQL would be. You want convincing... (Ill try) Cassandra shards through node joins and handles replication. If you start off with a Mysql master/slave architecture, or using id hash(key) mod 3. It is not clear how you grow that cluster with demand. If you make a choice that is not scalable, when you get 'slashdotted' you will not be ready. What is worse you will have no easy way out of the problem.
Re: Taking down a node in a 3-node cluster, RF=2
OK. To sum up: RF=2 and QUORUM are incompatible (if you want to be able to take a node down). Right? On Sun, Nov 28, 2010 at 7:59 PM, Jake Luciani jak...@gmail.com wrote: I was wrong on this scenario and I'll explain where I was incorrect. Hints are stored for a downed node but they don't count towards meeting a consistency level. Let's take 2 scenarios: RF=6, Nodes=10 If you READ/WRITE with CL.QUORUM you will need 4 alive nodes if one is down it will still have 4 active replicas to write to, one of these will store a hint and update the downed node when it comes back. RF=2, Nodes=3 If you READ/WRITE with CL.QUORUM you need 2 live nodes. If one of these 2 are down you can't meet the QUORUM level so the write will fail. In your scenario your best bet is to update to RF=3, then any two nodes will accept QUORUM Sorry for the confusion, -Jake On Sun, Nov 28, 2010 at 12:26 PM, David Boxenhorn da...@lookin2.comwrote: Thank you, Jake. It does... except that in another context you told me: Hints only happen when a node is unavailable and you are writing with CL.ANY If you never write with CL.ANY then you can turn off hinted handoff. How do I reconcile this? On Sun, Nov 28, 2010 at 7:11 PM, Jake Luciani jak...@gmail.com wrote: If you read/write data with quorum then you can safely take a node down in this scenario. Subsequent writes will use hinted handoff to be passed to the node when it comes back up. More info is here: http://wiki.apache.org/cassandra/HintedHandoff Does that answer your question? -Jake On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory ran...@gmail.com wrote: to me it makes sense that if hinted handoff is off then cassandra cannot satisfy 2 out of every 3rd writes writes when one of the nodes is down since this node is the designated node of 2/3 writes. But I don't remember reading this somewhere. Does hinted handoff affect David's situation? (David, did you disable HH in your storage-config? HintedHandoffEnabledfalse/HintedHandoffEnabled) On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn da...@lookin2.comwrote: For the vast majority of my data usage eventual consistency is fine (i.e. CL=ONE) but I have a small amount of critical data for which I read and write using CL=QUORUM. If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean that a value can be read from or written to any 2 nodes, or does it have to be the particular 2 nodes that store the data? If it is the particular 2 nodes that store the data, that means that I can't even take down one node, since it will be the mandatory 2nd node for 1/3 of my data... -- /Ran
Re: Taking down a node in a 3-node cluster, RF=2
Right. On Sun, Nov 28, 2010 at 1:03 PM, David Boxenhorn da...@lookin2.com wrote: OK. To sum up: RF=2 and QUORUM are incompatible (if you want to be able to take a node down). Right? On Sun, Nov 28, 2010 at 7:59 PM, Jake Luciani jak...@gmail.com wrote: I was wrong on this scenario and I'll explain where I was incorrect. Hints are stored for a downed node but they don't count towards meeting a consistency level. Let's take 2 scenarios: RF=6, Nodes=10 If you READ/WRITE with CL.QUORUM you will need 4 alive nodes if one is down it will still have 4 active replicas to write to, one of these will store a hint and update the downed node when it comes back. RF=2, Nodes=3 If you READ/WRITE with CL.QUORUM you need 2 live nodes. If one of these 2 are down you can't meet the QUORUM level so the write will fail. In your scenario your best bet is to update to RF=3, then any two nodes will accept QUORUM Sorry for the confusion, -Jake On Sun, Nov 28, 2010 at 12:26 PM, David Boxenhorn da...@lookin2.comwrote: Thank you, Jake. It does... except that in another context you told me: Hints only happen when a node is unavailable and you are writing with CL.ANY If you never write with CL.ANY then you can turn off hinted handoff. How do I reconcile this? On Sun, Nov 28, 2010 at 7:11 PM, Jake Luciani jak...@gmail.com wrote: If you read/write data with quorum then you can safely take a node down in this scenario. Subsequent writes will use hinted handoff to be passed to the node when it comes back up. More info is here: http://wiki.apache.org/cassandra/HintedHandoff Does that answer your question? -Jake On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory ran...@gmail.com wrote: to me it makes sense that if hinted handoff is off then cassandra cannot satisfy 2 out of every 3rd writes writes when one of the nodes is down since this node is the designated node of 2/3 writes. But I don't remember reading this somewhere. Does hinted handoff affect David's situation? (David, did you disable HH in your storage-config? HintedHandoffEnabledfalse/HintedHandoffEnabled) On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn da...@lookin2.comwrote: For the vast majority of my data usage eventual consistency is fine (i.e. CL=ONE) but I have a small amount of critical data for which I read and write using CL=QUORUM. If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean that a value can be read from or written to any 2 nodes, or does it have to be the particular 2 nodes that store the data? If it is the particular 2 nodes that store the data, that means that I can't even take down one node, since it will be the mandatory 2nd node for 1/3 of my data... -- /Ran
Re: Issues getting JNA to work correctly under centos 5.5 using cassandra 0.7.0-rc1 and JNA 2.7.3
I'm not familiar with ulimit on RedHat systems, but are you sure you have ulimit set correctly? Did you set it to '0' or 'unlimited'? I ask because on a Debian system, I get this: tho...@~ $ ulimit -l unlimited Where you said that you got back '0'. - Tyler On Sun, Nov 28, 2010 at 1:15 AM, Jason Pell ja...@pellcorp.com wrote: Hi, I have selinux disabled via /etc/sysconfig/selinux already. But I did as you suggested anyway, even restarted the whole machine again too and still no difference. Do you know if there is a way to discover exactly what this error means? THanks Jason On Sat, Nov 27, 2010 at 3:59 AM, Nate McCall n...@riptano.com wrote: This might be an issue with selinux. You can try this quickly to temporarily disable selinux enforcement: /usr/sbin/setenforce 0 (as root) and then start cassandra as your user. On Fri, Nov 26, 2010 at 1:00 AM, Jason Pell jasonmp...@gmail.com wrote: I restarted the box :-) so it's well and truly set Sent from my iPhone On Nov 26, 2010, at 17:57, Brandon Williams dri...@gmail.com wrote: On Thu, Nov 25, 2010 at 10:02 PM, Jason Pell ja...@pellcorp.com wrote: Hi, I have set the memlock limit to unlimited in /etc/security/limits.conf [devel...@localhost apache-cassandra-0.7.0-rc1]$ ulimit -l 0 Running as a non root user gets me a Unknown mlockall error 1 Have you tried logging out and back in after changing limits.conf? -Brandon
Re: Issues getting JNA to work correctly under centos 5.5 using cassandra 0.7.0-rc1 and JNA 2.7.3
Ok that's a good point i will check - I am not sure. Sent from my iPhone On Nov 29, 2010, at 5:53, Tyler Hobbs ty...@riptano.com wrote: I'm not familiar with ulimit on RedHat systems, but are you sure you have ulimit set correctly? Did you set it to '0' or 'unlimited'? I ask because on a Debian system, I get this: tho...@~ $ ulimit -l unlimited Where you said that you got back '0'. - Tyler On Sun, Nov 28, 2010 at 1:15 AM, Jason Pell ja...@pellcorp.com wrote: Hi, I have selinux disabled via /etc/sysconfig/selinux already. But I did as you suggested anyway, even restarted the whole machine again too and still no difference. Do you know if there is a way to discover exactly what this error means? THanks Jason On Sat, Nov 27, 2010 at 3:59 AM, Nate McCall n...@riptano.com wrote: This might be an issue with selinux. You can try this quickly to temporarily disable selinux enforcement: /usr/sbin/setenforce 0 (as root) and then start cassandra as your user. On Fri, Nov 26, 2010 at 1:00 AM, Jason Pell jasonmp...@gmail.com wrote: I restarted the box :-) so it's well and truly set Sent from my iPhone On Nov 26, 2010, at 17:57, Brandon Williams dri...@gmail.com wrote: On Thu, Nov 25, 2010 at 10:02 PM, Jason Pell ja...@pellcorp.com wrote: Hi, I have set the memlock limit to unlimited in /etc/security/limits.conf [devel...@localhost apache-cassandra-0.7.0-rc1]$ ulimit -l 0 Running as a non root user gets me a Unknown mlockall error 1 Have you tried logging out and back in after changing limits.conf? -Brandon
Re: Taking down a node in a 3-node cluster, RF=2
Put another way, for RF=2, QUORUM is equivalent to ALL. On Sun, Nov 28, 2010 at 12:03 PM, David Boxenhorn da...@lookin2.com wrote: OK. To sum up: RF=2 and QUORUM are incompatible (if you want to be able to take a node down). Right? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
