Re: horizontal query scaling issues follow on
different client levels for the smaller cluster you may see improved performance as the data is pulled into file cache across test runs, and then when you build your larger cluster this is lost so performance appears to degrade (for instance). On Fri, Jul 18, 2014 at 12:25 PM, Diane Griffith dfgriff...@gmail.com wrote: The column family schema is: CREATE TABLE IF NOT EXISTS foo (key text, col_name text, col_value text, PRIMARY KEY(key, col_name)) where the key is a generated uuid and all keys were inserted in random order but in the end we were compacting down to one sstable per node. So we were doing it this way to achieve dynamic columns. Thanks, Diane On Fri, Jul 18, 2014 at 12:19 AM, Jack Krupansky j...@basetechnology.com wrote: Sorry I may have confused the discussion by mentioning tokens – I wasn’t intending to refer to vnodes or the num_tokens property, but merely referring to the token range of a node and that the partition key hashes to a token value. The main question is what you use for your primary key and whether you are using a small number of partition keys and a large number of clustering columns, or does each row have a unique partition key and no clustering columns. -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 6:21 PM *To:* user user@cassandra.apache.org *Subject:* Re: horizontal query scaling issues follow on So do partitions equate to tokens/vnodes? If so we had configured all cluster nodes/vms with num_tokens: 256 instead of setting init_token and assigning ranges. I am still not getting why in Cassandra 2.0, I would assign my own ranges via init_token and this was based on the documentation and even this blog item http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 that made it seem right for us to always configure our cluster vms with num_tokens: 256 in the cassandra.yaml file. Also in all testing, all vms were of equal sizing so one was not more powerful than another. I didn't think I was hitting an i/o wall on the client vm (separate vm) where we command line scripted our query call to the cassandra cluster. I can break the client call load across vms which I tried early on. Happy to verify that again though. So given that I was assuming the partitions were such that it wasn't a problem. Is that an incorrect assumption and something to dig into more? Thanks, Diane On Thu, Jul 17, 2014 at 3:01 PM, Jack Krupansky j...@basetechnology.com wrote: How many partitions are you spreading those 18 million rows over? That many rows in a single partition will not be a sweet spot for Cassandra. It’s not exceeding any hard limit (2 billion), but some internal operations may cache the partition rather than the logical row. And all those rows in a single partition would certainly not be a test of “horizontal scaling” (adding nodes to handle more data – more token values or partitions.) -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 1:33 PM *To:* user user@cassandra.apache.org *Subject:* horizontal query scaling issues follow on This is a follow on re-post to clarify what we are trying to do, providing information that was missing or not clear. Goal: Verify horizontal scaling for random non duplicating key reads using the simplest configuration (or minimal configuration) possible. Background: A couple years ago we did similar performance testing with Cassandra for both read and write performance and found excellent (essentially linear) horizontal scalability. That project got put on hold. We are now moving forward with an operational system and are having scaling problems. During the prior testing (3 years ago) we were using a much older version of Cassandra (0.8 or older), the THRIFT API, and Amazon AWS rather than OpenStack VMs. We are now using the latest Cassandra and the CQL interface. We did try moving from OpenStack to AWS/EC2 but that did not materially change our (poor) results. Test Procedure: - Inserted 54 million cells in 18 million rows (so 3 cells per row), using randomly generated row keys. That was to be our data control for the test. - Spawn a client on a different VM to query 100k rows and do that for 100 reps. Each row key queried is drawn randomly from the set of existing row keys, and then not re-used, so all 10 million row queries use a different (valid) row key. This test is a specific use case of our system we are trying to show will scale Result: - 2 nodes performed better than 1 node test but 4 nodes showed decreased performance over 2 nodes. So that did not show horizontal scaling Notes: - We have replication factor set to 1 as we were trying to keep the control test simple to prove out horizontal scaling. - When we tried to add threading to see if it would help it had
Re: horizontal query scaling issues follow on
(*) from foo will report if I add the limit command to let it scan all rows. Does anything seem like it is hurting our chances to horizontally scale with the data/schema? Thanks, Diane On Fri, Jul 18, 2014 at 6:46 AM, Benedict Elliott Smith belliottsm...@datastax.com wrote: How many columns are you inserting/querying per key? Could we see some example CQL statements for the insert/read workload? If you are maxing out at 10 clients, something fishy is going on. In general, though, if you find that adding nodes causes performance to degrade I would suspect that you are querying data in one CQL statement that is spread over multiple partitions, and so extra work needs to be done cross-cluster to service your requests as more nodes are added. I would also consider what effect the file cache may be having on your workload, as it sounds small enough to fit in memory, so is likely a major determining factor for performance of your benchmark. As you try different client levels for the smaller cluster you may see improved performance as the data is pulled into file cache across test runs, and then when you build your larger cluster this is lost so performance appears to degrade (for instance). On Fri, Jul 18, 2014 at 12:25 PM, Diane Griffith dfgriff...@gmail.com wrote: The column family schema is: CREATE TABLE IF NOT EXISTS foo (key text, col_name text, col_value text, PRIMARY KEY(key, col_name)) where the key is a generated uuid and all keys were inserted in random order but in the end we were compacting down to one sstable per node. So we were doing it this way to achieve dynamic columns. Thanks, Diane On Fri, Jul 18, 2014 at 12:19 AM, Jack Krupansky j...@basetechnology.com wrote: Sorry I may have confused the discussion by mentioning tokens – I wasn’t intending to refer to vnodes or the num_tokens property, but merely referring to the token range of a node and that the partition key hashes to a token value. The main question is what you use for your primary key and whether you are using a small number of partition keys and a large number of clustering columns, or does each row have a unique partition key and no clustering columns. -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 6:21 PM *To:* user user@cassandra.apache.org *Subject:* Re: horizontal query scaling issues follow on So do partitions equate to tokens/vnodes? If so we had configured all cluster nodes/vms with num_tokens: 256 instead of setting init_token and assigning ranges. I am still not getting why in Cassandra 2.0, I would assign my own ranges via init_token and this was based on the documentation and even this blog item http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 that made it seem right for us to always configure our cluster vms with num_tokens: 256 in the cassandra.yaml file. Also in all testing, all vms were of equal sizing so one was not more powerful than another. I didn't think I was hitting an i/o wall on the client vm (separate vm) where we command line scripted our query call to the cassandra cluster.I can break the client call load across vms which I tried early on. Happy to verify that again though. So given that I was assuming the partitions were such that it wasn't a problem. Is that an incorrect assumption and something to dig into more? Thanks, Diane On Thu, Jul 17, 2014 at 3:01 PM, Jack Krupansky j...@basetechnology.com wrote: How many partitions are you spreading those 18 million rows over? That many rows in a single partition will not be a sweet spot for Cassandra. It’s not exceeding any hard limit (2 billion), but some internal operations may cache the partition rather than the logical row. And all those rows in a single partition would certainly not be a test of “horizontal scaling” (adding nodes to handle more data – more token values or partitions.) -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 1:33 PM *To:* user user@cassandra.apache.org *Subject:* horizontal query scaling issues follow on This is a follow on re-post to clarify what we are trying to do, providing information that was missing or not clear. Goal: Verify horizontal scaling for random non duplicating key reads using the simplest configuration (or minimal configuration) possible. Background: A couple years ago we did similar performance testing with Cassandra for both read and write performance and found excellent (essentially linear) horizontal scalability. That project got put on hold. We are now moving forward with an operational system and are having scaling problems. During the prior testing (3 years ago) we were using a much older version of Cassandra (0.8 or older), the THRIFT API, and Amazon AWS rather than OpenStack VMs. We are now using the latest Cassandra and the CQL
Re: horizontal query scaling issues follow on
Hello, Here is the documentation for cfhistograms, which is in microseconds. http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsCFhisto.html Your question about setting timeouts is subjective, but you have set your timeout limits to 4 mins, which seems excessive. The default timeout values should be appropriate for a well sized and operating cluster. Increasing timeouts to achieve stability isn't a recommended practice. You're VMs are undersized, and therefore, it is recommended that you reduce your workload or add nodes until stability is achieved. The goal of your exersize is to prove out linear scalability, correct? Then it is recommended to find the load your small nodes/cluster can handle without increasing timeout values, i.e. your cluster can remain stable. Once you found the sweet spot for load on your cluster, increase load by X% while increasing cluster size by X%. Do this for a few iterations so you can see that the processing capabilities of your cluster increases proportionally, and linearly, to the amount of load you are putting on your cluster. Note, with small VM's, you will not receive production-like performance from individual nodes. Also, what type of storage do you have under the VMs? It's not recommended to leverage shared storage. Leveraging shared storage will, more than likely, not allow you to achieve linear scalability. This is because your hardware will not be scaling linearly fully through the stack. Hope this helps Jonathan On Sun, Jul 20, 2014 at 9:12 PM, Diane Griffith dfgriff...@gmail.com wrote: I am running tests again across different number of client threads and number of nodes but this time I tweaked some of the timeouts configured for the nodes in the cluster. I was able to get better performance on the nodes at 10 client threads by upping 4 timeout values in cassandra.yaml to 24: - read_request_timeout_in_ms - range_request_timeout_in_ms - write_request_timeout_in_ms - request_timeout_in_ms I did this because of my interpretation of the cfhistograms output on one of the nodes. So 3 questions that come to mind: 1. Did I interpret the histogram information correctly in cassandra 2.0.6 nodetool output? That the 2 column read latency output is the offset or left column is the time in milliseconds and the right column is number of requests that fell into that bucket range. 2. Was it reasonable for me to boost those 4 timeouts and just those? 3. What are reasonable timeout values for smaller vm sizes (i.e. 8GB RAM, 4 CPUs)? If anyone has any insight it would be appreciated. Thanks, Diane On Fri, Jul 18, 2014 at 2:23 PM, Tyler Hobbs ty...@datastax.com wrote: On Fri, Jul 18, 2014 at 8:01 AM, Diane Griffith dfgriff...@gmail.com wrote: Partition Size (bytes) 1109 bytes: 1800 Cell Count per Partition 8 cells: 1800 meaning I can't glean anything about how it partitioned or if it broke a key across partitions from this right? Does it mean for 1800 (the number of unique keys) that each has 8 cells? Yes, your interpretation is correct. Each of your 1800 partitions has 8 cells (taking up 1109 bytes). -- Tyler Hobbs DataStax http://datastax.com/ -- Jonathan Lacefield Solutions Architect, DataStax (404) 822 3487 http://www.linkedin.com/in/jlacefield http://www.datastax.com/cassandrasummit14
Re: horizontal query scaling issues follow on
So I appreciate all the help so far. Upfront, it is possible the schema and data query pattern could be contributing to the problem. The schema was born out of certain design requirements. If it proves to be part of what makes the scalability crumble, then I hope it will help shape the design requirements. Anyway, the premise of the question was my struggle where scalability metrics fell apart going from 2 nodes to 4 nodes for the current schema and query access pattern being modeled: - 1 node was producing acceptable response times seemed to be the consensus - 2 nodes showed marked improvement to the response times for the query scenario being modeled which was welcomed news - 4 nodes showed a decrease in performance and it was not clear why going 2 to 4 nodes triggered the decrease Also what contributed to the question was 2 more items: - cassandra-env.sh - where in the example for HEAP_NEWSIZE states in the comments it assumes a modern 8 core machine for pause times - a wiki article I had found and I am trying to relocate where a person set up very small nodes for developers on that team and talked through all the paramters that had to be changed from the default to get good throughput. It sort of implied the defaults maybe were based on a certain sized vm. That was the main driver for those questions. I agree it does not seem correct to boost the values let alone so high to minimize impact in some respects (i.e. not trigger the reads to time out and start over given the retry policy). So the question really was are the defaults sized with the assumption of a certain minimal vm size (i.e. the comment in cassandra-env.sh) Does that explain where I am coming from better? My question, despite being naive and ignoring other impacts still stands, is there a minimal vm size that is more of the sweet spot for cassandra and the defaults. I get the point that a column family schema as it relates to the desired queries can and do impact that answer. I guess what bothered me was it didn't impact that answer going from 1 node to 2 nodes but started showing up going from 2 nodes to 4 nodes. I'm building whatever facts I can to support the schema and query pattern scales or does not. If it does not, then I am trying to pull information from some metrics outputted by nodetool or log statements on the cassandra log files to support a case to change the design requirements. Thanks, Diane On Mon, Jul 21, 2014 at 8:15 PM, Robert Coli rc...@eventbrite.com wrote: On Sun, Jul 20, 2014 at 6:12 PM, Diane Griffith dfgriff...@gmail.com wrote: I am running tests again across different number of client threads and number of nodes but this time I tweaked some of the timeouts configured for the nodes in the cluster. I was able to get better performance on the nodes at 10 client threads by upping 4 timeout values in cassandra.yaml to 24: If you have to tune these timeout values, you have probably modeled data in such a way that each of your requests is quite large or quite slow. This is usually, but not always, an indicator that you are Doing It Wrong. Massively multithreaded things don't generally like their threads to be long-lived, for what should hopefully be obvious reasons. I did this because of my interpretation of the cfhistograms output on one of the nodes. Could you be more specific? So 3 questions that come to mind: 1. Did I interpret the histogram information correctly in cassandra 2.0.6 nodetool output? That the 2 column read latency output is the offset or left column is the time in milliseconds and the right column is number of requests that fell into that bucket range. 2. Was it reasonable for me to boost those 4 timeouts and just those? Not really. In 5 years of operating Cassandra, I've never had a problem whose solution was to increase these timeouts from their default. 1. What are reasonable timeout values for smaller vm sizes (i.e. 8GB RAM, 4 CPUs)? As above, I question the premise of this question. =Rob
Re: horizontal query scaling issues follow on
I am running tests again across different number of client threads and number of nodes but this time I tweaked some of the timeouts configured for the nodes in the cluster. I was able to get better performance on the nodes at 10 client threads by upping 4 timeout values in cassandra.yaml to 24: - read_request_timeout_in_ms - range_request_timeout_in_ms - write_request_timeout_in_ms - request_timeout_in_ms I did this because of my interpretation of the cfhistograms output on one of the nodes. So 3 questions that come to mind: 1. Did I interpret the histogram information correctly in cassandra 2.0.6 nodetool output? That the 2 column read latency output is the offset or left column is the time in milliseconds and the right column is number of requests that fell into that bucket range. 2. Was it reasonable for me to boost those 4 timeouts and just those? 3. What are reasonable timeout values for smaller vm sizes (i.e. 8GB RAM, 4 CPUs)? If anyone has any insight it would be appreciated. Thanks, Diane On Fri, Jul 18, 2014 at 2:23 PM, Tyler Hobbs ty...@datastax.com wrote: On Fri, Jul 18, 2014 at 8:01 AM, Diane Griffith dfgriff...@gmail.com wrote: Partition Size (bytes) 1109 bytes: 1800 Cell Count per Partition 8 cells: 1800 meaning I can't glean anything about how it partitioned or if it broke a key across partitions from this right? Does it mean for 1800 (the number of unique keys) that each has 8 cells? Yes, your interpretation is correct. Each of your 1800 partitions has 8 cells (taking up 1109 bytes). -- Tyler Hobbs DataStax http://datastax.com/
Re: horizontal query scaling issues follow on
The column family schema is: CREATE TABLE IF NOT EXISTS foo (key text, col_name text, col_value text, PRIMARY KEY(key, col_name)) where the key is a generated uuid and all keys were inserted in random order but in the end we were compacting down to one sstable per node. So we were doing it this way to achieve dynamic columns. Thanks, Diane On Fri, Jul 18, 2014 at 12:19 AM, Jack Krupansky j...@basetechnology.com wrote: Sorry I may have confused the discussion by mentioning tokens – I wasn’t intending to refer to vnodes or the num_tokens property, but merely referring to the token range of a node and that the partition key hashes to a token value. The main question is what you use for your primary key and whether you are using a small number of partition keys and a large number of clustering columns, or does each row have a unique partition key and no clustering columns. -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 6:21 PM *To:* user user@cassandra.apache.org *Subject:* Re: horizontal query scaling issues follow on So do partitions equate to tokens/vnodes? If so we had configured all cluster nodes/vms with num_tokens: 256 instead of setting init_token and assigning ranges. I am still not getting why in Cassandra 2.0, I would assign my own ranges via init_token and this was based on the documentation and even this blog item http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 that made it seem right for us to always configure our cluster vms with num_tokens: 256 in the cassandra.yaml file. Also in all testing, all vms were of equal sizing so one was not more powerful than another. I didn't think I was hitting an i/o wall on the client vm (separate vm) where we command line scripted our query call to the cassandra cluster. I can break the client call load across vms which I tried early on. Happy to verify that again though. So given that I was assuming the partitions were such that it wasn't a problem. Is that an incorrect assumption and something to dig into more? Thanks, Diane On Thu, Jul 17, 2014 at 3:01 PM, Jack Krupansky j...@basetechnology.com wrote: How many partitions are you spreading those 18 million rows over? That many rows in a single partition will not be a sweet spot for Cassandra. It’s not exceeding any hard limit (2 billion), but some internal operations may cache the partition rather than the logical row. And all those rows in a single partition would certainly not be a test of “horizontal scaling” (adding nodes to handle more data – more token values or partitions.) -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 1:33 PM *To:* user user@cassandra.apache.org *Subject:* horizontal query scaling issues follow on This is a follow on re-post to clarify what we are trying to do, providing information that was missing or not clear. Goal: Verify horizontal scaling for random non duplicating key reads using the simplest configuration (or minimal configuration) possible. Background: A couple years ago we did similar performance testing with Cassandra for both read and write performance and found excellent (essentially linear) horizontal scalability. That project got put on hold. We are now moving forward with an operational system and are having scaling problems. During the prior testing (3 years ago) we were using a much older version of Cassandra (0.8 or older), the THRIFT API, and Amazon AWS rather than OpenStack VMs. We are now using the latest Cassandra and the CQL interface. We did try moving from OpenStack to AWS/EC2 but that did not materially change our (poor) results. Test Procedure: - Inserted 54 million cells in 18 million rows (so 3 cells per row), using randomly generated row keys. That was to be our data control for the test. - Spawn a client on a different VM to query 100k rows and do that for 100 reps. Each row key queried is drawn randomly from the set of existing row keys, and then not re-used, so all 10 million row queries use a different (valid) row key. This test is a specific use case of our system we are trying to show will scale Result: - 2 nodes performed better than 1 node test but 4 nodes showed decreased performance over 2 nodes. So that did not show horizontal scaling Notes: - We have replication factor set to 1 as we were trying to keep the control test simple to prove out horizontal scaling. - When we tried to add threading to see if it would help it had interesting side behavior which did not prove out horizontal scaling. - We are using CQL versus THRIFT API for Cassandra 2.0.6 Does anyone have any feedback that either threading or replication factor is necessary to show horizontal scaling of Cassandra versus the minimal way of just continue to add nodes to help throughput
Re: horizontal query scaling issues follow on
How many columns are you inserting/querying per key? Could we see some example CQL statements for the insert/read workload? If you are maxing out at 10 clients, something fishy is going on. In general, though, if you find that adding nodes causes performance to degrade I would suspect that you are querying data in one CQL statement that is spread over multiple partitions, and so extra work needs to be done cross-cluster to service your requests as more nodes are added. I would also consider what effect the file cache may be having on your workload, as it sounds small enough to fit in memory, so is likely a major determining factor for performance of your benchmark. As you try different client levels for the smaller cluster you may see improved performance as the data is pulled into file cache across test runs, and then when you build your larger cluster this is lost so performance appears to degrade (for instance). On Fri, Jul 18, 2014 at 12:25 PM, Diane Griffith dfgriff...@gmail.com wrote: The column family schema is: CREATE TABLE IF NOT EXISTS foo (key text, col_name text, col_value text, PRIMARY KEY(key, col_name)) where the key is a generated uuid and all keys were inserted in random order but in the end we were compacting down to one sstable per node. So we were doing it this way to achieve dynamic columns. Thanks, Diane On Fri, Jul 18, 2014 at 12:19 AM, Jack Krupansky j...@basetechnology.com wrote: Sorry I may have confused the discussion by mentioning tokens – I wasn’t intending to refer to vnodes or the num_tokens property, but merely referring to the token range of a node and that the partition key hashes to a token value. The main question is what you use for your primary key and whether you are using a small number of partition keys and a large number of clustering columns, or does each row have a unique partition key and no clustering columns. -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 6:21 PM *To:* user user@cassandra.apache.org *Subject:* Re: horizontal query scaling issues follow on So do partitions equate to tokens/vnodes? If so we had configured all cluster nodes/vms with num_tokens: 256 instead of setting init_token and assigning ranges. I am still not getting why in Cassandra 2.0, I would assign my own ranges via init_token and this was based on the documentation and even this blog item http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 that made it seem right for us to always configure our cluster vms with num_tokens: 256 in the cassandra.yaml file. Also in all testing, all vms were of equal sizing so one was not more powerful than another. I didn't think I was hitting an i/o wall on the client vm (separate vm) where we command line scripted our query call to the cassandra cluster. I can break the client call load across vms which I tried early on. Happy to verify that again though. So given that I was assuming the partitions were such that it wasn't a problem. Is that an incorrect assumption and something to dig into more? Thanks, Diane On Thu, Jul 17, 2014 at 3:01 PM, Jack Krupansky j...@basetechnology.com wrote: How many partitions are you spreading those 18 million rows over? That many rows in a single partition will not be a sweet spot for Cassandra. It’s not exceeding any hard limit (2 billion), but some internal operations may cache the partition rather than the logical row. And all those rows in a single partition would certainly not be a test of “horizontal scaling” (adding nodes to handle more data – more token values or partitions.) -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 1:33 PM *To:* user user@cassandra.apache.org *Subject:* horizontal query scaling issues follow on This is a follow on re-post to clarify what we are trying to do, providing information that was missing or not clear. Goal: Verify horizontal scaling for random non duplicating key reads using the simplest configuration (or minimal configuration) possible. Background: A couple years ago we did similar performance testing with Cassandra for both read and write performance and found excellent (essentially linear) horizontal scalability. That project got put on hold. We are now moving forward with an operational system and are having scaling problems. During the prior testing (3 years ago) we were using a much older version of Cassandra (0.8 or older), the THRIFT API, and Amazon AWS rather than OpenStack VMs. We are now using the latest Cassandra and the CQL interface. We did try moving from OpenStack to AWS/EC2 but that did not materially change our (poor) results. Test Procedure: - Inserted 54 million cells in 18 million rows (so 3 cells per row), using randomly generated row keys. That was to be our data control for the test. - Spawn a client
Re: horizontal query scaling issues follow on
Working on getting some samples but grabbed the last part of the nodetool cfhistograms for one of the column families on one of the nodes. What does it mean for the partition information: Partition Size (bytes) 1109 bytes: 1800 Cell Count per Partition 8 cells: 1800 meaning I can't glean anything about how it partitioned or if it broke a key across partitions from this right? Does it mean for 1800 (the number of unique keys) that each has 8 cells? Thanks, Diane On Fri, Jul 18, 2014 at 6:46 AM, Benedict Elliott Smith belliottsm...@datastax.com wrote: How many columns are you inserting/querying per key? Could we see some example CQL statements for the insert/read workload? If you are maxing out at 10 clients, something fishy is going on. In general, though, if you find that adding nodes causes performance to degrade I would suspect that you are querying data in one CQL statement that is spread over multiple partitions, and so extra work needs to be done cross-cluster to service your requests as more nodes are added. I would also consider what effect the file cache may be having on your workload, as it sounds small enough to fit in memory, so is likely a major determining factor for performance of your benchmark. As you try different client levels for the smaller cluster you may see improved performance as the data is pulled into file cache across test runs, and then when you build your larger cluster this is lost so performance appears to degrade (for instance). On Fri, Jul 18, 2014 at 12:25 PM, Diane Griffith dfgriff...@gmail.com wrote: The column family schema is: CREATE TABLE IF NOT EXISTS foo (key text, col_name text, col_value text, PRIMARY KEY(key, col_name)) where the key is a generated uuid and all keys were inserted in random order but in the end we were compacting down to one sstable per node. So we were doing it this way to achieve dynamic columns. Thanks, Diane On Fri, Jul 18, 2014 at 12:19 AM, Jack Krupansky j...@basetechnology.com wrote: Sorry I may have confused the discussion by mentioning tokens – I wasn’t intending to refer to vnodes or the num_tokens property, but merely referring to the token range of a node and that the partition key hashes to a token value. The main question is what you use for your primary key and whether you are using a small number of partition keys and a large number of clustering columns, or does each row have a unique partition key and no clustering columns. -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 6:21 PM *To:* user user@cassandra.apache.org *Subject:* Re: horizontal query scaling issues follow on So do partitions equate to tokens/vnodes? If so we had configured all cluster nodes/vms with num_tokens: 256 instead of setting init_token and assigning ranges. I am still not getting why in Cassandra 2.0, I would assign my own ranges via init_token and this was based on the documentation and even this blog item http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 that made it seem right for us to always configure our cluster vms with num_tokens: 256 in the cassandra.yaml file. Also in all testing, all vms were of equal sizing so one was not more powerful than another. I didn't think I was hitting an i/o wall on the client vm (separate vm) where we command line scripted our query call to the cassandra cluster. I can break the client call load across vms which I tried early on. Happy to verify that again though. So given that I was assuming the partitions were such that it wasn't a problem. Is that an incorrect assumption and something to dig into more? Thanks, Diane On Thu, Jul 17, 2014 at 3:01 PM, Jack Krupansky j...@basetechnology.com wrote: How many partitions are you spreading those 18 million rows over? That many rows in a single partition will not be a sweet spot for Cassandra. It’s not exceeding any hard limit (2 billion), but some internal operations may cache the partition rather than the logical row. And all those rows in a single partition would certainly not be a test of “horizontal scaling” (adding nodes to handle more data – more token values or partitions.) -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 1:33 PM *To:* user user@cassandra.apache.org *Subject:* horizontal query scaling issues follow on This is a follow on re-post to clarify what we are trying to do, providing information that was missing or not clear. Goal: Verify horizontal scaling for random non duplicating key reads using the simplest configuration (or minimal configuration) possible. Background: A couple years ago we did similar performance testing with Cassandra for both read and write performance and found excellent (essentially linear) horizontal scalability. That project got put on hold. We are now moving forward
Re: horizontal query scaling issues follow on
On Fri, Jul 18, 2014 at 8:01 AM, Diane Griffith dfgriff...@gmail.com wrote: Partition Size (bytes) 1109 bytes: 1800 Cell Count per Partition 8 cells: 1800 meaning I can't glean anything about how it partitioned or if it broke a key across partitions from this right? Does it mean for 1800 (the number of unique keys) that each has 8 cells? Yes, your interpretation is correct. Each of your 1800 partitions has 8 cells (taking up 1109 bytes). -- Tyler Hobbs DataStax http://datastax.com/
Re: horizontal query scaling issues follow on
primary key and whether you are using a small number of partition keys and a large number of clustering columns, or does each row have a unique partition key and no clustering columns. -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 6:21 PM *To:* user user@cassandra.apache.org *Subject:* Re: horizontal query scaling issues follow on So do partitions equate to tokens/vnodes? If so we had configured all cluster nodes/vms with num_tokens: 256 instead of setting init_token and assigning ranges. I am still not getting why in Cassandra 2.0, I would assign my own ranges via init_token and this was based on the documentation and even this blog item http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 that made it seem right for us to always configure our cluster vms with num_tokens: 256 in the cassandra.yaml file. Also in all testing, all vms were of equal sizing so one was not more powerful than another. I didn't think I was hitting an i/o wall on the client vm (separate vm) where we command line scripted our query call to the cassandra cluster. I can break the client call load across vms which I tried early on. Happy to verify that again though. So given that I was assuming the partitions were such that it wasn't a problem. Is that an incorrect assumption and something to dig into more? Thanks, Diane On Thu, Jul 17, 2014 at 3:01 PM, Jack Krupansky j...@basetechnology.com wrote: How many partitions are you spreading those 18 million rows over? That many rows in a single partition will not be a sweet spot for Cassandra. It’s not exceeding any hard limit (2 billion), but some internal operations may cache the partition rather than the logical row. And all those rows in a single partition would certainly not be a test of “horizontal scaling” (adding nodes to handle more data – more token values or partitions.) -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 1:33 PM *To:* user user@cassandra.apache.org *Subject:* horizontal query scaling issues follow on This is a follow on re-post to clarify what we are trying to do, providing information that was missing or not clear. Goal: Verify horizontal scaling for random non duplicating key reads using the simplest configuration (or minimal configuration) possible. Background: A couple years ago we did similar performance testing with Cassandra for both read and write performance and found excellent (essentially linear) horizontal scalability. That project got put on hold. We are now moving forward with an operational system and are having scaling problems. During the prior testing (3 years ago) we were using a much older version of Cassandra (0.8 or older), the THRIFT API, and Amazon AWS rather than OpenStack VMs. We are now using the latest Cassandra and the CQL interface. We did try moving from OpenStack to AWS/EC2 but that did not materially change our (poor) results. Test Procedure: - Inserted 54 million cells in 18 million rows (so 3 cells per row), using randomly generated row keys. That was to be our data control for the test. - Spawn a client on a different VM to query 100k rows and do that for 100 reps. Each row key queried is drawn randomly from the set of existing row keys, and then not re-used, so all 10 million row queries use a different (valid) row key. This test is a specific use case of our system we are trying to show will scale Result: - 2 nodes performed better than 1 node test but 4 nodes showed decreased performance over 2 nodes. So that did not show horizontal scaling Notes: - We have replication factor set to 1 as we were trying to keep the control test simple to prove out horizontal scaling. - When we tried to add threading to see if it would help it had interesting side behavior which did not prove out horizontal scaling. - We are using CQL versus THRIFT API for Cassandra 2.0.6 Does anyone have any feedback that either threading or replication factor is necessary to show horizontal scaling of Cassandra versus the minimal way of just continue to add nodes to help throughput? Any suggestions of minimal configuration necessary to show scaling of our query use case 100k requests for random non repeating keys constantly coming in over a period of time? Thanks, Diane
horizontal query scaling issues follow on
This is a follow on re-post to clarify what we are trying to do, providing information that was missing or not clear. Goal: Verify horizontal scaling for random non duplicating key reads using the simplest configuration (or minimal configuration) possible. Background: A couple years ago we did similar performance testing with Cassandra for both read and write performance and found excellent (essentially linear) horizontal scalability. That project got put on hold. We are now moving forward with an operational system and are having scaling problems. During the prior testing (3 years ago) we were using a much older version of Cassandra (0.8 or older), the THRIFT API, and Amazon AWS rather than OpenStack VMs. We are now using the latest Cassandra and the CQL interface. We did try moving from OpenStack to AWS/EC2 but that did not materially change our (poor) results. Test Procedure: - Inserted 54 million cells in 18 million rows (so 3 cells per row), using randomly generated row keys. That was to be our data control for the test. - Spawn a client on a different VM to query 100k rows and do that for 100 reps. Each row key queried is drawn randomly from the set of existing row keys, and then not re-used, so all 10 million row queries use a different (valid) row key. This test is a specific use case of our system we are trying to show will scale Result: - 2 nodes performed better than 1 node test but 4 nodes showed decreased performance over 2 nodes. So that did not show horizontal scaling Notes: - We have replication factor set to 1 as we were trying to keep the control test simple to prove out horizontal scaling. - When we tried to add threading to see if it would help it had interesting side behavior which did not prove out horizontal scaling. - We are using CQL versus THRIFT API for Cassandra 2.0.6 Does anyone have any feedback that either threading or replication factor is necessary to show horizontal scaling of Cassandra versus the minimal way of just continue to add nodes to help throughput? Any suggestions of minimal configuration necessary to show scaling of our query use case 100k requests for random non repeating keys constantly coming in over a period of time? Thanks, Diane
Re: horizontal query scaling issues follow on
How many partitions are you spreading those 18 million rows over? That many rows in a single partition will not be a sweet spot for Cassandra. It’s not exceeding any hard limit (2 billion), but some internal operations may cache the partition rather than the logical row. And all those rows in a single partition would certainly not be a test of “horizontal scaling” (adding nodes to handle more data – more token values or partitions.) -- Jack Krupansky From: Diane Griffith Sent: Thursday, July 17, 2014 1:33 PM To: user Subject: horizontal query scaling issues follow on This is a follow on re-post to clarify what we are trying to do, providing information that was missing or not clear. Goal: Verify horizontal scaling for random non duplicating key reads using the simplest configuration (or minimal configuration) possible. Background: A couple years ago we did similar performance testing with Cassandra for both read and write performance and found excellent (essentially linear) horizontal scalability. That project got put on hold. We are now moving forward with an operational system and are having scaling problems. During the prior testing (3 years ago) we were using a much older version of Cassandra (0.8 or older), the THRIFT API, and Amazon AWS rather than OpenStack VMs. We are now using the latest Cassandra and the CQL interface. We did try moving from OpenStack to AWS/EC2 but that did not materially change our (poor) results. Test Procedure: a.. Inserted 54 million cells in 18 million rows (so 3 cells per row), using randomly generated row keys. That was to be our data control for the test. b.. Spawn a client on a different VM to query 100k rows and do that for 100 reps. Each row key queried is drawn randomly from the set of existing row keys, and then not re-used, so all 10 million row queries use a different (valid) row key. This test is a specific use case of our system we are trying to show will scale Result: a.. 2 nodes performed better than 1 node test but 4 nodes showed decreased performance over 2 nodes. So that did not show horizontal scaling Notes: a.. We have replication factor set to 1 as we were trying to keep the control test simple to prove out horizontal scaling. b.. When we tried to add threading to see if it would help it had interesting side behavior which did not prove out horizontal scaling. c.. We are using CQL versus THRIFT API for Cassandra 2.0.6 Does anyone have any feedback that either threading or replication factor is necessary to show horizontal scaling of Cassandra versus the minimal way of just continue to add nodes to help throughput? Any suggestions of minimal configuration necessary to show scaling of our query use case 100k requests for random non repeating keys constantly coming in over a period of time? Thanks, Diane
Re: horizontal query scaling issues follow on
So do partitions equate to tokens/vnodes? If so we had configured all cluster nodes/vms with num_tokens: 256 instead of setting init_token and assigning ranges. I am still not getting why in Cassandra 2.0, I would assign my own ranges via init_token and this was based on the documentation and even this blog item http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 that made it seem right for us to always configure our cluster vms with num_tokens: 256 in the cassandra.yaml file. Also in all testing, all vms were of equal sizing so one was not more powerful than another. I didn't think I was hitting an i/o wall on the client vm (separate vm) where we command line scripted our query call to the cassandra cluster. I can break the client call load across vms which I tried early on. Happy to verify that again though. So given that I was assuming the partitions were such that it wasn't a problem. Is that an incorrect assumption and something to dig into more? Thanks, Diane On Thu, Jul 17, 2014 at 3:01 PM, Jack Krupansky j...@basetechnology.com wrote: How many partitions are you spreading those 18 million rows over? That many rows in a single partition will not be a sweet spot for Cassandra. It’s not exceeding any hard limit (2 billion), but some internal operations may cache the partition rather than the logical row. And all those rows in a single partition would certainly not be a test of “horizontal scaling” (adding nodes to handle more data – more token values or partitions.) -- Jack Krupansky *From:* Diane Griffith dfgriff...@gmail.com *Sent:* Thursday, July 17, 2014 1:33 PM *To:* user user@cassandra.apache.org *Subject:* horizontal query scaling issues follow on This is a follow on re-post to clarify what we are trying to do, providing information that was missing or not clear. Goal: Verify horizontal scaling for random non duplicating key reads using the simplest configuration (or minimal configuration) possible. Background: A couple years ago we did similar performance testing with Cassandra for both read and write performance and found excellent (essentially linear) horizontal scalability. That project got put on hold. We are now moving forward with an operational system and are having scaling problems. During the prior testing (3 years ago) we were using a much older version of Cassandra (0.8 or older), the THRIFT API, and Amazon AWS rather than OpenStack VMs. We are now using the latest Cassandra and the CQL interface. We did try moving from OpenStack to AWS/EC2 but that did not materially change our (poor) results. Test Procedure: - Inserted 54 million cells in 18 million rows (so 3 cells per row), using randomly generated row keys. That was to be our data control for the test. - Spawn a client on a different VM to query 100k rows and do that for 100 reps. Each row key queried is drawn randomly from the set of existing row keys, and then not re-used, so all 10 million row queries use a different (valid) row key. This test is a specific use case of our system we are trying to show will scale Result: - 2 nodes performed better than 1 node test but 4 nodes showed decreased performance over 2 nodes. So that did not show horizontal scaling Notes: - We have replication factor set to 1 as we were trying to keep the control test simple to prove out horizontal scaling. - When we tried to add threading to see if it would help it had interesting side behavior which did not prove out horizontal scaling. - We are using CQL versus THRIFT API for Cassandra 2.0.6 Does anyone have any feedback that either threading or replication factor is necessary to show horizontal scaling of Cassandra versus the minimal way of just continue to add nodes to help throughput? Any suggestions of minimal configuration necessary to show scaling of our query use case 100k requests for random non repeating keys constantly coming in over a period of time? Thanks, Diane
Re: horizontal query scaling issues follow on
On Thu, Jul 17, 2014 at 3:21 PM, Diane Griffith dfgriff...@gmail.com
wrote:
So do partitions equate to tokens/vnodes?
A partition is what used to be called a row.
Each individual token in the token ring can contain a partition, which you
request using the token as the key.
A token range is the space between two tokens.
If so we had configured all cluster nodes/vms with num_tokens: 256 instead
of setting init_token and assigning ranges. I am still not getting why in
Cassandra 2.0, I would assign my own ranges via init_token and this was
based on the documentation and even this blog item
http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 that
made it seem right for us to always configure our cluster vms with
num_tokens: 256 in the cassandra.yaml file.
If you are using vnodes and don't want to try to figure out what ideally
random token ranges for them are, you should, generally :
1) start the node with num_tokens set to a value greater than 1
2) once succesffully bootstrapped, dump all node tokens with :
nodetool info -T | grep Token | awk '{print $3}' | paste -s -d,
3) put list from 2) in initial_token list in cassandra.yaml
4) (optional) restart and verify that your node has the tokens you expect
So given that I was assuming the partitions were such that it wasn't a
problem. Is that an incorrect assumption and something to dig into more?
How many client threads do you have? Your OP suggested a low number, which
will not have good results in terms of throughput?
=Rob
Re: horizontal query scaling issues follow on
So I stripped out the number of clients experiment path information. It is
unclear if I can only show horizontal scaling by also spawning many client
requests all working at once. So that is why I stripped that information
out to distill what our original attempt was at how to show horizontal
scaling.
I did tests comparing 1, 2, 10, 20, 50, 100 clients spawned all querying.
Performance on 2 nodes starts to degrade from 10 clients on. I saw
similar behavior on 4 nodes but haven't done the official runs on that yet.
When I tried to grab the list of tokens assigned and populate it in the
cassandra.yaml I never got it right.
I basically did the command and it was outputting 256 tokens on each node
and comma separated. So I tried taking that string and setting that as the
value to initial_token but the node wouldn't start up.
Not sure if I maybe had a carriage return in there and that was the problem.
And if I do that do I need to do more than comment out num_tokens?
Thanks,
Diane
On Thu, Jul 17, 2014 at 6:58 PM, Robert Coli rc...@eventbrite.com wrote:
On Thu, Jul 17, 2014 at 3:21 PM, Diane Griffith dfgriff...@gmail.com
wrote:
So do partitions equate to tokens/vnodes?
A partition is what used to be called a row.
Each individual token in the token ring can contain a partition, which you
request using the token as the key.
A token range is the space between two tokens.
If so we had configured all cluster nodes/vms with num_tokens: 256
instead of setting init_token and assigning ranges. I am still not getting
why in Cassandra 2.0, I would assign my own ranges via init_token and this
was based on the documentation and even this blog item
http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2 that
made it seem right for us to always configure our cluster vms with
num_tokens: 256 in the cassandra.yaml file.
If you are using vnodes and don't want to try to figure out what ideally
random token ranges for them are, you should, generally :
1) start the node with num_tokens set to a value greater than 1
2) once succesffully bootstrapped, dump all node tokens with :
nodetool info -T | grep Token | awk '{print $3}' | paste -s -d,
3) put list from 2) in initial_token list in cassandra.yaml
4) (optional) restart and verify that your node has the tokens you expect
So given that I was assuming the partitions were such that it wasn't a
problem. Is that an incorrect assumption and something to dig into more?
How many client threads do you have? Your OP suggested a low number, which
will not have good results in terms of throughput?
=Rob
Re: horizontal query scaling issues follow on
On Thu, Jul 17, 2014 at 5:16 PM, Diane Griffith dfgriff...@gmail.com wrote: I did tests comparing 1, 2, 10, 20, 50, 100 clients spawned all querying. Performance on 2 nodes starts to degrade from 10 clients on. I saw similar behavior on 4 nodes but haven't done the official runs on that yet. Ok, if you've multi-threaded your client, then you aren't starving for client thread paralellism, and that rules out another scalability bottleneck. As a brief aside, you only lose from vnodes until your cluster is larger than a certain sizes, and then only when adding or removing nodes from a cluster. Perhaps if you are ramping up and scientifically testing smaller cluster sizes, you should start at first with a token per range, ie pre-vnodes operation? I basically did the command and it was outputting 256 tokens on each node and comma separated. So I tried taking that string and setting that as the value to initial_token but the node wouldn't start up. Not sure if I maybe had a carriage return in there and that was the problem. It should take a comma delimited list of tokens, did the failed node startup log any error? And if I do that do I need to do more than comment out num_tokens? No, though you probably should anyway in order to be unambiguous. =Rob
Re: horizontal query scaling issues follow on
Sorry I may have confused the discussion by mentioning tokens – I wasn’t intending to refer to vnodes or the num_tokens property, but merely referring to the token range of a node and that the partition key hashes to a token value. The main question is what you use for your primary key and whether you are using a small number of partition keys and a large number of clustering columns, or does each row have a unique partition key and no clustering columns. -- Jack Krupansky From: Diane Griffith Sent: Thursday, July 17, 2014 6:21 PM To: user Subject: Re: horizontal query scaling issues follow on So do partitions equate to tokens/vnodes? If so we had configured all cluster nodes/vms with num_tokens: 256 instead of setting init_token and assigning ranges. I am still not getting why in Cassandra 2.0, I would assign my own ranges via init_token and this was based on the documentation and even this blog item that made it seem right for us to always configure our cluster vms with num_tokens: 256 in the cassandra.yaml file. Also in all testing, all vms were of equal sizing so one was not more powerful than another. I didn't think I was hitting an i/o wall on the client vm (separate vm) where we command line scripted our query call to the cassandra cluster.I can break the client call load across vms which I tried early on. Happy to verify that again though. So given that I was assuming the partitions were such that it wasn't a problem. Is that an incorrect assumption and something to dig into more? Thanks, Diane On Thu, Jul 17, 2014 at 3:01 PM, Jack Krupansky j...@basetechnology.com wrote: How many partitions are you spreading those 18 million rows over? That many rows in a single partition will not be a sweet spot for Cassandra. It’s not exceeding any hard limit (2 billion), but some internal operations may cache the partition rather than the logical row. And all those rows in a single partition would certainly not be a test of “horizontal scaling” (adding nodes to handle more data – more token values or partitions.) -- Jack Krupansky From: Diane Griffith Sent: Thursday, July 17, 2014 1:33 PM To: user Subject: horizontal query scaling issues follow on This is a follow on re-post to clarify what we are trying to do, providing information that was missing or not clear. Goal: Verify horizontal scaling for random non duplicating key reads using the simplest configuration (or minimal configuration) possible. Background: A couple years ago we did similar performance testing with Cassandra for both read and write performance and found excellent (essentially linear) horizontal scalability. That project got put on hold. We are now moving forward with an operational system and are having scaling problems. During the prior testing (3 years ago) we were using a much older version of Cassandra (0.8 or older), the THRIFT API, and Amazon AWS rather than OpenStack VMs. We are now using the latest Cassandra and the CQL interface. We did try moving from OpenStack to AWS/EC2 but that did not materially change our (poor) results. Test Procedure: a.. Inserted 54 million cells in 18 million rows (so 3 cells per row), using randomly generated row keys. That was to be our data control for the test. b.. Spawn a client on a different VM to query 100k rows and do that for 100 reps. Each row key queried is drawn randomly from the set of existing row keys, and then not re-used, so all 10 million row queries use a different (valid) row key. This test is a specific use case of our system we are trying to show will scale Result: a.. 2 nodes performed better than 1 node test but 4 nodes showed decreased performance over 2 nodes. So that did not show horizontal scaling Notes: a.. We have replication factor set to 1 as we were trying to keep the control test simple to prove out horizontal scaling. b.. When we tried to add threading to see if it would help it had interesting side behavior which did not prove out horizontal scaling. c.. We are using CQL versus THRIFT API for Cassandra 2.0.6 Does anyone have any feedback that either threading or replication factor is necessary to show horizontal scaling of Cassandra versus the minimal way of just continue to add nodes to help throughput? Any suggestions of minimal configuration necessary to show scaling of our query use case 100k requests for random non repeating keys constantly coming in over a period of time? Thanks, Diane
Re: horizontal query scaling issues follow on
The problem with starting without vnodes is moving to them is a bit hairy. In particular, nodetool shuffle has been reported to take an extremely long time (days, weeks). I would start with vnodes if you have any intent on using them. On Thu, Jul 17, 2014 at 6:03 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Jul 17, 2014 at 5:16 PM, Diane Griffith dfgriff...@gmail.com wrote: I did tests comparing 1, 2, 10, 20, 50, 100 clients spawned all querying. Performance on 2 nodes starts to degrade from 10 clients on. I saw similar behavior on 4 nodes but haven't done the official runs on that yet. Ok, if you've multi-threaded your client, then you aren't starving for client thread paralellism, and that rules out another scalability bottleneck. As a brief aside, you only lose from vnodes until your cluster is larger than a certain sizes, and then only when adding or removing nodes from a cluster. Perhaps if you are ramping up and scientifically testing smaller cluster sizes, you should start at first with a token per range, ie pre-vnodes operation? I basically did the command and it was outputting 256 tokens on each node and comma separated. So I tried taking that string and setting that as the value to initial_token but the node wouldn't start up. Not sure if I maybe had a carriage return in there and that was the problem. It should take a comma delimited list of tokens, did the failed node startup log any error? And if I do that do I need to do more than comment out num_tokens? No, though you probably should anyway in order to be unambiguous. =Rob -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
