RE: no change observed in read latency after switching from EBS to SSD storage
Chris, I agree that reading 250k row is a bit excessive and that breaking up the partition would help reduce the query time. That part is well understood. The part that we can't figure out is why read time did not change when we switched from a slow Network Attached Storage (AWS EBS) to local SSD. One possibility is that the read is not bound by disk i/o, but it is not cpu or memory bound either. So where is it spending all that time? Another possibility is that even though it is returning only 193311 cells, C* reads the entire partition, which may have a lot more cells. But even in that case reading from a local SSD should have been a lot faster than reading from non-provisioned EBS. Mohammed From: Chris Lohfink [mailto:clohf...@blackbirdit.com] Sent: Wednesday, September 17, 2014 7:17 PM To: user@cassandra.apache.org Subject: Re: no change observed in read latency after switching from EBS to SSD storage Read 193311 live and 0 tombstoned cells is your killer. returning 250k rows is a bit excessive, you should really page this in smaller chunks, what client are you using to access the data? This partition (a, b, c, d, e, f) may be too large as well (can check partition max size from output of nodetool cfstats), may be worth including g to break it up more - but I dont know enough about your data model. --- Chris Lohfink On Sep 17, 2014, at 4:53 PM, Mohammed Guller moham...@glassbeam.commailto:moham...@glassbeam.com wrote: Thank you all for your responses. Alex - Instance (ephemeral) SSD Ben - the query reads data from just one partition. If disk i/o is the bottleneck, then in theory, if reading from EBS takes 10 seconds, then it should take lot less when reading the same amount of data from local SSD. My question is not about why it is taking 10 seconds, but why is the read time same for both EBS (network attached storage) and local SSD? Tony - if the data was cached in memory, then a read should not take 10 seconds just for 20MB data Rob - Here is the schema, query, and trace. I masked the actual column names to protect the innocents :) create table dummy( a varchar, b varchar, c varchar, d varchar, e varchar, f varchar, g varchar, h timestamp, i int, non_key1 varchar, ... non_keyN varchar, PRIMARY KEY ((a, b, c, d, e, f), g, h, i) ) WITH CLUSTERING ORDER BY (g ASC, h DESC, i ASC) SELECT h, non_key100, non_key200 FROM dummy WHERE a='' AND b='bb' AND c='ccc' AND d='dd' AND e='' AND f='ff' AND g='g'AND h ='2014-09-10T00:00:00' AND h='2014-09-10T23:40:41'; The above query returns around 250,000 CQL rows. cqlsh trace: activity | timestamp| source | source_elapsed - execute_cql3_query | 21:57:16,830 | 10.10.100.5 | 0 Parsing query; | 21:57:16,830 | 10.10.100.5 |673 Preparing statement | 21:57:16,831 | 10.10.100.5 | 1602 Executing single-partition query on event | 21:57:16,845 | 10.10.100.5 | 14871 Acquiring sstable references | 21:57:16,845 | 10.10.100.5 | 14896 Merging memtable tombstones | 21:57:16,845 | 10.10.100.5 | 14954 Bloom filter allows skipping sstable 1049 | 21:57:16,845 | 10.10.100.5 | 15090 Bloom filter allows skipping sstable 989 | 21:57:16,845 | 10.10.100.5 | 15146 Partition index with 0 entries found for sstable 937 | 21:57:16,845 | 10.10.100.5 | 15565 Seeking to partition indexed section in data file | 21:57:16,845 | 10.10.100.5 | 15581 Partition index with 7158 entries found for sstable 884 | 21:57:16,898 | 10.10.100.5 | 68644 Seeking to partition indexed section in data file | 21:57:16,899 | 10.10.100.5 | 69014 Partition index with 20819 entries found for sstable 733 | 21:57:16,916 | 10.10.100.5 | 86121 Seeking to partition indexed section in data file | 21:57:16,916 | 10.10.100.5 | 86412 Skipped 1/6 non-slice-intersecting sstables, included 0 due to tombstones | 21:57:16,916 | 10.10.100.5 | 86494 Merging data from memtables and 3 sstables | 21:57:16,916 | 10.10.100.5 | 86522 Read 193311 live and 0 tombstoned cells | 21:57:24,552 | 10.10.100.5 | 7722425 Request complete | 21:57:29,074 | 10.10.100.5 | 12244832 Mohammed From: Alex Major [mailto:al3...@gmail.com] Sent: Wednesday, September 17, 2014 3:47 AM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: no change observed in read latency after switching from EBS to SSD storage When you say you moved from EBS to SSD, do you mean the EBS HDD drives to EBS SSD drives? Or instance SSD drives? The m3.large only comes with 32GB of instance based SSD storage. If you're using EBS SSD drives then network will still be the slowest thing so switching won't likely make much of a difference. On Wed, Sep 17, 2014 at 6:00 AM,
Re: no change observed in read latency after switching from EBS to SSD storage
It is possible this is CPU bound. In 2.1 we have optimised the comparison of clustering columns (CASSANDRA-5417 https://issues.apache.org/jira/browse/CASSANDRA-5417), but in 2.0 it is quite expensive. So for a large row with several million comparisons to perform (to merge, filter, etc.) it could be a significant proportion of the cost. Note that these costs for a given query are all bound by a single core, there is no parallelism, since the assumption is we are serving more queries at once than there are cores (in general Cassandra is not designed to serve workloads consisting of single large queries, at least not yet) On Thu, Sep 18, 2014 at 7:29 AM, Mohammed Guller moham...@glassbeam.com wrote: Chris, I agree that reading 250k row is a bit excessive and that breaking up the partition would help reduce the query time. That part is well understood. The part that we can’t figure out is why read time did not change when we switched from a slow Network Attached Storage (AWS EBS) to local SSD. One possibility is that the read is not bound by disk i/o, but it is not cpu or memory bound either. So where is it spending all that time? Another possibility is that even though it is returning only 193311 cells, C* reads the entire partition, which may have a lot more cells. But even in that case reading from a local SSD should have been a lot faster than reading from non-provisioned EBS. Mohammed *From:* Chris Lohfink [mailto:clohf...@blackbirdit.com] *Sent:* Wednesday, September 17, 2014 7:17 PM *To:* user@cassandra.apache.org *Subject:* Re: no change observed in read latency after switching from EBS to SSD storage Read 193311 live and 0 tombstoned cells is your killer. returning 250k rows is a bit excessive, you should really page this in smaller chunks, what client are you using to access the data? This partition (a, b, c, d, e, f) may be too large as well (can check partition max size from output of *nodetool cfstats*), may be worth including g to break it up more - but I dont know enough about your data model. --- Chris Lohfink On Sep 17, 2014, at 4:53 PM, Mohammed Guller moham...@glassbeam.com wrote: Thank you all for your responses. Alex – Instance (ephemeral) SSD Ben – the query reads data from just one partition. If disk i/o is the bottleneck, then in theory, if reading from EBS takes 10 seconds, then it should take lot less when reading the same amount of data from local SSD. My question is not about why it is taking 10 seconds, but why is the read time same for both EBS (network attached storage) and local SSD? Tony – if the data was cached in memory, then a read should not take 10 seconds just for 20MB data Rob – Here is the schema, query, and trace. I masked the actual column names to protect the innocents J create table dummy( a varchar, b varchar, c varchar, d varchar, e varchar, f varchar, g varchar, h timestamp, i int, non_key1 varchar, ... non_keyN varchar, PRIMARY KEY ((a, b, c, d, e, f), g, h, i) ) WITH CLUSTERING ORDER BY (g ASC, h DESC, i ASC) SELECT h, non_key100, non_key200 FROM dummy WHERE a='' AND b='bb' AND c='ccc' AND d='dd' AND e='' AND f='ff' AND g='g'AND h ='2014-09-10T00:00:00' AND h='2014-09-10T23:40:41'; The above query returns around 250,000 CQL rows. cqlsh trace: activity | timestamp| source | source_elapsed - execute_cql3_query | 21:57:16,830 | 10.10.100.5 | 0 Parsing query; | 21:57:16,830 | 10.10.100.5 |673 Preparing statement | 21:57:16,831 | 10.10.100.5 | 1602 Executing single-partition query on event | 21:57:16,845 | 10.10.100.5 | 14871 Acquiring sstable references | 21:57:16,845 | 10.10.100.5 | 14896 Merging memtable tombstones | 21:57:16,845 | 10.10.100.5 | 14954 Bloom filter allows skipping sstable 1049 | 21:57:16,845 | 10.10.100.5 | 15090 Bloom filter allows skipping sstable 989 | 21:57:16,845 | 10.10.100.5 | 15146 Partition index with 0 entries found for sstable 937 | 21:57:16,845 | 10.10.100.5 | 15565 Seeking to partition indexed section in data file | 21:57:16,845 | 10.10.100.5 | 15581 Partition index with 7158 entries found for sstable 884 | 21:57:16,898 | 10.10.100.5 | 68644 Seeking to partition indexed section in data file | 21:57:16,899 | 10.10.100.5 | 69014 Partition index with 20819 entries found for sstable 733 | 21:57:16,916 | 10.10.100.5 | 86121 Seeking to partition indexed section in data file | 21:57:16,916 | 10.10.100.5 | 86412 Skipped 1/6 non-slice-intersecting sstables, included 0 due to tombstones | 21:57:16,916 | 10.10.100.5 | 86494 Merging data from
Re: cqlsh 2.1 to cassandra 2.0.7
There is now a ticket open to look into this and produce a more informative
error message:
https://datastax-oss.atlassian.net/browse/PYTHON-157
Adam
On Wed, Sep 17, 2014 at 4:47 PM, Adam Holmberg adam.holmb...@datastax.com
wrote:
This is not really supported. Presently cqlsh hard-codes CQL and protocol
to versions only supported in 2.1:
https://github.com/apache/cassandra/blob/cassandra-2.1.0/bin/cqlsh#L144-L147
Your best bet is probably downloading a 2.0.x tarball and running out of
there.
Adam Holmberg
On Wed, Sep 17, 2014 at 4:33 PM, Tyler Tolley thattolley...@gmail.com
wrote:
Just upgraded cassandra to 2.1 on my own box and when I try to connect to
our server running 2.0.7, I get the following error.
Connection error: ('Unable to connect to any servers', {'10.0.16.144':
ConnectionShutdown('Connection AsyncoreConnection(25708560)
10.0.16.144:9042 (closed) is already closed',)})
With debug on I get the following output.
Using CQL driver: module 'cassandra' from
'/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.0.post.zip/cassandra-driver-2.1.0.post/cassandra/__init__.py'
Connection error: ('Unable to connect to any servers', {'10.0.16.144':
ConnectionShutdown('Connection AsyncoreConnection(26384400)
10.0.16.144:9042 (closed) is already closed',)})
If I try to connect using an earlier version of cqlsh, it connects fine.
Anyone seen this before or have any advice on how to dig into the problem
further? Thanks,
-Tyler
Re: cqlsh 2.1 to cassandra 2.0.7
did you try running cqlsh with --cqlversion?
On Thu, Sep 18, 2014 at 9:17 AM, Adam Holmberg adam.holmb...@datastax.com
wrote:
There is now a ticket open to look into this and produce a more
informative error message:
https://datastax-oss.atlassian.net/browse/PYTHON-157
Adam
On Wed, Sep 17, 2014 at 4:47 PM, Adam Holmberg adam.holmb...@datastax.com
wrote:
This is not really supported. Presently cqlsh hard-codes CQL and protocol
to versions only supported in 2.1:
https://github.com/apache/cassandra/blob/cassandra-2.1.0/bin/cqlsh#L144-L147
Your best bet is probably downloading a 2.0.x tarball and running out of
there.
Adam Holmberg
On Wed, Sep 17, 2014 at 4:33 PM, Tyler Tolley thattolley...@gmail.com
wrote:
Just upgraded cassandra to 2.1 on my own box and when I try to connect
to our server running 2.0.7, I get the following error.
Connection error: ('Unable to connect to any servers', {'10.0.16.144':
ConnectionShutdown('Connection AsyncoreConnection(25708560)
10.0.16.144:9042 (closed) is already closed',)})
With debug on I get the following output.
Using CQL driver: module 'cassandra' from
'/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.0.post.zip/cassandra-driver-2.1.0.post/cassandra/__init__.py'
Connection error: ('Unable to connect to any servers', {'10.0.16.144':
ConnectionShutdown('Connection AsyncoreConnection(26384400)
10.0.16.144:9042 (closed) is already closed',)})
If I try to connect using an earlier version of cqlsh, it connects fine.
Anyone seen this before or have any advice on how to dig into the problem
further? Thanks,
-Tyler
index partition key
Hi,
This question is just for curiosity purposes, I don't need this in my
solution, but it's something I was asking myself...
Is there a way of indexing the partition key values in Cassandra? Does
anyone needed this and found a solution of any kind?
For example, I know the sample bellow doesn't work, but would it be
possible somehow?
I know select * from testcf where token(key1) token('Lucas'); works, but
the question is ordering by the values, even using another CF or index of
any kind...
CREATE KEYSPACE IF NOT EXISTS testks
WITH REPLICATION = { 'class' : 'SimpleStrategy',
'replication_factor': '1' };
CREATE TABLE IF NOT EXISTS testcf (
key1 varchar,
value varchar,
PRIMARY KEY ((key1)));
CREATE INDEX testidx on testks.testcf (key1);
insert into testcf (key1, value) values ('Lucas', 'value1');
insert into testcf (key1, value) values ('Luiz', 'value2');
insert into testcf (key1, value) values ('Edu', 'value3');
insert into testcf (key1, value) values ('Marcelo', 'value4');
select * from testcf order by key1;
select * from testcf where key1 'Lucas';
Best regards,
Marcelo.
New Debian/Ubuntu Repository GPG Signer
The Apache Cassandra apt repository welcomes T Jake Luciani j...@apache.org as a new package signer. This means that apt repository users will need to import a new GPG key (the wiki page is being updated). $ gpg --keyserver pgp.mit.edu --recv-keys 749D6EEC0353B12C $ gpg --export --armor 749D6EEC0353B12C | sudo apt-key add - For context, I'm attaching the output you might see from `apt-get update`. Kind regards, Michael mshuler@hana:~$ sudo apt-get update ... Reading package lists... Done W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://www.apache.org 21x InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 749D6EEC0353B12C W: Failed to fetch http://www.apache.org/dist/cassandra/debian/dists/21x/InRelease W: Some index files failed to download. They have been ignored, or old ones used instead. mshuler@hana:~$ apt-cache policy cassandra cassandra: Installed: (none) Candidate: 2.1.0 Version table: 2.1.0 0 500 http://www.apache.org/dist/cassandra/debian/ 21x/main amd64 Packages mshuler@hana:~$ gpg --keyserver pgp.mit.edu --recv-keys 749D6EEC0353B12C gpg: requesting key 0353B12C from hkp server pgp.mit.edu gpg: key 0353B12C: public key T Jake Luciani j...@apache.org imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1) mshuler@hana:~$ gpg --export --armor 749D6EEC0353B12C | sudo apt-key add - OK mshuler@hana:~$ sudo apt-get update ... Reading package lists... Done mshuler@hana:~$ apt-cache policy cassandra cassandra: Installed: (none) Candidate: 2.1.0-2 Version table: 2.1.0-2 0 500 http://www.apache.org/dist/cassandra/debian/ 21x/main amd64 Packages
Re: cqlsh 2.1 to cassandra 2.0.7
I did and got the same error.
On Thu, Sep 18, 2014 at 9:10 AM, groupie02 groupi...@gmail.com wrote:
did you try running cqlsh with --cqlversion?
On Thu, Sep 18, 2014 at 9:17 AM, Adam Holmberg adam.holmb...@datastax.com
wrote:
There is now a ticket open to look into this and produce a more
informative error message:
https://datastax-oss.atlassian.net/browse/PYTHON-157
Adam
On Wed, Sep 17, 2014 at 4:47 PM, Adam Holmberg
adam.holmb...@datastax.com wrote:
This is not really supported. Presently cqlsh hard-codes CQL and
protocol to versions only supported in 2.1:
https://github.com/apache/cassandra/blob/cassandra-2.1.0/bin/cqlsh#L144-L147
Your best bet is probably downloading a 2.0.x tarball and running out of
there.
Adam Holmberg
On Wed, Sep 17, 2014 at 4:33 PM, Tyler Tolley thattolley...@gmail.com
wrote:
Just upgraded cassandra to 2.1 on my own box and when I try to connect
to our server running 2.0.7, I get the following error.
Connection error: ('Unable to connect to any servers', {'10.0.16.144':
ConnectionShutdown('Connection AsyncoreConnection(25708560)
10.0.16.144:9042 (closed) is already closed',)})
With debug on I get the following output.
Using CQL driver: module 'cassandra' from
'/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.0.post.zip/cassandra-driver-2.1.0.post/cassandra/__init__.py'
Connection error: ('Unable to connect to any servers', {'10.0.16.144':
ConnectionShutdown('Connection AsyncoreConnection(26384400)
10.0.16.144:9042 (closed) is already closed',)})
If I try to connect using an earlier version of cqlsh, it connects
fine. Anyone seen this before or have any advice on how to dig into the
problem further? Thanks,
-Tyler
purge gossip info
I recently added nodes to existing cluster and removed some. nodetool gossipinfo doesn't show the non existing nodes but a thread dump on cassandra reveals it is trying to write to the non existing old node. I tried restarting the cluster using -Dcassandra.load_ring_state=false on each node but has no effect to the behavior. Is there a way to purge this through CQL? Thanks,SC
Re: index partition key
Your query select * from testcf where key1 'Lucas'; does work if you
choose the old OrderPartioner but since it's considered absolute evil (for
good reason, load distribution is horrendous), pratically it is not
possible to do your query.
And it's the same thing with secondary index. Query with inequality on
secondary index translates into full cluster scan...
Regards
Duy Hai DOAN
On Thu, Sep 18, 2014 at 5:44 PM, Marcelo Elias Del Valle
marc...@s1mbi0se.com.br wrote:
Hi,
This question is just for curiosity purposes, I don't need this in my
solution, but it's something I was asking myself...
Is there a way of indexing the partition key values in Cassandra? Does
anyone needed this and found a solution of any kind?
For example, I know the sample bellow doesn't work, but would it be
possible somehow?
I know select * from testcf where token(key1) token('Lucas'); works, but
the question is ordering by the values, even using another CF or index of
any kind...
CREATE KEYSPACE IF NOT EXISTS testks
WITH REPLICATION = { 'class' : 'SimpleStrategy',
'replication_factor': '1' };
CREATE TABLE IF NOT EXISTS testcf (
key1 varchar,
value varchar,
PRIMARY KEY ((key1)));
CREATE INDEX testidx on testks.testcf (key1);
insert into testcf (key1, value) values ('Lucas', 'value1');
insert into testcf (key1, value) values ('Luiz', 'value2');
insert into testcf (key1, value) values ('Edu', 'value3');
insert into testcf (key1, value) values ('Marcelo', 'value4');
select * from testcf order by key1;
select * from testcf where key1 'Lucas';
Best regards,
Marcelo.
Re: How to get rid of stale info in gossip
unsafeAssassinate remove gossip info and token range ownership for a node WITHOUT streaming data. On Tue, Aug 5, 2014 at 2:56 AM, Rahul Neelakantan ra...@rahul.be wrote: It no longer shows up, but I will definitely investigate that option. What exactly does it do? Rahul Neelakantan On Aug 4, 2014, at 7:49 PM, Patricia Gorla patri...@thelastpickle.com wrote: Rahul, If the removed node is still showing up in gossip state, you can remove it with 'unsafeAssassinate' via a JMX tool such as jconsole or jmxterm. On Fri, Jul 25, 2014 at 6:10 AM, Rahul Neelakantan ra...@rahul.be wrote: Yes, and this is a really old version of casandra 1.0.8. Rahul Neelakantan 678-451-4545 On Jul 25, 2014, at 7:29 AM, Mark Reddy mark.re...@boxever.com wrote: After removing a node, it's information can persist in the Gossiper for up to 3 days, after which time it should be removed. Are you having issues with a removed node state persisting for longer? Mark On Fri, Jul 25, 2014 at 11:33 AM, Rahul Neelakantan ra...@rahul.be wrote: Is there a way to get rid of stale information that shows up for removed/dead nodes in gossip, without a complete cluster bounce? Rahul Neelakantan -- Patricia Gorla @patriciagorla Consultant Apache Cassandra Consulting http://www.thelastpickle.com http://thelastpickle.com
[RELEASE] Apache Cassandra 1.2.19 released
The Cassandra team is pleased to announce the release of Apache Cassandra version 1.2.19. Cassandra is a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. You can read more here: http://cassandra.apache.org/ Downloads of source and binary distributions are listed in our download section: http://cassandra.apache.org/download/ This version is a maintenance/bug fix release[1] on the 1.2 series. As always, please pay attention to the release notes[2] and Let us know[3] if you were to encounter any problem. This will likely be the final release in the 1.2 series. Enjoy! [1]: http://goo.gl/F6szqv (CHANGES.txt) [2]: http://goo.gl/9VsZ88 (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA
Re: index partition key
Yes, indeed using the evil ordered partitioner it would work, but it should
be avoided at all costs.
What I am trying to figure is: Can I tell Cassandra is not good for cases
where you need range queries over all the cluster?
It's just a kind of architectural rule I am trying to use to decide
whether when to use a kind of technology or another.
Solr, for instance, allows you to query the way you want over all the
cluster, so data stax enterprise could solve this problem, but I wonder
if it's the only way.
[]s
2014-09-18 13:35 GMT-03:00 DuyHai Doan doanduy...@gmail.com:
Your query select * from testcf where key1 'Lucas'; does work if you
choose the old OrderPartioner but since it's considered absolute evil (for
good reason, load distribution is horrendous), pratically it is not
possible to do your query.
And it's the same thing with secondary index. Query with inequality on
secondary index translates into full cluster scan...
Regards
Duy Hai DOAN
On Thu, Sep 18, 2014 at 5:44 PM, Marcelo Elias Del Valle
marc...@s1mbi0se.com.br wrote:
Hi,
This question is just for curiosity purposes, I don't need this in my
solution, but it's something I was asking myself...
Is there a way of indexing the partition key values in Cassandra? Does
anyone needed this and found a solution of any kind?
For example, I know the sample bellow doesn't work, but would it be
possible somehow?
I know select * from testcf where token(key1) token('Lucas'); works,
but the question is ordering by the values, even using another CF or index
of any kind...
CREATE KEYSPACE IF NOT EXISTS testks
WITH REPLICATION = { 'class' : 'SimpleStrategy',
'replication_factor': '1' };
CREATE TABLE IF NOT EXISTS testcf (
key1 varchar,
value varchar,
PRIMARY KEY ((key1)));
CREATE INDEX testidx on testks.testcf (key1);
insert into testcf (key1, value) values ('Lucas', 'value1');
insert into testcf (key1, value) values ('Luiz', 'value2');
insert into testcf (key1, value) values ('Edu', 'value3');
insert into testcf (key1, value) values ('Marcelo', 'value4');
select * from testcf order by key1;
select * from testcf where key1 'Lucas';
Best regards,
Marcelo.
RE: no change observed in read latency after switching from EBS to SSD storage
Benedict, That makes perfect sense. Even though the node has multiple cores, I do see that only one core is pegged at 100%. Interestingly, after I switched to 2.1, cqlsh trace now shows that the same query takes only 600ms. However, cqlsh still waits for almost 20-30 seconds before it starts showing the result. I noticed similar latency when I ran the query from our app, which uses the Astyanax driver. So I thought perhaps there is a bug in the cqlsh code that tracks the statistics and the reported numbers are incorrect. But, I guess the numbers shown by cqlsh trace is correct, but the bottleneck is somewhere else now. In other words, the read operation itself is much faster in 2.1, but something else delays the response back to the client. Mohammed From: Benedict Elliott Smith [mailto:belliottsm...@datastax.com] Sent: Thursday, September 18, 2014 2:15 AM To: user@cassandra.apache.org Cc: Chris Lohfink Subject: Re: no change observed in read latency after switching from EBS to SSD storage It is possible this is CPU bound. In 2.1 we have optimised the comparison of clustering columns (CASSANDRA-5417https://issues.apache.org/jira/browse/CASSANDRA-5417), but in 2.0 it is quite expensive. So for a large row with several million comparisons to perform (to merge, filter, etc.) it could be a significant proportion of the cost. Note that these costs for a given query are all bound by a single core, there is no parallelism, since the assumption is we are serving more queries at once than there are cores (in general Cassandra is not designed to serve workloads consisting of single large queries, at least not yet) On Thu, Sep 18, 2014 at 7:29 AM, Mohammed Guller moham...@glassbeam.commailto:moham...@glassbeam.com wrote: Chris, I agree that reading 250k row is a bit excessive and that breaking up the partition would help reduce the query time. That part is well understood. The part that we can’t figure out is why read time did not change when we switched from a slow Network Attached Storage (AWS EBS) to local SSD. One possibility is that the read is not bound by disk i/o, but it is not cpu or memory bound either. So where is it spending all that time? Another possibility is that even though it is returning only 193311 cells, C* reads the entire partition, which may have a lot more cells. But even in that case reading from a local SSD should have been a lot faster than reading from non-provisioned EBS. Mohammed From: Chris Lohfink [mailto:clohf...@blackbirdit.commailto:clohf...@blackbirdit.com] Sent: Wednesday, September 17, 2014 7:17 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: no change observed in read latency after switching from EBS to SSD storage Read 193311 live and 0 tombstoned cells is your killer. returning 250k rows is a bit excessive, you should really page this in smaller chunks, what client are you using to access the data? This partition (a, b, c, d, e, f) may be too large as well (can check partition max size from output of nodetool cfstats), may be worth including g to break it up more - but I dont know enough about your data model. --- Chris Lohfink On Sep 17, 2014, at 4:53 PM, Mohammed Guller moham...@glassbeam.commailto:moham...@glassbeam.com wrote: Thank you all for your responses. Alex – Instance (ephemeral) SSD Ben – the query reads data from just one partition. If disk i/o is the bottleneck, then in theory, if reading from EBS takes 10 seconds, then it should take lot less when reading the same amount of data from local SSD. My question is not about why it is taking 10 seconds, but why is the read time same for both EBS (network attached storage) and local SSD? Tony – if the data was cached in memory, then a read should not take 10 seconds just for 20MB data Rob – Here is the schema, query, and trace. I masked the actual column names to protect the innocents ☺ create table dummy( a varchar, b varchar, c varchar, d varchar, e varchar, f varchar, g varchar, h timestamp, i int, non_key1 varchar, ... non_keyN varchar, PRIMARY KEY ((a, b, c, d, e, f), g, h, i) ) WITH CLUSTERING ORDER BY (g ASC, h DESC, i ASC) SELECT h, non_key100, non_key200 FROM dummy WHERE a='' AND b='bb' AND c='ccc' AND d='dd' AND e='' AND f='ff' AND g='g'AND h ='2014-09-10T00:00:00' AND h='2014-09-10T23:40:41'; The above query returns around 250,000 CQL rows. cqlsh trace: activity | timestamp| source | source_elapsed - execute_cql3_query | 21:57:16,830 | 10.10.100.5 | 0 Parsing query; | 21:57:16,830 | 10.10.100.5 |673 Preparing statement | 21:57:16,831 | 10.10.100.5 | 1602 Executing single-partition query on event | 21:57:16,845 | 10.10.100.5 | 14871 Acquiring sstable references | 21:57:16,845 | 10.10.100.5 |
Why column with timestamp datatype come in different format?
I have a Cassandra cluster version as - cqlsh:dataks show version; [cqlsh 2.3.0 | Cassandra 2.0.6 | CQL spec 3.0.0 | Thrift protocol 19.39.0] And I have a table like this - CREATE TABLE data_test ( valid_id int, data_id text, client_name text, creation_date timestamp, last_modified_date timestamp, PRIMARY KEY (valid_id, data_id) ) I am inserting the data like this in my above table - insert into data_test (valid_id, data_id, client_name, creation_date, last_modified_date) values (1, 'b4b61aa', 'TTLAP', dateOf(now()), dateOf(now())); After I do a select on my table, I am seeing creation_date and last_modified_date coming in some other format, not sure why? valid_id | data_id | client_name | creation_date | last_modified_date --+ 1 | b4b61aa | TTLAP | \x00\x00\x01H\x89\xf0\xb6I | \x00\x00\x01H\x89\xf0\xb6I Does anyone know why creation_date and last_modified_date is coming like this and how we can get actual timestamp in those columns?
Re: [RELEASE] Apache Cassandra 1.2.19 released
Apologies, the correct url for CHANGES.txt is http://goo.gl/eB973i On Thu, Sep 18, 2014 at 12:58 PM, Jake Luciani j...@apache.org wrote: The Cassandra team is pleased to announce the release of Apache Cassandra version 1.2.19. Cassandra is a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. You can read more here: http://cassandra.apache.org/ Downloads of source and binary distributions are listed in our download section: http://cassandra.apache.org/download/ This version is a maintenance/bug fix release[1] on the 1.2 series. As always, please pay attention to the release notes[2] and Let us know[3] if you were to encounter any problem. This will likely be the final release in the 1.2 series. Enjoy! [1]: http://goo.gl/F6szqv (CHANGES.txt) [2]: http://goo.gl/9VsZ88 (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA
Re: hs_err_pid3013.log, out of memory?
On Wed, Sep 17, 2014 at 8:35 PM, Yatong Zhang bluefl...@gmail.com wrote: @Chris Lohfink I have 16G memory per node, all the other settings are default @J. Ryan Earl I am not sure. I am using the default settings. But I've found out it might be because some settings in '/etc/sysctl.conf'. I am still testing it If JNA is installed, it will try to memlock all of the JVM process. For this to happen, you have to adjust the settings for the user you run Cassandra as under /etc/security/limits.conf or limits.d/ and you have to modifying kernel.shmmax in sysctl.conf accordingly. If you do not, and JNA is installed, the memlock will fail with the error you gave.
Re: ava.lang.OutOfMemoryError: unable to create new native thread
What's the 'ulimit -a' output of the user cassandra runs as? From this and your previous OOM thread, is sounds like you skipped the requisite OS configuration. On Wed, Sep 17, 2014 at 9:43 AM, Yatong Zhang bluefl...@gmail.com wrote: Hi there, I am using leveled compaction strategy and have many sstable files. The error was during the startup, so any idea about this? ERROR [FlushWriter:4] 2014-09-17 22:36:59,383 CassandraDaemon.java (line 199) Exception in thread Thread[FlushWriter:4,5,main] java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:693) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) ERROR [FlushWriter:2] 2014-09-17 22:36:59,472 CassandraDaemon.java (line 199) Exception in thread Thread[FlushWriter:2,5,main] FSReadError in /data5/cass/system/compactions_in_progress/system-compactions_in_progress-jb-23-Index.db at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:200) at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:168) at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:334) at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:324) at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:394) at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:342) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849) at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192) ... 10 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:846) ... 11 more
Re: Why column with timestamp datatype come in different format?
Hi, Have tried removing your dateOf calls and only inserting now()? Cheers, Jens On Thu, Sep 18, 2014 at 8:13 PM, Check Peck comptechge...@gmail.com wrote: I have a Cassandra cluster version as - cqlsh:dataks show version; [cqlsh 2.3.0 | Cassandra 2.0.6 | CQL spec 3.0.0 | Thrift protocol 19.39.0] And I have a table like this - CREATE TABLE data_test ( valid_id int, data_id text, client_name text, creation_date timestamp, last_modified_date timestamp, PRIMARY KEY (valid_id, data_id) ) I am inserting the data like this in my above table - insert into data_test (valid_id, data_id, client_name, creation_date, last_modified_date) values (1, 'b4b61aa', 'TTLAP', dateOf(now()), dateOf(now())); After I do a select on my table, I am seeing creation_date and last_modified_date coming in some other format, not sure why? valid_id | data_id | client_name | creation_date | last_modified_date --+ 1 | b4b61aa | TTLAP | \x00\x00\x01H\x89\xf0\xb6I | \x00\x00\x01H\x89\xf0\xb6I Does anyone know why creation_date and last_modified_date is coming like this and how we can get actual timestamp in those columns?
