Re: Files not removed after compaction
You can't delete by year unless tables are organized by year. For deleting tables, cassandra keeeps a copy of the file history which you can delete. Also, you could delete all the extra rows using CQL. On Wed, Jun 10, 2015 at 7:15 PM, Jeff Ferland j...@tubularlabs.com wrote: Compaction finished yesterday, but I still have this going on: Space used (live), bytes: 878681716067 Space used (total), bytes: 2227857083852 jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ sudo lsof *-Data.db COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java4473 cassandra 446r REG 0,26 17582559172 39241 trends-trends-jb-144864-Data.db java4473 cassandra 448r REG 0,26 62040962 37431 trends-trends-jb-144731-Data.db java4473 cassandra 449r REG 0,26 829935047545 21150 trends-trends-jb-143581-Data.db java4473 cassandra 452r REG 0,26 8980406 39503 trends-trends-jb-144882-Data.db java4473 cassandra 454r REG 0,26 8980406 39503 trends-trends-jb-144882-Data.db java4473 cassandra 462r REG 0,26 9487703 39542 trends-trends-jb-144883-Data.db java4473 cassandra 463r REG 0,26 36158226 39629 trends-trends-jb-144889-Data.db java4473 cassandra 468r REG 0,26105693505 39447 trends-trends-jb-144881-Data.db java4473 cassandra 530r REG 0,26 17582559172 39241 trends-trends-jb-144864-Data.db java4473 cassandra 535r REG 0,26105693505 39447 trends-trends-jb-144881-Data.db java4473 cassandra 542r REG 0,26 9487703 39542 trends-trends-jb-144883-Data.db java4473 cassandra 553u REG 0,26 6431729821 39556 trends-trends-tmp-jb-144884-Data.db jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ ls *-Data.db trends-trends-jb-142631-Data.db trends-trends-jb-143562-Data.db trends-trends-jb-143581-Data.db trends-trends-jb-144731-Data.db trends-trends-jb-144883-Data.db trends-trends-jb-142633-Data.db trends-trends-jb-143563-Data.db trends-trends-jb-144530-Data.db trends-trends-jb-144864-Data.db trends-trends-jb-144889-Data.db trends-trends-jb-143026-Data.db trends-trends-jb-143564-Data.db trends-trends-jb-144551-Data.db trends-trends-jb-144881-Data.db trends-trends-tmp-jb-144884-Data.db trends-trends-jb-143533-Data.db trends-trends-jb-143578-Data.db trends-trends-jb-144552-Data.db trends-trends-jb-144882-Data.db jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ cd - /mnt/cassandra/data/trends/trends jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ sudo lsof * jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ ls *-Data.db trends-trends-jb-124502-Data.db trends-trends-jb-141113-Data.db trends-trends-jb-141377-Data.db trends-trends-jb-141846-Data.db trends-trends-jb-144890-Data.db trends-trends-jb-125457-Data.db trends-trends-jb-141123-Data.db trends-trends-jb-141391-Data.db trends-trends-jb-141871-Data.db trends-trends-jb-41121-Data.db trends-trends-jb-130016-Data.db trends-trends-jb-141137-Data.db trends-trends-jb-141538-Data.db trends-trends-jb-141883-Data.db trends-trends.trends_date_idx-jb-2100-Data.db trends-trends-jb-139563-Data.db trends-trends-jb-141358-Data.db trends-trends-jb-141806-Data.db trends-trends-jb-142033-Data.db trends-trends-jb-141102-Data.db trends-trends-jb-141363-Data.db trends-trends-jb-141829-Data.db trends-trends-jb-144553-Data.db Essentially, everything was brought down in a major compaction but Cassandra won’t let go of the files and I can’t figure out as to why they’re not being cleaned up even when they’re not open. I’d really like to get those files cleared out safely and rsync away from the EBS volume I used to survive a compaction. While I’m at it, is there any way to anti-compact a table by year? Since I’ve switched this recently to date tiered compaction, breaking up that big one would probably be helpful. -Jeff
Re: Dropped mutation messages
Internode messages which are received by a node, but do not get not to be processed within rpc_timeout are dropped rather than processed. As the coordinator node will no longer be waiting for a response. If the Coordinator node does not receive Consistency Level responses before the rpc_timeout it will return a TimedOutException to the client. I understand that, but that’s where this makes no sense. I’m running with RF=1, and CL=QUORUM, which means each update goes to one node, and I need one response for a success. I have many thousands of dropped mutation messages, but no TimedOutExceptions thrown back to the client. If I have GC problems, or other issues that are making my cluster unresponsive, I can deal with that. But having writes that fail and no error is clearly not acceptable. How is it possible to be getting errors and not be informed about them? Thanks Robert
Re: Seed Node OOM
The commitlog size is likely a red herring. In 2.0 we had 1gb commitlogs by default. In 2.1 we have 8gb commitlogs by default. This is configurable at the yaml. Not sure what's causing the OOM. Did it generate an hprof file you can analyze? On Jun 13, 2015 7:42 AM, Oleksandr Petrov oleksandr.pet...@gmail.com wrote: Sorry I completely forgot to mention it in an original message: we have rather large commitlog directory (which is usually rather small), 8G of commitlogs. Draining and flushing didn't help. On Sat, Jun 13, 2015 at 1:39 PM, Oleksandr Petrov oleksandr.pet...@gmail.com wrote: Hi, We're using Cassandra, recently migrated to 2.1.6, and we're experiencing constant OOMs in one of our clusters. It's a rather small cluster: 3 nodes, EC2 xlarge: 2CPUs, 8GB RAM, set up with datastax AMI. Configs (yaml and env.sh) are rather default: we've changed only concurrent compactions to 2 (although tried 1, too), tried setting HEAP and NEW to different values, ranging from 4G/200 to 6G/200M. Write load is rather small: 200-300 small payloads (4 varchar fields as a primary key, 2 varchar fields and a couple of long/double fields), plus some larger (1-2kb) payloads with a rate of 10-20 messages per second. We do a lot of range scans, but they are rather quick. It kind of started overnight. Compaction is taking a long time. Other two nodes in a cluster behave absolutely normally: no hinted handoffs, normal heap sizes. There were no write bursts, no tables added no indexes changed. Anyone experienced something similar? Maybe any pointers? -- alex p -- alex p
Re: Seed Node OOM
Sorry I completely forgot to mention it in an original message: we have rather large commitlog directory (which is usually rather small), 8G of commitlogs. Draining and flushing didn't help. On Sat, Jun 13, 2015 at 1:39 PM, Oleksandr Petrov oleksandr.pet...@gmail.com wrote: Hi, We're using Cassandra, recently migrated to 2.1.6, and we're experiencing constant OOMs in one of our clusters. It's a rather small cluster: 3 nodes, EC2 xlarge: 2CPUs, 8GB RAM, set up with datastax AMI. Configs (yaml and env.sh) are rather default: we've changed only concurrent compactions to 2 (although tried 1, too), tried setting HEAP and NEW to different values, ranging from 4G/200 to 6G/200M. Write load is rather small: 200-300 small payloads (4 varchar fields as a primary key, 2 varchar fields and a couple of long/double fields), plus some larger (1-2kb) payloads with a rate of 10-20 messages per second. We do a lot of range scans, but they are rather quick. It kind of started overnight. Compaction is taking a long time. Other two nodes in a cluster behave absolutely normally: no hinted handoffs, normal heap sizes. There were no write bursts, no tables added no indexes changed. Anyone experienced something similar? Maybe any pointers? -- alex p -- alex p
Seed Node OOM
Hi, We're using Cassandra, recently migrated to 2.1.6, and we're experiencing constant OOMs in one of our clusters. It's a rather small cluster: 3 nodes, EC2 xlarge: 2CPUs, 8GB RAM, set up with datastax AMI. Configs (yaml and env.sh) are rather default: we've changed only concurrent compactions to 2 (although tried 1, too), tried setting HEAP and NEW to different values, ranging from 4G/200 to 6G/200M. Write load is rather small: 200-300 small payloads (4 varchar fields as a primary key, 2 varchar fields and a couple of long/double fields), plus some larger (1-2kb) payloads with a rate of 10-20 messages per second. We do a lot of range scans, but they are rather quick. It kind of started overnight. Compaction is taking a long time. Other two nodes in a cluster behave absolutely normally: no hinted handoffs, normal heap sizes. There were no write bursts, no tables added no indexes changed. Anyone experienced something similar? Maybe any pointers? -- alex p
Re: Lucene index plugin for Apache Cassandra
Thanks for showing interest. Faceting is not yet supported, but it is in our roadmap. Our goal is to add to Cassandra as many Lucene features as possible. 2015-06-12 18:21 GMT+02:00 Mohammed Guller moham...@glassbeam.com: The plugin looks cool. Thank you for open sourcing it. Does it support faceting and other Solr functionality? Mohammed *From:* Andres de la Peña [mailto:adelap...@stratio.com] *Sent:* Friday, June 12, 2015 3:43 AM *To:* user@cassandra.apache.org *Subject:* Re: Lucene index plugin for Apache Cassandra I really appreciate your interest Well, the first recommendation is to not use it unless you need it, because a properly Cassandra denormalized model is almost always preferable to indexing. Lucene indexing is a good option when there is no viable denormalization alternative. This is the case of range queries over multiple dimensions, full-text search or maybe complex boolean predicates. It's also appropriate for Spark/Hadoop jobs mapping a small fraction of the total amount of rows in a certain table, if you can pay the cost of indexing. Lucene indexes run inside C*, so users should closely monitor the amount of used memory. It's also a good idea to put the Lucene directory files in a separate disk to those used by C* itself. Additionally, you should consider that indexed tables write throughput will be appreciably reduced, maybe to a few thousands rows per second. It's really hard to estimate the amount of resources needed by the index due to the great variety of indexing and querying ways that Lucene offers, so the only thing we can suggest is to empirically find the optimal setup for your use case. 2015-06-12 12:00 GMT+02:00 Carlos Rolo r...@pythian.com: Seems like an interesting tool! What operational recommendations would you make to users of this tool (Extra hardware capacity, extra metrics to monitor, etc)? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 www.pythian.com On Fri, Jun 12, 2015 at 11:07 AM, Andres de la Peña adelap...@stratio.com wrote: Unfortunately, we don't have published any benchmarks yet, but we have plans to do it as soon as possible. However, you can expect a similar behavior as those of Elasticsearch or Solr, with some overhead due to the need for indexing both the Cassandra's row key and the partition's token. You can also take a look at this presentation http://planetcassandra.org/video-presentations/vp/cassandra-summit-europe-2014/vd/stratio-advanced-search-and-top-k-queries-in-cassandra/ to see how cluster distribution is done. 2015-06-12 0:45 GMT+02:00 Ben Bromhead b...@instaclustr.com: Looks awesome, do you have any examples/benchmarks of using these indexes for various cluster sizes e.g. 20 nodes, 60 nodes, 100s+? On 10 June 2015 at 09:08, Andres de la Peña adelap...@stratio.com wrote: Hi all, With the release of Cassandra 2.1.6, Stratio is glad to present its open source Lucene-based implementation of C* secondary indexes https://github.com/Stratio/cassandra-lucene-index as a plugin that can be attached to Apache Cassandra. Before the above changes, Lucene index was distributed inside a fork of Apache Cassandra, with all the difficulties implied. As of now, the fork is discontinued and new users should use the recently created plugin, which maintains all the features of Stratio Cassandra https://github.com/Stratio/stratio-cassandra. Stratio's Lucene index extends Cassandra’s functionality to provide near real-time distributed search engine capabilities such as with ElasticSearch or Solr, including full text search capabilities, free multivariable search, relevance queries and field-based sorting. Each node indexes its own data, so high availability and scalability is guaranteed. We hope this will be useful to the Apache Cassandra community. Regards, -- Andrés de la Peña http://www.stratio.com/ Avenida de Europa, 26. Ática 5. 3ª Planta 28224 Pozuelo de Alarcón, Madrid Tel: +34 91 352 59 42 // *@stratiobd https://twitter.com/StratioBD* -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | (650) 284 9692 -- Andrés de la Peña http://www.stratio.com/ Avenida de Europa, 26. Ática 5. 3ª Planta 28224 Pozuelo de Alarcón, Madrid Tel: +34 91 352 59 42 // *@stratiobd https://twitter.com/StratioBD* -- -- Andrés de la Peña http://www.stratio.com/ Avenida de Europa, 26. Ática 5. 3ª Planta 28224 Pozuelo de Alarcón, Madrid Tel: +34 91 352 59 42 // *@stratiobd https://twitter.com/StratioBD* -- Andrés de la Peña http://www.stratio.com/ Avenida de Europa, 26. Ática 5. 3ª Planta 28224 Pozuelo de Alarcón, Madrid Tel:
Re: Dropped mutation messages
U said RF=1...missed that..so not sure eventual consistency is creating issues.. Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Anuj Wadehra anujw_2...@yahoo.co.in Date:Sat, 13 Jun, 2015 at 11:31 pm Subject:Re: Dropped mutation messages I think the messages dropped are the asynchronous ones required to maintain eventual consistency. Client may not be complaining as the data gets commited to one node synchronously..but dropped when sent to other nodes asynchronously.. We resolved similar issue in our cluster by increasing memtable_flush_writers to 3 from 1 ( we were writing to multiple cf simultaneously). We also fixed GC issues and reduced total_memtable_size_in_mb to ensure that most memtables are flushed early in heavy write loads. Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Robert Wille rwi...@fold3.com Date:Sat, 13 Jun, 2015 at 8:29 pm Subject:Re: Dropped mutation messages Internode messages which are received by a node, but do not get not to be processed within rpc_timeout are dropped rather than processed. As the coordinator node will no longer be waiting for a response. If the Coordinator node does not receive Consistency Level responses before the rpc_timeout it will return a TimedOutException to the client. I understand that, but that’s where this makes no sense. I’m running with RF=1, and CL=QUORUM, which means each update goes to one node, and I need one response for a success. I have many thousands of dropped mutation messages, but no TimedOutExceptions thrown back to the client. If I have GC problems, or other issues that are making my cluster unresponsive, I can deal with that. But having writes that fail and no error is clearly not acceptable. How is it possible to be getting errors and not be informed about them? Thanks Robert
Re: Dropped mutation messages
I think the messages dropped are the asynchronous ones required to maintain eventual consistency. Client may not be complaining as the data gets commited to one node synchronously..but dropped when sent to other nodes asynchronously.. We resolved similar issue in our cluster by increasing memtable_flush_writers to 3 from 1 ( we were writing to multiple cf simultaneously). We also fixed GC issues and reduced total_memtable_size_in_mb to ensure that most memtables are flushed early in heavy write loads. Thanks Anuj Wadehra Sent from Yahoo Mail on Android From:Robert Wille rwi...@fold3.com Date:Sat, 13 Jun, 2015 at 8:29 pm Subject:Re: Dropped mutation messages Internode messages which are received by a node, but do not get not to be processed within rpc_timeout are dropped rather than processed. As the coordinator node will no longer be waiting for a response. If the Coordinator node does not receive Consistency Level responses before the rpc_timeout it will return a TimedOutException to the client. I understand that, but that’s where this makes no sense. I’m running with RF=1, and CL=QUORUM, which means each update goes to one node, and I need one response for a success. I have many thousands of dropped mutation messages, but no TimedOutExceptions thrown back to the client. If I have GC problems, or other issues that are making my cluster unresponsive, I can deal with that. But having writes that fail and no error is clearly not acceptable. How is it possible to be getting errors and not be informed about them? Thanks Robert
