Re: Compaction failing to trigger
On Wed, Jan 21, 2015 at 10:10 AM, Flavien Charlon flavien.char...@gmail.com wrote: https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ This doesn't really answer my question, I asked whether this particular bug (which I can't find in JIRA) is planned to be fixed in 2.1.3, not whether 2.1.3 would be production ready. No idea, but as I didn't recognize your name/email and you were encountering problems with an IMO not-ready-for-production version. Many people who are new to Cassandra and pre-or-close-to-production might be better served by running a slightly older version and focusing on the challenge of writing their app against a mostly-working distributed database instead of troubleshooting Cassandra bugs. tl;dr - Cassandra bugs in cutting edge versions are likely best encountered by experienced operators who can recognize them and respond, not new operators. While we're on this topic, the version numbering is very misleading. Version which are not recommended for production should be very explicitly labelled as such (beta for example), and 2.1.0 should really be what you call now 2.1.6. That's why I wrote the blog post. It is however important to note that I speak in no official capacity for Apache Cassandra or Datastax. The intent of the project is for x.y.0 to be production ready, and in fairness they have recently added new QA processes which are likely to drive the production ready version down from x.y.6. They are only human, however, and as human developers are likely have slightly different (lower) standards for production readiness than the typical operator. I wrote that blog post to help set operator-appropriate expectations, so people are not disappointed with the overall stability of Cassandra. I personally operate Cassandra slightly on the trailing edge, and as a result only encounter a limited subset of the problems I assist people with on the list and IRC. =Rob
Re: Compaction failing to trigger
What version of Cassandra are you running? 2.1.2 Are they all live? Are there pending compactions, or exceptions regarding compactions in your logs? Yes they are all live according to cfstats. There is no pending compaction or exception in the logs. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ This doesn't really answer my question, I asked whether this particular bug (which I can't find in JIRA) is planned to be fixed in 2.1.3, not whether 2.1.3 would be production ready. While we're on this topic, the version numbering is very misleading. Version which are not recommended for production should be very explicitly labelled as such (beta for example), and 2.1.0 should really be what you call now 2.1.6. Setting 'cold_reads_to_omit' to 0 did the job for me Thanks, I've tried it, and it works. This should probably be made the default IMO. Flavien On 20 January 2015 at 22:51, Eric Stevens migh...@gmail.com wrote: @Rob - he's probably referring to the thread titled Reasons for nodes not compacting? where Tyler speculates that the tables are falling below the cold read threshold for compaction. He speculated it may be a bug. At the same time in a different thread, Roland had a similar problem, and Tyler's proposed workaround seemed to work for him. On Tue, Jan 20, 2015 at 3:35 PM, Robert Coli rc...@eventbrite.com wrote: On Sun, Jan 18, 2015 at 6:06 PM, Flavien Charlon flavien.char...@gmail.com wrote: It's set on all the tables, as I'm using the default for all the tables. But for that particular table there are 41 SSTables between 60MB and 85MB, it should only take 4 for the compaction to kick in. What version of Cassandra are you running? Are they all live? Are there pending compactions, or exceptions regarding compactions in your logs? As this is probably a bug and going back in the mailing list archive, it seems it's already been reported: This is a weird statement. Are you saying that you've found it in the mailing list archives? If so, why not paste the threads so those of us who might remember can refer to them? - Will it be fixed in 2.1.3? https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob
Re: Compaction failing to trigger
On Sun, Jan 18, 2015 at 6:06 PM, Flavien Charlon flavien.char...@gmail.com wrote: It's set on all the tables, as I'm using the default for all the tables. But for that particular table there are 41 SSTables between 60MB and 85MB, it should only take 4 for the compaction to kick in. What version of Cassandra are you running? Are they all live? Are there pending compactions, or exceptions regarding compactions in your logs? As this is probably a bug and going back in the mailing list archive, it seems it's already been reported: This is a weird statement. Are you saying that you've found it in the mailing list archives? If so, why not paste the threads so those of us who might remember can refer to them? - Will it be fixed in 2.1.3? https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob
Re: Compaction failing to trigger
@Rob - he's probably referring to the thread titled Reasons for nodes not compacting? where Tyler speculates that the tables are falling below the cold read threshold for compaction. He speculated it may be a bug. At the same time in a different thread, Roland had a similar problem, and Tyler's proposed workaround seemed to work for him. On Tue, Jan 20, 2015 at 3:35 PM, Robert Coli rc...@eventbrite.com wrote: On Sun, Jan 18, 2015 at 6:06 PM, Flavien Charlon flavien.char...@gmail.com wrote: It's set on all the tables, as I'm using the default for all the tables. But for that particular table there are 41 SSTables between 60MB and 85MB, it should only take 4 for the compaction to kick in. What version of Cassandra are you running? Are they all live? Are there pending compactions, or exceptions regarding compactions in your logs? As this is probably a bug and going back in the mailing list archive, it seems it's already been reported: This is a weird statement. Are you saying that you've found it in the mailing list archives? If so, why not paste the threads so those of us who might remember can refer to them? - Will it be fixed in 2.1.3? https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob
Re: Compaction failing to trigger
Thanks Roland. Good to know, I will try that. Do you know the JIRA ticket
number of that bug?
Thanks,
Flavien
On 19 January 2015 at 06:15, Roland Etzenhammer r.etzenham...@t-online.de
wrote:
Hi Flavien,
I hit some problem with minor compations recently (just some days ago) -
but with many more tables. In my case compactions got not triggered, you
can check this with nodetool compactionstats.
Reason for me was that those minor compactions did not get triggered since
there were almost no reads on that tables. Setting 'cold_reads_to_omit' to
0 did the job for me:
ALTER TABLE tablename WITH compaction = {'class':
'SizeTieredCompactionStrategy', 'min_threshold': '4', 'max_threshold':
'32', 'cold_reads_to_omit': 0.0};
Credits to Tyler and Eric for the pointers.
Cheers,
Roland
Re: Compaction failing to trigger
It's set on all the tables, as I'm using the default for all the tables. But for that particular table there are 41 SSTables between 60MB and 85MB, it should only take 4 for the compaction to kick in. As this is probably a bug and going back in the mailing list archive, it seems it's already been reported: - Is there a workaround? - What is the JIRA ticket number? - Will it be fixed in 2.1.3? Thanks Flavien On 19 January 2015 at 01:23, 严超 yanchao...@gmail.com wrote: Seems like Size Tier Compaction is based on table, which table did you set the compaction strategy? A minor compaction does not involve all the tables in a keyspace. Ref: http://datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_configure_compaction_t.html http://datastax.com/documentation/cql/3.1/cql/cql_reference/tabProp.html?scroll=tabProp__moreCompaction *Best Regards！* *Chao Yan--**My twitter：Andy Yan @yanchao727 https://twitter.com/yanchao727* *My Weibo：http://weibo.com/herewearenow http://weibo.com/herewearenow--* 2015-01-19 3:51 GMT+08:00 Flavien Charlon flavien.char...@gmail.com: Hi, I am using Size Tier Compaction (Cassandra 2.1.2). Minor compaction is not triggering even though it should. See the SSTables on disk: http://pastebin.com/PSwZ5mrT You can see that we have 41 SSTable between 60MB and 85MB, which should trigger compaction unless I am missing something. Is that a bug? Thanks, Flavien
Re: Compaction failing to trigger
Hi Flavien,
I hit some problem with minor compations recently (just some days ago) -
but with many more tables. In my case compactions got not triggered, you
can check this with nodetool compactionstats.
Reason for me was that those minor compactions did not get triggered
since there were almost no reads on that tables. Setting
'cold_reads_to_omit' to 0 did the job for me:
ALTER TABLE tablename WITH compaction = {'class':
'SizeTieredCompactionStrategy', 'min_threshold': '4', 'max_threshold': '32',
'cold_reads_to_omit': 0.0};
Credits to Tyler and Eric for the pointers.
Cheers,
Roland
Re: Compaction failing to trigger
Seems like Size Tier Compaction is based on table, which table did you set the compaction strategy? A minor compaction does not involve all the tables in a keyspace. Ref: http://datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_configure_compaction_t.html http://datastax.com/documentation/cql/3.1/cql/cql_reference/tabProp.html?scroll=tabProp__moreCompaction *Best Regards！* *Chao Yan--**My twitter：Andy Yan @yanchao727 https://twitter.com/yanchao727* *My Weibo：http://weibo.com/herewearenow http://weibo.com/herewearenow--* 2015-01-19 3:51 GMT+08:00 Flavien Charlon flavien.char...@gmail.com: Hi, I am using Size Tier Compaction (Cassandra 2.1.2). Minor compaction is not triggering even though it should. See the SSTables on disk: http://pastebin.com/PSwZ5mrT You can see that we have 41 SSTable between 60MB and 85MB, which should trigger compaction unless I am missing something. Is that a bug? Thanks, Flavien
