Re: 1.1 not removing commit log files?
In theory the the logging configurator is watching the log4j-server.properties file and check for changes every 10 seconds. I've nevery had much luck with it, but assumed it was me getting something wrong. Or you can modify the values on the fly using the setLog4jLevel() on the StorageService MBean via JMX. It will log set log level to… to say the value has changed. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 5/06/2012, at 9:58 AM, Bryce Godfrey wrote: I’ll try to get some log files for this with DEBUG enabled. Tough on production though. From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Monday, June 04, 2012 11:15 AM To: user@cassandra.apache.org Subject: Re: 1.1 not removing commit log files? Apply the local hint mutation follows the same code path and regular mutations. When the commit log is being truncated you should see flush activity, logged from the ColumnFamilyStore with Enqueuing flush of messages. If you set DEBUG logging for the org.apache.cassandra.db.ColumnFamilyStore it will log if it things the CF is clean and no flush takes place. If you set DEBUG logging on org.apache.cassandra.db.commitlog.CommitLog we will see if the commit log file could not be deleted because a dirty CF was not flushed. Cheers A - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 2/06/2012, at 4:43 AM, Rob Coli wrote: On Thu, May 31, 2012 at 7:01 PM, aaron morton aa...@thelastpickle.com wrote: But that talks about segments not being cleared at startup. Does not explain why they were allowed to get past the limit in the first place. Perhaps the commit log size tracking for this limit does not, for some reason, track hints? This seems like the obvious answer given the state which appears to trigger it? This doesn't explain why the files aren't getting deleted after the hints are delivered, of course... =Rob -- =Robert Coli AIMGTALK - rc...@palominodb.com YAHOO - rcoli.palominob SKYPE - rcoli_palominodb
Re: 1.1 not removing commit log files?
Apply the local hint mutation follows the same code path and regular mutations. When the commit log is being truncated you should see flush activity, logged from the ColumnFamilyStore with Enqueuing flush of messages. If you set DEBUG logging for the org.apache.cassandra.db.ColumnFamilyStore it will log if it things the CF is clean and no flush takes place. If you set DEBUG logging on org.apache.cassandra.db.commitlog.CommitLog we will see if the commit log file could not be deleted because a dirty CF was not flushed. Cheers A - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 2/06/2012, at 4:43 AM, Rob Coli wrote: On Thu, May 31, 2012 at 7:01 PM, aaron morton aa...@thelastpickle.com wrote: But that talks about segments not being cleared at startup. Does not explain why they were allowed to get past the limit in the first place. Perhaps the commit log size tracking for this limit does not, for some reason, track hints? This seems like the obvious answer given the state which appears to trigger it? This doesn't explain why the files aren't getting deleted after the hints are delivered, of course... =Rob -- =Robert Coli AIMGTALK - rc...@palominodb.com YAHOO - rcoli.palominob SKYPE - rcoli_palominodb
RE: 1.1 not removing commit log files?
I'll try to get some log files for this with DEBUG enabled. Tough on production though. From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Monday, June 04, 2012 11:15 AM To: user@cassandra.apache.org Subject: Re: 1.1 not removing commit log files? Apply the local hint mutation follows the same code path and regular mutations. When the commit log is being truncated you should see flush activity, logged from the ColumnFamilyStore with Enqueuing flush of messages. If you set DEBUG logging for the org.apache.cassandra.db.ColumnFamilyStore it will log if it things the CF is clean and no flush takes place. If you set DEBUG logging on org.apache.cassandra.db.commitlog.CommitLog we will see if the commit log file could not be deleted because a dirty CF was not flushed. Cheers A - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 2/06/2012, at 4:43 AM, Rob Coli wrote: On Thu, May 31, 2012 at 7:01 PM, aaron morton aa...@thelastpickle.commailto:aa...@thelastpickle.com wrote: But that talks about segments not being cleared at startup. Does not explain why they were allowed to get past the limit in the first place. Perhaps the commit log size tracking for this limit does not, for some reason, track hints? This seems like the obvious answer given the state which appears to trigger it? This doesn't explain why the files aren't getting deleted after the hints are delivered, of course... =Rob -- =Robert Coli AIMGTALK - rc...@palominodb.commailto:rc...@palominodb.com YAHOO - rcoli.palominob SKYPE - rcoli_palominodb
Re: 1.1 not removing commit log files?
On Thu, May 31, 2012 at 7:01 PM, aaron morton aa...@thelastpickle.com wrote: But that talks about segments not being cleared at startup. Does not explain why they were allowed to get past the limit in the first place. Perhaps the commit log size tracking for this limit does not, for some reason, track hints? This seems like the obvious answer given the state which appears to trigger it? This doesn't explain why the files aren't getting deleted after the hints are delivered, of course... =Rob -- =Robert Coli AIMGTALK - rc...@palominodb.com YAHOO - rcoli.palominob SKYPE - rcoli_palominodb
RE: 1.1 not removing commit log files?
So this happened to me again, but it was only when the cluster had a node down for a while. Then the commit logs started piling up past the limit I set in the config file, and filled the drive. After the node recovered and hints had replayed the space was never reclaimed. A flush or drain did not reclaim the space either and delete any log files. Bryce Godfrey | Sr. Software Engineer | Azaleos Corporationhttp://www.azaleos.com/ From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com] Sent: Tuesday, May 22, 2012 1:10 PM To: user@cassandra.apache.org Subject: RE: 1.1 not removing commit log files? The nodes appear to be holding steady at the 8G that I set it to in the config file now. I'll keep an eye on them. From: aaron morton [mailto:aa...@thelastpickle.com]mailto:[mailto:aa...@thelastpickle.com] Sent: Tuesday, May 22, 2012 4:08 AM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: 1.1 not removing commit log files? 4096 is also the internal hard coded default for commitlog_total_space_in_mb If you are seeing more that 4GB of commit log files let us know. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 22/05/2012, at 6:35 AM, Bryce Godfrey wrote: Thanks, I'll give it a try. -Original Message- From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]mailto:[mailto:arodr...@gmail.com] Sent: Monday, May 21, 2012 2:12 AM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: 1.1 not removing commit log files? commitlog_total_space_in_mb: 4096 By default this line is commented in 1.0.x if I remember well. I guess it is the same in 1.1. You really should remove this comment or your commit logs will entirely fill up your disk as it happened to me a while ago. Alain 2012/5/21 Pieter Callewaert pieter.callewa...@be-mobile.bemailto:pieter.callewa...@be-mobile.be: Hi, In 1.1 the commitlog files are pre-allocated with files of 128MB. (https://issues.apache.org/jira/browse/CASSANDRA-3411) This should however not exceed your commitlog size in Cassandra.yaml. commitlog_total_space_in_mb: 4096 Kind regards, Pieter Callewaert From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com]mailto:[mailto:bryce.godf...@azaleos.com] Sent: maandag 21 mei 2012 9:52 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: 1.1 not removing commit log files? The commit log drives on my nodes keep slowly filling up. I don't see any errors in my logs that are indicating any issues that I can map to this issue. Is this how 1.1 is supposed to work now? Previous versions seemed to keep this drive at a minimum as it flushed. /dev/mapper/mpathf 25G 21G 4.2G 83% /opt/cassandra/commitlog
Re: 1.1 not removing commit log files?
Could be this https://issues.apache.org/jira/browse/CASSANDRA-4201 But that talks about segments not being cleared at startup. Does not explain why they were allowed to get past the limit in the first place. Can you share some logs from the time the commit log got out of control ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 1/06/2012, at 9:34 AM, Bryce Godfrey wrote: So this happened to me again, but it was only when the cluster had a node down for a while. Then the commit logs started piling up past the limit I set in the config file, and filled the drive. After the node recovered and hints had replayed the space was never reclaimed. A flush or drain did not reclaim the space either and delete any log files. Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com] Sent: Tuesday, May 22, 2012 1:10 PM To: user@cassandra.apache.org Subject: RE: 1.1 not removing commit log files? The nodes appear to be holding steady at the 8G that I set it to in the config file now. I’ll keep an eye on them. From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Tuesday, May 22, 2012 4:08 AM To: user@cassandra.apache.org Subject: Re: 1.1 not removing commit log files? 4096 is also the internal hard coded default for commitlog_total_space_in_mb If you are seeing more that 4GB of commit log files let us know. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 22/05/2012, at 6:35 AM, Bryce Godfrey wrote: Thanks, I'll give it a try. -Original Message- From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: Monday, May 21, 2012 2:12 AM To: user@cassandra.apache.org Subject: Re: 1.1 not removing commit log files? commitlog_total_space_in_mb: 4096 By default this line is commented in 1.0.x if I remember well. I guess it is the same in 1.1. You really should remove this comment or your commit logs will entirely fill up your disk as it happened to me a while ago. Alain 2012/5/21 Pieter Callewaert pieter.callewa...@be-mobile.be: Hi, In 1.1 the commitlog files are pre-allocated with files of 128MB. (https://issues.apache.org/jira/browse/CASSANDRA-3411) This should however not exceed your commitlog size in Cassandra.yaml. commitlog_total_space_in_mb: 4096 Kind regards, Pieter Callewaert From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com] Sent: maandag 21 mei 2012 9:52 To: user@cassandra.apache.org Subject: 1.1 not removing commit log files? The commit log drives on my nodes keep slowly filling up. I don't see any errors in my logs that are indicating any issues that I can map to this issue. Is this how 1.1 is supposed to work now? Previous versions seemed to keep this drive at a minimum as it flushed. /dev/mapper/mpathf 25G 21G 4.2G 83% /opt/cassandra/commitlog
Re: 1.1 not removing commit log files?
4096 is also the internal hard coded default for commitlog_total_space_in_mb If you are seeing more that 4GB of commit log files let us know. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 22/05/2012, at 6:35 AM, Bryce Godfrey wrote: Thanks, I'll give it a try. -Original Message- From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: Monday, May 21, 2012 2:12 AM To: user@cassandra.apache.org Subject: Re: 1.1 not removing commit log files? commitlog_total_space_in_mb: 4096 By default this line is commented in 1.0.x if I remember well. I guess it is the same in 1.1. You really should remove this comment or your commit logs will entirely fill up your disk as it happened to me a while ago. Alain 2012/5/21 Pieter Callewaert pieter.callewa...@be-mobile.be: Hi, In 1.1 the commitlog files are pre-allocated with files of 128MB. (https://issues.apache.org/jira/browse/CASSANDRA-3411) This should however not exceed your commitlog size in Cassandra.yaml. commitlog_total_space_in_mb: 4096 Kind regards, Pieter Callewaert From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com] Sent: maandag 21 mei 2012 9:52 To: user@cassandra.apache.org Subject: 1.1 not removing commit log files? The commit log drives on my nodes keep slowly filling up. I don't see any errors in my logs that are indicating any issues that I can map to this issue. Is this how 1.1 is supposed to work now? Previous versions seemed to keep this drive at a minimum as it flushed. /dev/mapper/mpathf 25G 21G 4.2G 83% /opt/cassandra/commitlog
RE: 1.1 not removing commit log files?
The nodes appear to be holding steady at the 8G that I set it to in the config file now. I'll keep an eye on them. From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Tuesday, May 22, 2012 4:08 AM To: user@cassandra.apache.org Subject: Re: 1.1 not removing commit log files? 4096 is also the internal hard coded default for commitlog_total_space_in_mb If you are seeing more that 4GB of commit log files let us know. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 22/05/2012, at 6:35 AM, Bryce Godfrey wrote: Thanks, I'll give it a try. -Original Message- From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]mailto:[mailto:arodr...@gmail.com] Sent: Monday, May 21, 2012 2:12 AM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: 1.1 not removing commit log files? commitlog_total_space_in_mb: 4096 By default this line is commented in 1.0.x if I remember well. I guess it is the same in 1.1. You really should remove this comment or your commit logs will entirely fill up your disk as it happened to me a while ago. Alain 2012/5/21 Pieter Callewaert pieter.callewa...@be-mobile.bemailto:pieter.callewa...@be-mobile.be: Hi, In 1.1 the commitlog files are pre-allocated with files of 128MB. (https://issues.apache.org/jira/browse/CASSANDRA-3411) This should however not exceed your commitlog size in Cassandra.yaml. commitlog_total_space_in_mb: 4096 Kind regards, Pieter Callewaert From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com]mailto:[mailto:bryce.godf...@azaleos.com] Sent: maandag 21 mei 2012 9:52 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: 1.1 not removing commit log files? The commit log drives on my nodes keep slowly filling up. I don't see any errors in my logs that are indicating any issues that I can map to this issue. Is this how 1.1 is supposed to work now? Previous versions seemed to keep this drive at a minimum as it flushed. /dev/mapper/mpathf 25G 21G 4.2G 83% /opt/cassandra/commitlog
RE: 1.1 not removing commit log files?
Hi, In 1.1 the commitlog files are pre-allocated with files of 128MB. (https://issues.apache.org/jira/browse/CASSANDRA-3411) This should however not exceed your commitlog size in Cassandra.yaml. commitlog_total_space_in_mb: 4096 Kind regards, Pieter Callewaert From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com] Sent: maandag 21 mei 2012 9:52 To: user@cassandra.apache.org Subject: 1.1 not removing commit log files? The commit log drives on my nodes keep slowly filling up. I don't see any errors in my logs that are indicating any issues that I can map to this issue. Is this how 1.1 is supposed to work now? Previous versions seemed to keep this drive at a minimum as it flushed. /dev/mapper/mpathf 25G 21G 4.2G 83% /opt/cassandra/commitlog
Re: 1.1 not removing commit log files?
commitlog_total_space_in_mb: 4096 By default this line is commented in 1.0.x if I remember well. I guess it is the same in 1.1. You really should remove this comment or your commit logs will entirely fill up your disk as it happened to me a while ago. Alain 2012/5/21 Pieter Callewaert pieter.callewa...@be-mobile.be: Hi, In 1.1 the commitlog files are pre-allocated with files of 128MB. (https://issues.apache.org/jira/browse/CASSANDRA-3411) This should however not exceed your commitlog size in Cassandra.yaml. commitlog_total_space_in_mb: 4096 Kind regards, Pieter Callewaert From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com] Sent: maandag 21 mei 2012 9:52 To: user@cassandra.apache.org Subject: 1.1 not removing commit log files? The commit log drives on my nodes keep slowly filling up. I don’t see any errors in my logs that are indicating any issues that I can map to this issue. Is this how 1.1 is supposed to work now? Previous versions seemed to keep this drive at a minimum as it flushed. /dev/mapper/mpathf 25G 21G 4.2G 83% /opt/cassandra/commitlog
