So this happened to me again, but it was only when the cluster had a node down for a while. Then the commit logs started piling up past the limit I set in the config file, and filled the drive. After the node recovered and hints had replayed the space was never reclaimed. A flush or drain did not reclaim the space either and delete any log files.
Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation<http://www.azaleos.com/> From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com] Sent: Tuesday, May 22, 2012 1:10 PM To: user@cassandra.apache.org Subject: RE: 1.1 not removing commit log files? The nodes appear to be holding steady at the 8G that I set it to in the config file now. I'll keep an eye on them. From: aaron morton [mailto:aa...@thelastpickle.com]<mailto:[mailto:aa...@thelastpickle.com]> Sent: Tuesday, May 22, 2012 4:08 AM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: 1.1 not removing commit log files? 4096 is also the internal hard coded default for commitlog_total_space_in_mb If you are seeing more that 4GB of commit log files let us know. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 22/05/2012, at 6:35 AM, Bryce Godfrey wrote: Thanks, I'll give it a try. -----Original Message----- From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]<mailto:[mailto:arodr...@gmail.com]> Sent: Monday, May 21, 2012 2:12 AM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: 1.1 not removing commit log files? commitlog_total_space_in_mb: 4096 By default this line is commented in 1.0.x if I remember well. I guess it is the same in 1.1. You really should remove this comment or your commit logs will entirely fill up your disk as it happened to me a while ago. Alain 2012/5/21 Pieter Callewaert <pieter.callewa...@be-mobile.be<mailto:pieter.callewa...@be-mobile.be>>: Hi, In 1.1 the commitlog files are pre-allocated with files of 128MB. (https://issues.apache.org/jira/browse/CASSANDRA-3411) This should however not exceed your commitlog size in Cassandra.yaml. commitlog_total_space_in_mb: 4096 Kind regards, Pieter Callewaert From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com]<mailto:[mailto:bryce.godf...@azaleos.com]> Sent: maandag 21 mei 2012 9:52 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: 1.1 not removing commit log files? The commit log drives on my nodes keep slowly filling up. I don't see any errors in my logs that are indicating any issues that I can map to this issue. Is this how 1.1 is supposed to work now? Previous versions seemed to keep this drive at a minimum as it flushed. /dev/mapper/mpathf 25G 21G 4.2G 83% /opt/cassandra/commitlog
