Large commit log reasons
Hi fellows. I current have 3 nodes cluster running with a replication factor of 1. It's a pretty simple deployment and all my enforcements are focused in writes rather than reads. Actually I'm noticing that my commit log size is always very big if compared to the ammout of data being persisted(which varies on 5gb). So, that lead me to three doubts: 1- When a commit log gets bigger, does it mean that cassandra hasnt processed yet those writes? 2- How could I speed my flushes to sstables? 3- Does my commit log decrease as much as my sstable increases? Is it a rule?
Large commit log reasons
Hi fellows. I current have 3 nodes cluster running with a replication factor of 1. It's a pretty simple deployment and all my enforcements are focused in writes rather than reads. Actually I'm noticing that my commit log size is always very big if compared to the ammout of data being persisted(which varies on 5gb). So, that lead me to three doubts: 1- When a commit log gets bigger, does it mean that cassandra hasnt processed yet those writes? 2- How could I speed my flushes to sstables? 3- Does my commit log decrease as much as my sstable increases? Is it a rule? -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Large-commit-log-reasons-tp7584964.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: Large commit log reasons
By default Cassandra uses 1/3rd heap size for memtable storage. If you make sure memtables smaller they should flush faster and you commit logs should not grow large. Large commit logs are not a problem, some use cases that write to some Column Families more then other can make the commit log directory grow. Basically the commit log does not get removed until everything in it is flushed. We have a nagios alarm on ours, if it hits 8GB something is wrong, but again large commit log is normal and I would not worry. Edward On Wed, Jan 23, 2013 at 10:42 AM, vhmolinar vhmoli...@gmail.com wrote: Hi fellows. I current have 3 nodes cluster running with a replication factor of 1. It's a pretty simple deployment and all my enforcements are focused in writes rather than reads. Actually I'm noticing that my commit log size is always very big if compared to the ammout of data being persisted(which varies on 5gb). So, that lead me to three doubts: 1- When a commit log gets bigger, does it mean that cassandra hasnt processed yet those writes? 2- How could I speed my flushes to sstables? 3- Does my commit log decrease as much as my sstable increases? Is it a rule? -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Large-commit-log-reasons-tp7584964.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: Large commit log reasons
1. The commit log is only read on startup. W: If writes are unflushed then the commit logs need to be replayed 2: shrink the memtable settings. but you dont want to do this. 3. Commit log size is not directly related to sstable size. E.g. if you write the same row a billion times the commit log size will be large but the sstable will be 1 row. On Wed, Jan 23, 2013 at 11:10 AM, vhmolinar vhmoli...@gmail.com wrote: W
Re: Large commit log reasons
I current have 3 nodes cluster running with a replication factor of 1. That is a very strange setup. Actually I'm noticing that my commit log size is always very big How big? The commitlog_total_space_in_mb setting in the yaml file controls the maximum size. note that these files are pre-allocated and are recycled. 1- When a commit log gets bigger, does it mean that cassandra hasnt processed yet those writes? No. They are written to the commit log, added to memory and then flushed to disk when need. 2- How could I speed my flushes to sstables? Do you need to? 3- Does my commit log decrease as much as my sstable increases? Is it a rule? No. The commit log is all writes, the memtables and the sstables are the latest writes for the columns. Cheers - Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 24/01/2013, at 4:48 AM, Víctor Hugo Oliveira Molinar vhmoli...@gmail.com wrote: Hi fellows. I current have 3 nodes cluster running with a replication factor of 1. It's a pretty simple deployment and all my enforcements are focused in writes rather than reads. Actually I'm noticing that my commit log size is always very big if compared to the ammout of data being persisted(which varies on 5gb). So, that lead me to three doubts: 1- When a commit log gets bigger, does it mean that cassandra hasnt processed yet those writes? 2- How could I speed my flushes to sstables? 3- Does my commit log decrease as much as my sstable increases? Is it a rule?
