On Tue, Nov 20, 2012 at 5:23 PM, aaron morton <aa...@thelastpickle.com> wrote: > My understanding of the compaction process was that since data files keep > continuously merging we should not have data files with very old last > modified timestamps > > It is perfectly OK to have very old SSTables. > > But performing an upgradesstables did decrease the number of data files and > removed all the data files with the old timestamps. > > upgradetables re-writes every sstable to have the same contents in the > newest format. > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 19/11/2012, at 4:57 PM, Ananth Gundabattula <agundabatt...@gmail.com> > wrote: > > Hello Aaron, > > Thanks a lot for the reply. > > Looks like the documentation is confusing. Here is the link I am referring > to: http://www.datastax.com/docs/1.1/operations/tuning#tuning-compaction > > >> It does not disable compaction. > As per the above url, " After running a major compaction, automatic minor > compactions are no longer triggered, frequently requiring you to manually > run major compactions on a routine basis." ( Just before the heading Tuning > Column Family compression in the above link) > > With respect to the replies below : > > >> it creates one big file, which will not be compacted until there are (by >> default) 3 other very big files. > This is for the minor compaction and major compaction should theoretically > result in one large file irrespective of the number of data files initially? > >>This is not something you have to worry about. Unless you are seeing >> 1,000's of files using the default compaction. > > Well my worry has been because of the large amount of node movements we have > done in the ring. We started off with 6 nodes and increased the capacity to > 12 with disproportionate increases every time which resulted in a lot of > clean of data folders except system, run repair and then a cleanup with an > aborted attempt in between. > > There were some data.db files older by more than 2 weeks and were not > modified since then. My understanding of the compaction process was that > since data files keep continuously merging we should not have data files > with very old last modified timestamps (assuming there is a good amount of > writes to the table continuously) I did not have a for sure way of telling > if everything is alright with the compaction looking at the last modified > timestamps of all the data.db files. > >>What are the compaction issues you are having ? > Your replies confirm that the timestamps should not be an issue to worry > about. So I guess I should not be calling them as issues any more. But > performing an upgradesstables did decrease the number of data files and > removed all the data files with the old timestamps. > > > > Regards, > Ananth > > > On Mon, Nov 19, 2012 at 6:54 AM, aaron morton <aa...@thelastpickle.com> > wrote: >> >> As per datastax documentation, a manual compaction forces the admin to >> start compaction manually and disables the automated compaction (atleast for >> major compactions but not minor compactions ) >> >> It does not disable compaction. >> it creates one big file, which will not be compacted until there are (by >> default) 3 other very big files. >> >> >> 1. Does a nodetool stop compaction also force the admin to manually run >> major compaction ( I.e. disable automated major compactions ? ) >> >> No. >> Stop just stops the current compaction. >> Nothing is disabled. >> >> 2. Can a node restart reset the automated major compaction if a node gets >> into a manual mode compaction for whatever reason ? >> >> Major compaction is not automatic. It is the manual nodetool compact >> command. >> Automatic (minor) compaction is controlled by min_compaction_threshold and >> max_compaction_threshold (for the default compaction strategy). >> >> 3. What is the ideal number of SSTables for a table in a keyspace ( I >> mean are there any indicators as to whether my compaction is alright or not >> ? ) >> >> This is not something you have to worry about. >> Unless you are seeing 1,000's of files using the default compaction. >> >> For example, I have seen SSTables on the disk more than 10 days old >> wherein there were other SSTables belonging to the same table but much >> younger than the older SSTables ( >> >> No problems. >> >> 4. Does a upgradesstables fix any compaction issues ? >> >> What are the compaction issues you are having ? >> >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Developer >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 18/11/2012, at 1:18 AM, Ananth Gundabattula <agundabatt...@gmail.com> >> wrote: >> >> >> We have a cluster running cassandra 1.1.4. On this cluster, >> >> 1. We had to move the nodes around a bit when we were adding new nodes >> (there was quite a good amount of node movement ) >> >> 2. We had to stop compactions during some of the days to save some disk >> space on some of the nodes when they were running very very low on disk >> spaces. (via nodetool stop COMPACTION) >> >> >> As per datastax documentation, a manual compaction forces the admin to >> start compaction manually and disables the automated compaction (atleast for >> major compactions but not minor compactions ) >> >> >> Here are the questions I have regarding compaction: >> >> 1. Does a nodetool stop compaction also force the admin to manually run >> major compaction ( I.e. disable automated major compactions ? ) >> >> 2. Can a node restart reset the automated major compaction if a node gets >> into a manual mode compaction for whatever reason ? >> >> 3. What is the ideal number of SSTables for a table in a keyspace ( I >> mean are there any indicators as to whether my compaction is alright or not >> ? ) . For example, I have seen SSTables on the disk more than 10 days old >> wherein there were other SSTables belonging to the same table but much >> younger than the older SSTables ( The node movement and repair and cleanup >> happened between the older SSTables and the new SSTables being >> touched/modified) >> >> 4. Does a upgradesstables fix any compaction issues ? >> >> Regards, >> Ananth >> >> > >
"it is perfectly OK to have old sstables." Except for the fact that you can not repair and join new nodes until the cluster is on all on the same version all on the same files. Your gc_grace_time defaults to 10 days. This means that if you don't repair every node every 10 days something wonky can happen if you do deletes. Also in the past there was an issue if you upgraded from 0.8.X to 1.0.X. 1.0.X did not read some 0.8.X bloom filter files correctly. So you could get bad reads until you upgraded tables. These factors cause me to upgrade sstables as soon as possible after an upgrade.
