No real evidence it’s the case hear but the one time I’ve seen tombstones that refused to go away despite many attempts at compactions, etc it turned out to be due to the data being written (and deleted) with invalid timestamps years in the future (we guessed due to the time being set wrong somewhere at some point in time).
Cheers Ben On Thu, 28 Jul 2016 at 09:17 Alain RODRIGUEZ <arodr...@gmail.com> wrote: > Hi, > > I just released a detailed post about tombstones today that might be of > some interest for you: > http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html > > 220kb worth of tombstones doesn’t seem like enough to worry about. > > > +1 > > I believe you might be missing some other bigger SSTable having a lot of > tombstones as well. Finding the biggest sstable and reading the tombstone > ratio from there might be more relevant. > > You also should give a try to: "unchecked_tombstone_compaction" set to > true rather than tuning other options so aggressively. The "single SSTable > compaction" section of my post might help you on this issue: > http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html#single-sstable-compaction > > Other thoughts: > > Also if you use TTLs and timeseries, using TWCS instead of STCS could be > more efficient evicting tombstones. > > we have a columnfamily that has around 1000 rows, with one row is really >> huge (million columns) > > > I am sorry to say that this model does not look that great. Imbalances > might become an issue as a few nodes will handle a lot more load than the > rest of the nodes. Also even if this is getting improved in newer versions > of Cassandra, wide rows are something you want to avoid while using 2.0.14 > (which is no longer supported for about a year now). I know it is not > always easy and never the good time, but maybe should you consider > upgrading both your model and your version of Cassandra (regardless of the > fact you manage to solve this issue or not with > "unchecked_tombstone_compaction"). > > Good luck, > > C*heers, > ----------------------- > Alain Rodriguez - al...@thelastpickle.com > France > > The Last Pickle - Apache Cassandra Consulting > http://www.thelastpickle.com > > 2016-07-28 0:00 GMT+02:00 sai krishnam raju potturi <pskraj...@gmail.com>: > >> The read queries are continuously failing though because of the >> tombstones. "Request did not complete within rpc_timeout." >> >> thanks >> >> >> On Wed, Jul 27, 2016 at 5:51 PM, Jeff Jirsa <jeff.ji...@crowdstrike.com> >> wrote: >> >>> 220kb worth of tombstones doesn’t seem like enough to worry about. >>> >>> >>> >>> >>> >>> *From: *sai krishnam raju potturi <pskraj...@gmail.com> >>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> >>> *Date: *Wednesday, July 27, 2016 at 2:43 PM >>> *To: *Cassandra Users <user@cassandra.apache.org> >>> *Subject: *Re: Re : Purging tombstones from a particular row in SSTable >>> >>> >>> >>> and also the sstable size in question is like 220 kb in size. >>> >>> >>> >>> thanks >>> >>> >>> >>> >>> >>> On Wed, Jul 27, 2016 at 5:41 PM, sai krishnam raju potturi < >>> pskraj...@gmail.com> wrote: >>> >>> it's set to 1800 Vinay. >>> >>> >>> >>> bloom_filter_fp_chance=0.010000 AND >>> >>> caching='KEYS_ONLY' AND >>> >>> comment='' AND >>> >>> dclocal_read_repair_chance=0.100000 AND >>> >>> gc_grace_seconds=1800 AND >>> >>> index_interval=128 AND >>> >>> read_repair_chance=0.000000 AND >>> >>> replicate_on_write='true' AND >>> >>> populate_io_cache_on_flush='false' AND >>> >>> default_time_to_live=0 AND >>> >>> speculative_retry='99.0PERCENTILE' AND >>> >>> memtable_flush_period_in_ms=0 AND >>> >>> compaction={'min_sstable_size': '1024', 'tombstone_threshold': '0.01', >>> 'tombstone_compaction_interval': '1800', 'class': >>> 'SizeTieredCompactionStrategy'} AND >>> >>> compression={'sstable_compression': 'LZ4Compressor'}; >>> >>> >>> >>> thanks >>> >>> >>> >>> >>> >>> On Wed, Jul 27, 2016 at 5:34 PM, Vinay Kumar Chella < >>> vinaykumar...@gmail.com> wrote: >>> >>> What is your GC_grace_seconds set to? >>> >>> >>> >>> On Wed, Jul 27, 2016 at 1:13 PM, sai krishnam raju potturi < >>> pskraj...@gmail.com> wrote: >>> >>> thanks Vinay and DuyHai. >>> >>> >>> >>> we are using verison 2.0.14. I did "user defined compaction" >>> following the instructions in the below link, The tombstones still persist >>> even after that. >>> >>> >>> >>> https://gist.github.com/jeromatron/e238e5795b3e79866b83 >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_jeromatron_e238e5795b3e79866b83&d=CwMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=-sQ3Vf5bs3z4cO36h_AU-kIhMGVKcb3eCtzIb-fZ1Fc&s=0RQ3r6c0L4vICot8eqpOBKBAuKiKEkoKdmcjLbvBBwY&e=> >>> >>> >>> >>> Also, we changed the tombstone_compaction_interval : 1800 >>> and tombstone_threshold : 0.1, but it did not help. >>> >>> >>> >>> thanks >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Jul 27, 2016 at 4:05 PM, DuyHai Doan <doanduy...@gmail.com> >>> wrote: >>> >>> This feature is also exposed directly in nodetool from version Cassandra >>> 3.4 >>> >>> >>> >>> nodetool compact --user-defined <SSTable file> >>> >>> >>> >>> On Wed, Jul 27, 2016 at 9:58 PM, Vinay Chella <vche...@netflix.com> >>> wrote: >>> >>> You can run file level compaction using JMX to get rid of tombstones in >>> one SSTable. Ensure you set GC_Grace_seconds such that >>> >>> >>> >>> current time >= deletion(tombstone time)+ GC_Grace_seconds >>> >>> >>> >>> File level compaction >>> >>> >>> >>> /usr/bin/java -jar cmdline-jmxclient-0.10.3.jar - localhost: >>> >>> { >>> >>> port} >>> >>> org.apache.cassandra.db:type=CompactionManager >>> forceUserDefinedCompaction="'${KEYSPACE}','${ >>> >>> SSTABLEFILENAME >>> >>> }'"" >>> >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Jul 27, 2016 at 11:59 AM, sai krishnam raju potturi < >>> pskraj...@gmail.com> wrote: >>> >>> hi; >>> >>> we have a columnfamily that has around 1000 rows, with one row is >>> really huge (million columns). 95% of the row contains tombstones. Since >>> there exists just one SSTable , there is going to be no compaction kicked >>> in. Any way we can get rid of the tombstones in that row? >>> >>> >>> >>> Userdefined compaction nor nodetool compact had no effect. Any ideas >>> folks? >>> >>> >>> >>> thanks >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >> > -- ———————— Ben Slater Chief Product Officer Instaclustr: Cassandra + Spark - Managed | Consulting | Support +61 437 929 798
