Re: Documentation about TTL and tombstones

2024-03-18 Thread Sebastian Marsching
> It's actually correct to do it how it is today. > Insertion date does not matter, what matters is the time after tombstones are > supposed to be deleted. > If the delete got to all nodes, sure, no problem, but if any of the nodes > didn't get the delete, and yo

Re: Documentation about TTL and tombstones

2024-03-17 Thread Gil Ganz
It's actually correct to do it how it is today. Insertion date does not matter, what matters is the time after tombstones are supposed to be deleted. If the delete got to all nodes, sure, no problem, but if any of the nodes didn't get the delete, and you would get rid of the tombstones before

Re: Documentation about TTL and tombstones

2024-03-16 Thread Sebastian Marsching
ht that gc_grace_seconds is all about being able to repair the table before tombstones are removed, so that deleted data cannot repappear. But when the data has a TTL, it should not matter whether the original data ore the tombstone is synchronized as part of the repair process. After all, the o

Re: Documentation about TTL and tombstones

2024-03-16 Thread Gil Ganz
ssandra.apache.org/doc/4.1/cassandra/operating/compaction/index.html#ttl > It mention that it creates a tombstone when data expired, how does it > possible without writing to the tombstone on the table ? I thought TTL > doesn't create tombstones since the ttl is present together with

Re: Documentation about TTL and tombstones

2024-03-14 Thread Sebastian Marsching
doesn't create tombstones since the ttl is present together with the write > time timestmap > at the row level If you read carefully, you will notice that no tombstone is created and instead the data is *converted* into a tombstone. So, after the TTL has expired, the inserted data effectivel

Documentation about TTL and tombstones

2024-03-14 Thread Jean Carlo
tombstones since the ttl is present together with the write time timestmap at the row level Greetings Jean Carlo "The best way to predict the future is to invent it" Alan Kay

Re: Removing row level tombstones

2022-03-23 Thread Bowen Song
rid of the tombstones and turn two SSTables into one, you will need to ensure the SSTables are either both repaired or both unrepaired. On 23/03/2022 16:43, MyWorld wrote: Hi all, I have a table (with default gc_grace_seconds) where we have some row tombstones created. Currently there are 2

Removing row level tombstones

2022-03-23 Thread MyWorld
Hi all, I have a table (with default gc_grace_seconds) where we have some row tombstones created. Currently there are 2 sstables for this. When we check the sstablemetadata, we found Estimated droppable tombstones: 0.689(for sstable1) and 1.094(for sstable2) After executing nodetool compact

Re: Tombstones? 4.0.1

2021-10-25 Thread Jeff Jirsa
), then you can read (date, 7) and (date, 996) and everything else in concurrent async queries, or something else that gives you deterministic partitioning so you're not walking past all of those dead tombstones. Reading with a naive SELECT with no WHERE is going to be perhaps the least performant

Re: Tombstones? 4.0.1

2021-10-25 Thread Joe Obernberger
results in an unusable table. I'm using Cassandra to de-duplicate data and that's not a good use case for it. -Joe On 10/25/2021 6:51 PM, Jeff Jirsa wrote: The tombstone threshold is "how many tombstones are encountered within a single read command", and the default is something li

Re: Tombstones? 4.0.1

2021-10-25 Thread Jeff Jirsa
The tombstone threshold is "how many tombstones are encountered within a single read command", and the default is something like 100,000 ( https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1293-L1294 ) Deletes are not forbidden, but you have to read in such a way that

Re: Tombstones? 4.0.1

2021-10-25 Thread Dmitry Saprykin
10/14/2021 3:38 PM, Joe Obernberger wrote: > > I'm not sure if tombstones is the issue; is it? Grace is set to 10 days, > that time has not passed yet. > > -Joe > On 10/14/2021 1:37 PM, James Brown wrote: > > What is gc_grace_seconds set to on the table? Once that passe

Re: Tombstones? 4.0.1

2021-10-25 Thread Joe Obernberger
rows, after deleting them, I can no longer do a select from the table as it times out. Thank you! -Joe On 10/14/2021 3:38 PM, Joe Obernberger wrote: I'm not sure if tombstones is the issue; is it?  Grace is set to 10 days, that time has not passed yet. -Joe On 10/14/2021 1:37 PM, James

Re: Tombstones? 4.0.1

2021-10-14 Thread Joe Obernberger
I'm not sure if tombstones is the issue; is it?  Grace is set to 10 days, that time has not passed yet. -Joe On 10/14/2021 1:37 PM, James Brown wrote: What is gc_grace_seconds set to on the table? Once that passes, you can do `nodetool scrub` to more emphatically remove tombstones... On Thu

Re: Tombstones? 4.0.1

2021-10-14 Thread James Brown
What is gc_grace_seconds set to on the table? Once that passes, you can do `nodetool scrub` to more emphatically remove tombstones... On Thu, Oct 14, 2021 at 8:49 AM Joe Obernberger < joseph.obernber...@gmail.com> wrote: > Hi all - I have a table where I've needed to delete a numbe

Tombstones? 4.0.1

2021-10-14 Thread Joe Obernberger
    Average live cells per slice (last five minutes): 73.53164556962025     Maximum live cells per slice (last five minutes): 5722     Average tombstones per slice (last five minutes): 1.0     Maximum tombstones per slice (last five minutes): 1

Re: Do TTLs generate tombstones?

2021-07-20 Thread Jeff Jirsa
Both are true If you have a cell that exists in a data file that has expired, but not yet been compacted away, the read path will treat it like a delete/tombstone. Imagine you write a value (no tombstone) - set val=A 2 minutes later, you write a new value (set val=B) on top of it with an

Re: How to remove tombstones in a levelled compaction table in Cassandra 2.1.16?

2021-07-06 Thread manish khandelwal
Thanks Jeff and Vytenis. Jeff, could you explain what do you mean by If you just pipe all of your sstables to user defined compaction jmx > endpoint one at a time you’ll purge many of the tombstones as long as you > don’t have a horrific data model. Regards Manish On Wed, Jul 7, 2021 a

Re: How to remove tombstones in a levelled compaction table in Cassandra 2.1.16?

2021-07-06 Thread Jeff Jirsa
In 2.1 the only option is enable auto compaction or queue up manual user defined compaction If you just pipe all of your sstables to user defined compaction jmx endpoint one at a time you’ll purge many of the tombstones as long as you don’t have a horrific data model. > On Jul 6, 2

Re: How to remove tombstones in a levelled compaction table in Cassandra 2.1.16?

2021-07-06 Thread vytenis silgalis
ted droppable tombstones: 1.0 ^ this is an extreme example but anything greater than .2 on a 10+ day sstable is a problem. By default the unchecked_tombstone_compaction setting is false which will lead to tombstones staying around if a partition spans multiple sstables (which may happen with LCS o

Re: How to remove tombstones in a levelled compaction table in Cassandra 2.1.16?

2021-07-05 Thread manish khandelwal
gt;> >> I suspect due to disabling of autocompaction lots of tombstones got >> created. And now while reading those are creating issues and queries are >> getting timed out. Am I right in my thinking? What is the possible way to >> get out of this? >> >> I thoug

Re: How to remove tombstones in a levelled compaction table in Cassandra 2.1.16?

2021-07-05 Thread Kane Wilson
In one of our LCS table auto compaction was disabled. Now after years of > run, range queries using spark-cassandra-connector are failing. Cassandra > version is 2.1.16. > > I suspect due to disabling of autocompaction lots of tombstones got > created. And now while reading tho

How to remove tombstones in a levelled compaction table in Cassandra 2.1.16?

2021-07-05 Thread manish khandelwal
In one of our LCS table auto compaction was disabled. Now after years of run, range queries using spark-cassandra-connector are failing. Cassandra version is 2.1.16. I suspect due to disabling of autocompaction lots of tombstones got created. And now while reading those are creating issues

best setup of tombstones cleanup over a "wide" table (was: efficient delete over a "wide" table?)

2020-09-05 Thread Attila Wind
ranges can not be done... Just one more question left in this case... As this way we will have lots of row tombstones generated over this "wide" table What would be your recommended table setup here (in terms of gc_grace_seconds, compaction, compression, etc etc)? Currently we have default

Re: tombstones - however there are no deletes

2020-08-21 Thread Alex Ott
Btw, if you seen the number of tombstones that is a multiply of number of scanned rows, like in your case - that’s a explicit signal of either null inserts, or non frozen collections... On Fri 21. Aug 2020 at 20:21, Attila Wind wrote: > > > > > > > > > > > right

Re: tombstones - however there are no deletes

2020-08-21 Thread Attila Wind
isting one - use UNSET instead of null On Fri, Aug 21, 2020 at 10:45 AM Attila Wind wrote: Thanks a lot! I will process every pointers you gave - appreciated! 1. we do have collection column in that table but that is (we have only 1 column) a frozen Map - so I guess "Tomb

Re: tombstones - however there are no deletes

2020-08-21 Thread Alex Ott
AM Attila Wind wrote: > Thanks a lot! I will process every pointers you gave - appreciated! > > 1. we do have collection column in that table but that is (we have only 1 > column) a frozen Map - so I guess "Tombstones are also implicitly created > any time you insert or up

Re: tombstones - however there are no deletes

2020-08-21 Thread Attila Wind
Thanks a lot! I will process every pointers you gave - appreciated! 1. we do have collection column in that table but that is (we have only 1 column) a frozen Map - so I guess "Tombstones are also implicitly created any time you insert or update a row which has an (unfrozen) collection c

Re: tombstones - however there are no deletes

2020-08-21 Thread Oleksandr Shulgin
On Fri, Aug 21, 2020 at 9:43 AM Tobias Eriksson wrote: > Isn’t it so that explicitly setting a column to NULL also result in a > tombstone > True, thanks for pointing that out! Then as mentioned the use of list,set,map can also result in tombstones > > See > https://w

Re: tombstones - however there are no deletes

2020-08-21 Thread Tobias Eriksson
Isn’t it so that explicitly setting a column to NULL also result in a tombstone Then as mentioned the use of list,set,map can also result in tombstones See https://www.instaclustr.com/cassandra-collections-hidden-tombstones-and-how-to-avoid-them/ -Tobias From: Oleksandr Shulgin Reply to: "

Re: tombstones - however there are no deletes

2020-08-21 Thread Oleksandr Shulgin
te path - however those tables are >counter tables... when we mass-read them into memory we also go with >"select *" logic reading up tons of rows. The point is we never saw such a >warning for these counter tables however we are handling them same >fashion... ok

Re: tombstones - however there are no deletes

2020-08-21 Thread Alex Ott
Tombstones could be not only generated by deletes. this happens when you: - When insert or full update of a non-frozen collection occurs, such as replacing the value of the column with another value like the UPDATE table SET field = new_value …, Cassandra inserts a tombstone marker

tombstones - however there are no deletes

2020-08-20 Thread Attila Wind
Hi Cassandra Gurus, Recently I captured a very interesting warning in the logs saying 2020-08-19 08:08:32.492 [cassandra-client-keytiles_data_webhits-nio-worker-2] WARN com.datastax.driver.core.RequestHandler - Query '[3 bound values] select * from visit_sess ion_by_start_time_v4 where

Re: To find top 10 tables with top 10 sstables per read and top 10 tables with top tombstones per read ?

2020-03-16 Thread Hossein Ghiyasi Mehr
I think it's better to change you solution for your purpose. Tables, partitions , tombstones, statistics, sstables, query and monitoring in Cassandra are different! You need noSQL concept before doing anything. Best regards, *---* *VafaTech <h

Re: To find top 10 tables with top 10 sstables per read and top 10 tables with top tombstones per read ?

2020-03-16 Thread Erick Ramirez
There's no out-of-the-box way of doing it so you'll need to scrape the output of nodetool tablestats on each node to get the top tables and top partitions. You should be able to do it with some simple shell scripting. Cheers!

To find top 10 tables with top 10 sstables per read and top 10 tables with top tombstones per read ?

2020-03-16 Thread Kiran mk
Hi All, Is there a way to find top 10 tables with top 10 sstables per read and top 10 tables with top tombstones per read in Cassandra? As In Opscenter everytime we have to select the tables to find whats the tombstones per read. There are chances that we might miss considering the tables which

RE: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-30 Thread Steinmaurer, Thomas
Subject: Re: Cassandra going OOM due to tombstones (heapdump screenshots provided) It looks like the number of tables is the problem, with 5,000 - 10,000 tables, that is way above the recommendations. Take a look here: https://docs.datastax.com/en/dse-planning/doc/planning/planningAntiPatterns.html

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-29 Thread Erick Ramirez
> > It looks like the number of tables is the problem, with 5,000 - 10,000 > tables, that is way above the recommendations. > Take a look here: > https://docs.datastax.com/en/dse-planning/doc/planning/planningAntiPatterns.html#planningAntiPatterns__AntiPatTooManyTables > This suggests that 5-10GB

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-29 Thread Hannu Kröger
It means that you are using 5-10GB of memory just to hold information about tables. Memtables hold the data that is written to the database until those are flushed to the disk, and those happen when memory is low or some other threshold is reached. Every table will have a memtable that takes

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-29 Thread Behroz Sikander
It doesn't seem to be the problem but I do not have deep knowledge of C* internals. When do memtable come into play? Only at startup? - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands,

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-29 Thread Paul Chandler
Hi Behroz, It looks like the number of tables is the problem, with 5,000 - 10,000 tables, that is way above the recommendations. Take a look here: https://docs.datastax.com/en/dse-planning/doc/planning/planningAntiPatterns.html#planningAntiPatterns__AntiPatTooManyTables

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-29 Thread Hannu Kröger
IIRC there is an overhead of about 1MB per table which you have about 5000-1 => 5GB - 10GB overhead of just having that many tables. To me it looks like that you need to increase the heap size and later potentially work on the data models to have less tables. Hannu > On 29. Jan 2020, at

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-29 Thread Behroz Sikander
>> If it's after the host comes online and it's hint replay from the other hosts, you probably want to throttle hint replay significantly on the rest of the cluster. Whatever your hinted handoff throttle is, consider dropping it by 50-90% to work around whichever of those two problems it is. This

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-29 Thread Behroz Sikander
>> Startup would replay commitlog, which would re-materialize all of those mutations and put them into the memtable. The memtable would flush over time to disk, and clear the commitlog. >From our observation, the node is already online and it seems to be happening >after the commit log replay

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-29 Thread Behroz Sikander
>> Some environment details like Cassandra version, amount of physical RAM, JVM configs (heap and others), and any other non-default cassandra.yaaml configs would help. The amount of data, number of keyspaces & tables, since you mention "clients", would also be helpful for people to suggest

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-24 Thread Reid Pinchback
-To: "user@cassandra.apache.org" Date: Friday, January 24, 2020 at 12:09 PM To: cassandra Subject: Re: Cassandra going OOM due to tombstones (heapdump screenshots provided) Message from External Sender Ah, I focused too much on the literal meaning of startup. If it's happening

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-24 Thread Jeff Jirsa
Ah, I focused too much on the literal meaning of startup. If it's happening JUST AFTER startup, it's probably getting flooded with hints from the other hosts when it comes online. If that's the case, it may be just simply overrunning the memtable, or it may be a deadlock like

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-24 Thread Jeff Jirsa
6 GB of mutations on heap Startup would replay commitlog, which would re-materialize all of those mutations and put them into the memtable. The memtable would flush over time to disk, and clear the commitlog. It looks like PERHAPS the commitlog replay is faster than the memtable flush, so you're

Re: Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-24 Thread Michael Shuler
Some environment details like Cassandra version, amount of physical RAM, JVM configs (heap and others), and any other non-default cassandra.yaaml configs would help. The amount of data, number of keyspaces & tables, since you mention "clients", would also be helpful for people to suggest

Cassandra going OOM due to tombstones (heapdump screenshots provided)

2020-01-24 Thread Behroz Sikander
We recently had a lot of OOM in C* and it was generally happening during startup. We took some heap dumps but still cannot pin point the exact reason. So, we need some help from experts. Our clients are not explicitly deleting data but they have TTL enabled. C* details: > show version [cqlsh

Re: Tombstones not getting purged

2019-06-20 Thread Léo FERLIN SUTTON
Thank you for the information ! On Thu, Jun 20, 2019 at 9:50 AM Alexander Dejanovski wrote: > Léo, > > if a major compaction isn't a viable option, you can give a go at > Instaclustr SSTables tools to target the partitions with the most > tombstones : > https://github.com/ins

Re: Tombstones not getting purged

2019-06-20 Thread Alexander Dejanovski
Léo, if a major compaction isn't a viable option, you can give a go at Instaclustr SSTables tools to target the partitions with the most tombstones : https://github.com/instaclustr/cassandra-sstable-tools/tree/cassandra-2.2#ic-purge It generates a report like this: Summary

Re: Tombstones not getting purged

2019-06-20 Thread Alexander Dejanovski
My bad on date formatting, it should have been : %Y/%m/%d Otherwise the SSTables aren't ordered properly. You have 2 SSTables that claim to cover timestamps from 1940 to 2262, which is weird. Aside from that, you have big overlaps all over the SSTables, so that's probably why your tombstones

Re: Tombstones not getting purged

2019-06-20 Thread Léo FERLIN SUTTON
eve they have changed. I do not remember the sstable name but the "last modified" has changed recently for these tables. > Could you run the following command to list SSTables and provide us the > output? It will display both their timestamp ranges along with the > estimated

Re: Tombstones not getting purged

2019-06-19 Thread Alexander Dejanovski
their timestamp ranges along with the estimated droppable tombstones ratio. for f in *Data.db; do meta=$(sstablemetadata -gc_grace_seconds 259200 $f); echo $(date --date=@$(echo "$meta" | grep Maximum\ time | cut -d" " -f3| cut -c 1-10) '+%m/%d/%Y %H:%M:%S') $(date --date=@$(echo "$m

Re: Tombstones not getting purged

2019-06-19 Thread Jeff Jirsa
Probably overlapping sstables Which compaction strategy? > On Jun 19, 2019, at 9:51 PM, Léo FERLIN SUTTON > wrote: > > I have used the following command to check if I had droppable tombstones : > `/usr/bin/sstablemetadata --gc_grace_seconds 259200 > /var/lib/cassandra/da

Tombstones not getting purged

2019-06-19 Thread Léo FERLIN SUTTON
I have used the following command to check if I had droppable tombstones : `/usr/bin/sstablemetadata --gc_grace_seconds 259200 /var/lib/cassandra/data/stats/tablename/md-sstablename-big-Data.db` I checked every sstable in a loop and had 4 sstables with droppable tombstones : ``` Estimated

RE: TWCS Compactions & Tombstones

2019-03-27 Thread Nick Hatfield
Awesome, thanks again! From: Jeff Jirsa [mailto:jji...@gmail.com] Sent: Wednesday, March 27, 2019 1:36 PM To: cassandra Subject: Re: TWCS Compactions & Tombstones You would need to swap your class from the com.jeffjirsa variant (probably from 2.1 / 2.2) to the official TWCS class.

Re: TWCS Compactions & Tombstones

2019-03-27 Thread Jeff Jirsa
een this yet. So we have this > enabled, I guess it will just take time to finally chew through it all? > > > > *From:* Jeff Jirsa [mailto:jji...@gmail.com] > *Sent:* Tuesday, March 26, 2019 9:41 PM > *To:* user@cassandra.apache.org > *Subject:* Re: TWCS Compactions &

RE: TWCS Compactions & Tombstones

2019-03-27 Thread Nick Hatfield
& Tombstones Or Upgrade to a version with https://issues.apache.org/jira/browse/CASSANDRA-13418 and enable that feature -- Jeff Jirsa On Mar 26, 2019, at 6:23 PM, Rahul Singh mailto:rahul.xavier.si...@gmail.com>> wrote: What's your timewindow? Roughly how much data is in each window? If y

Re: TWCS Compactions & Tombstones

2019-03-26 Thread James Brown
also had a read_repair = 0.1 and after correcting this to 0.0 I can clearly > see the affects over time on the new sstables. However, I still have old > sstables that date back some time last year, and I need to remove them: > > Max: 09/05/2018 Min: 09/04/2018 Estimated droppable tombstones:

RE: TWCS Compactions & Tombstones

2019-03-26 Thread Nick Hatfield
the sstable data so that I can verify that it is old data, other than by the min / max timestamps? Thanks for your help From: Rahul Singh [mailto:rahul.xavier.si...@gmail.com] Sent: Tuesday, March 26, 2019 9:24 PM To: user Subject: Re: TWCS Compactions & Tombstones What's your timewindow? Roughly

Re: TWCS Compactions & Tombstones

2019-03-26 Thread Jeff Jirsa
correcting this to 0.0 I can clearly see the >> affects over time on the new sstables. However, I still have old sstables >> that date back some time last year, and I need to remove them: >> >> Max: 09/05/2018 Min: 09/04/2018 Estimated droppable tombstones: &g

Re: TWCS Compactions & Tombstones

2019-03-26 Thread Rahul Singh
and after correcting this to 0.0 I can clearly > see the affects over time on the new sstables. However, I still have old > sstables that date back some time last year, and I need to remove them: > > Max: 09/05/2018 Min: 09/04/2018 Estimated droppable tombstones: > 0.883205790993204613

TWCS Compactions & Tombstones

2019-03-26 Thread Nick Hatfield
that date back some time last year, and I need to remove them: Max: 09/05/2018 Min: 09/04/2018 Estimated droppable tombstones: 0.883205790993204613G Mar 26 11:34 mc-254400-big-Data.db What is the best way to do this? This is on a production system so any help would be greatly appreciated

can i delete a sstable with Estimated droppable tombstones > 1, manually?

2019-03-19 Thread onmstester onmstester
Running: SSTablemetadata /THE_KEYSPACE_DIR/mc-1421-big-Data.db result was: Estimated droppable tombstones: 1.2 Having STCS and data disk usage of 80% (do not have enough free space for normal compaction), Is it OK to just: 1. stop Cassandra, 2. delete mc-1421* and then 3. start Cassandra

RE: tombstones threshold warning

2019-02-24 Thread Kenneth Brotman
Hi Ayub, The counting of tombstones is sort of involved. The JIRA tickets contain the discussions and where things are at. Several tickets to study. Maybe start with CASSANDRA-14149: https://issues.apache.org/jira/browse/CASSANDRA-14149 and follow the discussions back to previous tickets

Re: Tombstones in memtable

2019-02-24 Thread Jeff Jirsa
. IF you’re doing this, it may be a bit concerning, because it’s having to skip past a ton of tombstones on the read path - which is expensive; this is why the metric exists, but you’ve said you’re not doing this. You’re not going to be able to stop reading tombstones unless you can stop the app

Re: Tombstones in memtable

2019-02-24 Thread Rahul Reddy
Thanks Jeff. I'm trying to figure out why the tombstones scans are happening if possible eliminate it. On Sat, Feb 23, 2019, 10:50 PM Jeff Jirsa wrote: > G1GC with an 8g heap may be slower than CMS. Also you don’t typically set > new gen size on G1. > > Again though - what pro

Re: Tombstones in memtable

2019-02-23 Thread Rahul Reddy
From:* Rahul Reddy [mailto:rahulreddy1...@gmail.com] > *Sent:* Saturday, February 23, 2019 7:26 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Tombstones in memtable > > > > ```jvm setting > > > > -XX:+UseThreadPriorities > > -XX:ThreadPriorityPolicy=42 > > -XX:+

RE: Tombstones in memtable

2019-02-23 Thread Kenneth Brotman
When the CPU utilization spikes from 5-10% to 50%, how many nodes does it happen to at the same time? From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] Sent: Saturday, February 23, 2019 7:26 PM To: user@cassandra.apache.org Subject: Re: Tombstones in memtable ```jvm setting -XX

Re: Tombstones in memtable

2019-02-23 Thread Jeff Jirsa
G1GC with an 8g heap may be slower than CMS. Also you don’t typically set new gen size on G1. Again though - what problem are you solving here? If you’re serving reads and sitting under 50% cpu, it’s not clear to me what you’re trying to fix. Tombstones scanned won’t matter for your table, so

Re: Tombstones in memtable

2019-02-23 Thread Rahul Reddy
2 PM, Jeff Jirsa wrote: >> >> You’ll only ever have one tombstone per read, so your load is based on >> normal read rate not tombstones. The metric isn’t wrong, but it’s not >> indicative of a problem here given your data model. >> >> You’re using STCS do you may

Re: Tombstones in memtable

2019-02-23 Thread Rahul Reddy
can’t really do that > via email. Cassandra-8150 has some pointers. > > -- > Jeff Jirsa > > > On Feb 23, 2019, at 6:52 PM, Jeff Jirsa wrote: > > You’ll only ever have one tombstone per read, so your load is based on > normal read rate not tombstones. The metric isn’t wrong, but

Re: Tombstones in memtable

2019-02-23 Thread Jeff Jirsa
probably win some with basic perf and GC tuning, but can’t really do that via email. Cassandra-8150 has some pointers. -- Jeff Jirsa > On Feb 23, 2019, at 6:52 PM, Jeff Jirsa wrote: > > You’ll only ever have one tombstone per read, so your load is based on normal > read rate no

Re: Tombstones in memtable

2019-02-23 Thread Jeff Jirsa
You’ll only ever have one tombstone per read, so your load is based on normal read rate not tombstones. The metric isn’t wrong, but it’s not indicative of a problem here given your data model. You’re using STCS do you may be reading from more than one sstable if you update column2 for a given

RE: Tombstones in memtable

2019-02-23 Thread Kenneth Brotman
Rahul, You wrote that during peak hours you only have a couple hundred inserts per node so now I’m not sure why the default settings wouldn’t have worked just fine. I sense there is more to the story. What else could explain those tombstones? From: Rahul Reddy [mailto:rahulreddy1

Re: Tombstones in memtable

2019-02-23 Thread Rahul Reddy
Do you see anything wrong with this metric. metric to scan tombstones increase(cassandra_Table_TombstoneScannedHistogram{keyspace="mykeyspace",Table="tablename",function="Count"}[5m]) And sametime CPU Spike to 50% whenever I see high tombstone alert. On Sat, Fe

Re: Tombstones in memtable

2019-02-23 Thread Jeff Jirsa
ed table and took tsstabledump > grep -i '"expired" : true' SSTables.txt|wc -l > 16439 > grep -i '"expired" : false' SSTables.txt |wc -l > 2657 > > ttl is 4 hours. > > INSERT INTO keyspace."TABLE_NAME" ("column1", "column2"

RE: Tombstones in memtable

2019-02-23 Thread Kenneth Brotman
-patterns-queues-and-queue-like-datasets Kenneth Brotman From: Jeff Jirsa [mailto:jji...@gmail.com] Sent: Saturday, February 23, 2019 4:47 PM To: user@cassandra.apache.org Subject: Re: Tombstones in memtable Would also be good to see your schema (anonymized if needed) and the select

Re: Tombstones in memtable

2019-02-23 Thread Rahul Reddy
" ("column1", "column2") VALUES (?, ?) USING TTL(4hours) ?'; SELECT * FROM keyspace."TABLE_NAME" WHERE "column1" = ?'; metric to scan tombstones increase(cassandra_Table_TombstoneScannedHistogram{keyspace="mykeyspace",Table="tab

Re: Tombstones in memtable

2019-02-23 Thread Jeff Jirsa
22, 2019, 11:46 AM Jeff Jirsa wrote: >> If all of your data is TTL’d and you never explicitly delete a cell without >> using s TTL, you can probably drop your GCGS to 1 hour (or less). >> >> Which compaction strategy are you using? You need a way to clear out those &g

Re: Tombstones in memtable

2019-02-23 Thread Jeff Jirsa
;> >> Which compaction strategy are you using? You need a way to clear out those >> tombstones. There exist tombstone compaction sub properties that can help >> encourage compaction to grab sstables just because they’re full of >> tombstones which will probably help you.

Re: Tombstones in memtable

2019-02-23 Thread Rahul Reddy
u using? You need a way to clear out those > tombstones. There exist tombstone compaction sub properties that can help > encourage compaction to grab sstables just because they’re full of > tombstones which will probably help you. > > > -- > Jeff Jirsa > > > On Feb 22,

Re: tombstones threshold warning

2019-02-23 Thread Ayub M
Thanks Ken, further investigating what I found is the tombstones which I am seeing are from null values in the collection objects. Tombstones are also inserted into when initial collection values are inserted but seems like they are not counted towards threshold warning and do not show up

Re: Tombstones in memtable

2019-02-22 Thread Jeff Jirsa
If all of your data is TTL’d and you never explicitly delete a cell without using s TTL, you can probably drop your GCGS to 1 hour (or less). Which compaction strategy are you using? You need a way to clear out those tombstones. There exist tombstone compaction sub properties that can help

RE: Tombstones in memtable

2019-02-22 Thread Kenneth Brotman
Can we see the histogram? Why wouldn’t you at times have that many tombstones? Makes sense. Kenneth Brotman From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] Sent: Thursday, February 21, 2019 7:06 AM To: user@cassandra.apache.org Subject: Tombstones in memtable We have small

Tombstones in memtable

2019-02-21 Thread Rahul Reddy
We have small table records are about 5k . All the inserts comes as 4hr ttl and we have table level ttl 1 day and gc grace seconds has 3 hours. We do 5k reads a second during peak load. During the peak load seeing Alerts for tomstone scanned histogram reaching million. Cassandra version 3.11.1.

RE: tombstones threshold warning

2019-02-19 Thread Kenneth Brotman
There is another good article called Common Problems with Cassandra Tombstones by Alla Babkina at https://opencredo.com/blogs/cassandra-tombstones-common-issues/ . It says interesting stuff like: 1. You can get tombstones without deleting anything 2. Inserting null values

RE: tombstones threshold warning

2019-02-19 Thread Kenneth Brotman
Hi Ayub, Is everything flushing to SSTables? It has to be somewhere right? So is it in the memtables? Or is it that there are tombstones that are sometimes detected and sometimes not detected as described in the very detailed article on The Last Pickle by Alex Dejanovski called

tombstones threshold warning

2019-02-16 Thread Ayub M
tombstones for the collection columns when they were inserted. Does collection cell tombstones get counted as tombstones cells in the warning displayed? Did a small test to see if collection tombstones are counted as tombstones and it does not seem so. So wondering where are those tombstones

Re: Cassandra collection tombstones

2019-01-28 Thread Jeff Jirsa
confirm, range tombstones that we are talking about here is not > related to this jira: https://issues.apache.org/jira/browse/CASSANDRA-14861 ? > > Thanks a lot.

Re: Cassandra collection tombstones

2019-01-28 Thread Ahmed Eljami
Hi Alain, Just to confirm, range tombstones that we are talking about here is not related to this jira: https://issues.apache.org/jira/browse/CASSANDRA-14861 ? Thanks a lot.

Re: Cassandra collection tombstones

2019-01-28 Thread Alain RODRIGUEZ
e to provide > a -g or --gc-grace-seconds parameter. I am not sure where the "always > wrong" comes in as the quantity of data thats being shadowed is not what > its tracking (although it would be more meaningful for single sstable > compactions if it did), just when tombsto

Re: Cassandra collection tombstones

2019-01-27 Thread Ayub M
izeCell Count [root@ip-xxx-xxx-xxx-xxx nmtest-e1302500201d11e983bb693c02c04c62]# sstablemetadata --gc_grace_seconds 86400 mc-11-big-Data.db | grep "Estimated tombstone drop times" -A4 Estimated tombstone drop times: 1548577440: 1 Count Row SizeCell

Re: Cassandra collection tombstones

2019-01-25 Thread Chris Lohfink
ovide a -g or --gc-grace-seconds parameter. I am not sure where the "always wrong" comes in as the quantity of data thats being shadowed is not what its tracking (although it would be more meaningful for single sstable compactions if it did), just when tombstones can be purged. Chris

Re: Cassandra collection tombstones

2019-01-25 Thread Alain RODRIGUEZ
Hello, I think you might be inserting on the top of an existing collection, implicitly, Cassandra creates a range tombstone. Cassandra does not update/delete data, it always inserts (data or tombstone). Then eventually compaction merges the data and evict the tombstones. Thus, when overwriting

Cassandra collection tombstones

2019-01-24 Thread Ayub M
I have created a table with a collection. Inserted a record and took sstabledump of it and seeing there is range tombstone for it in the sstable. Does this tombstone ever get removed? Also when I run sstablemetadata on the only sstable, it shows "Estimated droppable tombstones" as 0.5&q

Re: [EXTERNAL] Howto avoid tombstones when inserting NULL values

2019-01-09 Thread Jonathan Haddad
> I’m still not sure if having tombstones vs. empty values / frozen UDTs will have the same results. When in doubt, benchmark. Good luck, Jon On Wed, Jan 9, 2019 at 3:02 PM Tomas Bartalos wrote: > Loosing atomic updates is a good point, but in my use case its not a > problem, since

Re: [EXTERNAL] Howto avoid tombstones when inserting NULL values

2019-01-09 Thread Tomas Bartalos
Loosing atomic updates is a good point, but in my use case its not a problem, since I always overwrite the whole record (no partitial updates). I’m still not sure if having tombstones vs. empty values / frozen UDTs will have the same results. When I update one row with 10 null columns

  1   2   3   4   5   6   >