Thanks a lot Alain!! This really cleared a lot of things for me.
Thanks again! Shalom Sagges DBA T: +972-74-700-4035 <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections On Mon, Dec 19, 2016 at 5:58 PM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: > From http://www.uberobert.com/cassandra_gc_grace_disables_hinted_handoff/ > > This is just a quick FYI post as I don't see this documented on the web >> elsewhere. As of now in all versions of Cassandra a gc_grace_seconds setting >> of 0 will disable hinted handoff. Basically to avoid an edge case that >> could cause data to reappear in a cluster (Detailed in Jira >> CASSANDRA-5314 <https://issues.apache.org/jira/browse/CASSANDRA-5314>) >> hints are stored with a TTL of gc_grace_seconds for the keyspace in >> question. A gc_grace_seconds setting of 0 will cause hints to TTL instantly >> and they will never be streamed off when a node comes back up. > > > I did not read the ticket yet, but it might bring some enlightening as > well regarding your question Cody, > > C*heers, > ----------------------- > Alain Rodriguez - @arodream - al...@thelastpickle.com > France > > The Last Pickle - Apache Cassandra Consulting > http://www.thelastpickle.com > > 2016-12-19 16:39 GMT+01:00 Cody Yancey <yan...@uber.com>: > >> >> Cassandra stores hints for the lowest of gc_grace_seconds and >> max_hint_window_in_ms >> >> Was this a tough design decision or just a bug? It is certainly very >> surprising behavior. Everything that I've read leads me to believe that >> gc_grace_seconds was only intended to affect the treatment of *expired* >> data. >> >> Thanks, >> Cody >> >> On Mon, Dec 19, 2016 at 8:10 AM, Alain RODRIGUEZ <arodr...@gmail.com> >> wrote: >> >>> Hi, >>> >>> >>>> - Why setting gc_grace_seconds=0 will disable hints for the table? >>>> >>>> It was the first time I heard about this as well when Alexander told us >>> about that. This read might be helpful http://www.uberobert.com/cassa >>> ndra_gc_grace_disables_hinted_handoff/. Also Alexander I know tested it. >>> >>> *tl;dr*: Cassandra stores hints for the lowest of gc_grace_seconds and >>> max_hint_window_in_ms >>> >>> Still I see no reason not to set gc_grace_seconds to 3 hours as a fix / >>> workaround. Keeping 3 hours of extra data on disk is something you >>> definitely want to be able to do. >>> >>> >>>> - How can an expired TTL record be deleted by Cassandra without >>>> tombstoning or compaction? Aren't SSTables immutable files, and expired >>>> records are removed through compaction? >>>> >>>> >>> This sounds magical to me as well. The only way I am aware of to drop >>> tombstone without compaction is having an entire "SSTable expired" that >>> would be soon be evicted, without compactions. TWCS relies on this property >>> and make a great use of it. Here is Jeff talk about TWCS: >>> https://www.youtube.com/watch?v=PWtekUWCIaw. I believe he mentioned >>> that. >>> >>> >>>> - If I only use TTL for deletion, do I still need gc_grace_seconds >>>> to be bigger than 0? >>>> >>>> >>>> - If I only use TTL for deletion, but use updates as well, do I >>>> need gc_grace_seconds to be bigger than 0? >>>> >>>> >>> Yes, if you care about hints. Anyway, setting gc_grace_seconds to 0 >>> brings more troubles than solutions in many cases. Use the value of >>> max_hint_window_in_ms as a minimal gc_grace_seconds (watch out for the time >>> units in use, do the math ;-) ) >>> >>> Here is a blog I wrote a few months ago about tombstones and deletes >>> http://thelastpickle.com/blog/2016/07/27/about-deletes-and-t >>> ombstones.html. I hope it will give you interesting insight about >>> tombstones, even if you do not care about all the "deletes" part. About >>> TTLs, see http://thelastpickle.com/blog/2016/07/27/about-deletes-a >>> nd-tombstones.html#tombstones-drop. There is no need for you to repair >>> within gc_grace_seconds, but given that "Cassandra stores hints for the >>> lowest of gc_grace_seconds and max_hint_window_in_ms" I would never use a >>> lower value than 3 hours (default max_hint_window_in_ms) for >>> gc_grace_seconds, on any table. >>> >>> C*heers, >>> >>> >>> 2016-12-19 15:07 GMT+01:00 Shalom Sagges <shal...@liveperson.com>: >>> >>>> Thanks for the explanation Matija, but fortunately, that I know. Forgot >>>> to mention that I'm using a multi DC cluster. >>>> I'll try to summarize just the questions I have, because my email was >>>> indeed quite long :-) >>>> >>>> >>>> - Why setting gc_grace_seconds=0 will disable hints for the table? >>>> - How can an expired TTL record be deleted by Cassandra without >>>> tombstoning or compaction? Aren't SSTables immutable files, and expired >>>> records are removed through compaction? >>>> - If I only use TTL for deletion, do I still need gc_grace_seconds >>>> to be bigger than 0? >>>> - If I only use TTL for deletion, but use updates as well, do I >>>> need gc_grace_seconds to be bigger than 0? >>>> >>>> >>>> Thanks! >>>> >>>> >>>> Shalom Sagges >>>> DBA >>>> T: +972-74-700-4035 <+972%2074-700-4035> >>>> <http://www.linkedin.com/company/164748> >>>> <http://twitter.com/liveperson> <http://www.facebook.com/LivePersonInc> We >>>> Create Meaningful Connections >>>> >>>> >>>> >>>> On Mon, Dec 19, 2016 at 2:39 PM, Matija Gobec <matija0...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> gc_grace_seconds is used to maintain data consistency in some failure >>>>> scenarios. When manually deleting data that action creates tombstones >>>>> which >>>>> are kept for that defined period before being compacted. If one of the >>>>> replica nodes is down while deleting data and it gets back up after the >>>>> gc_grace_seconds defined period your previously delete data will reappear >>>>> (ghost data). As it is stated in datastax documentation on a single node >>>>> you can set gc_grace_seconds to 0 and you can do the same for tables that >>>>> contain only data with TTL. In the mentioned failure scenario your downed >>>>> node will have data with TTL information and no data inconsistency will >>>>> happen. >>>>> >>>>> On Mon, Dec 19, 2016 at 1:00 PM, Shalom Sagges <shal...@liveperson.com >>>>> > wrote: >>>>> >>>>>> Hi Everyone, >>>>>> >>>>>> I was reading a blog on TWCS by Alex Dejanovski from The Last Pickle ( >>>>>> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html) >>>>>> >>>>>> When I got to the comments section, I didn't understand why setting >>>>>> gc_grace_seconds to 0 will disable hints for the associated table: >>>>>> *"It is a very good point that gc_grace_seconds shouldn't be lowered >>>>>> too much as its impact on hinted handoff is not a well known fact, and >>>>>> using a value of 0 will purely disable hints on the table."* >>>>>> >>>>>> When I tried to read some more about deletes and TTLs, I got to a >>>>>> Datastax documentation https://docs.datastax.com/en/cassandra/3.0/cas >>>>>> sandra/dml/dmlAboutDeletes.html >>>>>> stating the following: >>>>>> >>>>>> *Cassandra allows you to set a default_time_to_live property for an >>>>>> entire table. Columns and rows marked with regular TTLs are processed as >>>>>> described above; but when a record exceeds the table-level TTL, Cassandra >>>>>> deletes it immediately, without tombstoning or compaction.* >>>>>> >>>>>> Which got me a bit more confused. >>>>>> So I hope someone can shed some light on some questions I've got: >>>>>> >>>>>> >>>>>> - Why setting gc_grace_seconds=0 will disable hints for the table? >>>>>> - How can an expired TTL record be deleted by Cassandra without >>>>>> tombstoning or compaction? Aren't SSTables immutable files, and >>>>>> expired >>>>>> records are removed through compaction? >>>>>> - If I only use TTL for deletion, do I still need >>>>>> gc_grace_seconds to be bigger than 0? >>>>>> - If I only use TTL for deletion, but use updates as well, do I >>>>>> need gc_grace_seconds to be bigger than 0? >>>>>> >>>>>> >>>>>> Sorry for all those questions, I'm just really confused from all the >>>>>> TTL/tombstones subject (still a newbie). >>>>>> >>>>>> Thanks a lot! >>>>>> >>>>>> >>>>>> Shalom Sagges >>>>>> DBA >>>>>> T: +972-74-700-4035 <+972%2074-700-4035> >>>>>> <http://www.linkedin.com/company/164748> >>>>>> <http://twitter.com/liveperson> >>>>>> <http://www.facebook.com/LivePersonInc> We Create Meaningful >>>>>> Connections >>>>>> >>>>>> >>>>>> >>>>>> This message may contain confidential and/or privileged information. >>>>>> If you are not the addressee or authorized to receive this on behalf >>>>>> of the addressee you must not use, copy, disclose or take action based on >>>>>> this message or any information herein. >>>>>> If you have received this message in error, please advise the sender >>>>>> immediately by reply email and delete this message. Thank you. >>>>>> >>>>> >>>>> >>>> >>>> This message may contain confidential and/or privileged information. >>>> If you are not the addressee or authorized to receive this on behalf of >>>> the addressee you must not use, copy, disclose or take action based on this >>>> message or any information herein. >>>> If you have received this message in error, please advise the sender >>>> immediately by reply email and delete this message. Thank you. >>>> >>> >>> >> > -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.
