Re: [DISCUSS] CASSANDRA-13704 Safer handling of out of range tokens

[Jeff Jirsa]

On Sep 12, 2024, at 12:22 PM, J. D. Jordan <jeremiah.jor...@gmail.com> wrote:  I have lost sleep (and data) over this multiple times in the past few months, that was only recently tracked down to this exact scenario. +1 for including it in all active releases and enabling the failure of the writes on “wrong” nodes by default. I haven’t looked at the patch, but as long as only one node of three gets the data by mistake and fails, I would expect a LOCAL_QUORUM write with two ack one fail to still succeed? Yes. And most people won’t notice that. It’s just instantly safe and everyone will be happy.  What people will notice is that things like concurrent overlapping bootstrap will start failing because sstables will be the wrong range . Or repair during range movements.  -Jeremiah On Sep 12, 2024, at 1:47 PM, Jordan West <jw...@apache.org> wrote:  I think folks not losing sleep over this are only in that position because they don’t know it’s happening. Like Brandon said, ignorance is bliss (but it’s a false bliss).   Very few users do the work necessary to detect data loss outside the obvious paths. I agree with Caleb, if we log and give them no means to remediate we are giving them nightmares with no recourse. While failed writes will be a surprise it’s the correct solution because it’s the only one that prevents data loss which we should always strive to get rid of.  Jordan On Thu, Sep 12, 2024 at 11:31 Caleb Rackliffe <calebrackli...@gmail.com> wrote: We aren’t counting on users to read NEWS.txt. That’s the point. We’re saying we’re going to make things safer, as they should always have been, and if someone out there has tooling that somehow allows them to avoid the risks, they can disable rejection. > On Sep 12, 2024, at 1:21 PM, Brandon Williams <dri...@gmail.com> wrote: > > On Thu, Sep 12, 2024 at 1:13 PM Caleb Rackliffe > <calebrackli...@gmail.com> wrote: >> >> I think I can count at least 4 people on this thread who literally have lost sleep over this. > > Probably good examples of not being the majority though, heh. > > If we are counting on users to read NEWS.txt, can we not count on them > to enable rejection if this is important to them? > > Kind Regards, > Brandon