+1
> On 5 May 2025, at 23:24, Blake Eggleston <bl...@ultrablake.com> wrote:
>
> As mutation tracking relates to existing backup systems that account for
> repaired vs unrepaired sstables, mutation tracking will continue to promote
> sstables to repaired once we know they contain data that has been fully
> reconciled. The main difference is that they won’t be promoted as part of an
> explicit range repair, but by compaction as they’re able to be promoted.
>
> (also +1 to finishing witnesses)
>
> On Mon, May 5, 2025, at 11:45 AM, Benedict Elliott Smith wrote:
>> Consistent backup/restore is a fundamentally hard and unsolved problem for
>> Cassandra today (without any of the mentioned features). In particular, we
>> break the real-time guarantee of the linearizability property (most notably
>> for LWTs) between partitions for any backup/restore process today.
>>
>> Fixing this should be relatively straight-forward for Accord, and something
>> we intend to address in follow-up work. Fixing it for eventually consistent
>> (or Paxos/LWT) operations is I think achievable, with or without mutation
>> tracking (probably easier with mutation tracking). I’m not sure of any plans
>> to try to tackle this though.
>>
>> Witness replicas should not particularly matter at all to any of the above.
>>
>>> On 5 May 2025, at 18:49, Jon Haddad <j...@rustyrazorblade.com> wrote:
>>>
>>> It took me a bit to wrap my head around how this works, but now that I
>>> think I understand the idea, it sounds like a solid improvement. Being
>>> able to achieve the same results as quorum but costing 1/3 less is a *big
>>> deal* and I know several teams that would be interested.
>>>
>>> One thing I'm curious about (and we can break it out into a separate
>>> discussion), is how all the functionality that requires coordination and
>>> global state (repaired vs non-repaired) will affect backups. Without a
>>> synchronization primitive to take a cluster-wide snapshot, how can we
>>> safely restore from eventually consistent backups without risking
>>> consistency issues due to out-of-sync repaired status?
>>>
>>> I don't think we need to block any of the proposed work on this - it's just
>>> something that's been nagging at me, and I don't know enough about the
>>> nuance of Accord, Mutation Tracking or Witness Replicas to say if it
>>> affects things or not. If it does, let's make sure we have that documented
>>> [1]
>>>
>>> Jon
>>>
>>> [1]
>>> https://cassandra.apache.org/doc/latest/cassandra/managing/operating/backups.html
>>>
>>>
>>>
>>> On Mon, May 5, 2025 at 10:21 AM Nate McCall <zznat...@gmail.com
>>> <mailto:zznat...@gmail.com>> wrote:
>>> This sounds like a modern feature that will benefit a lot of folks in
>>> cutting storage costs, particularly in large deployments.
>>>
>>> I'd like to see a note on the CEP about documentation overhead as this is
>>> an important feature to communicate correctly, but that's just a nit. +1 on
>>> moving forward with this overall.
>>>
>>> On Sun, May 4, 2025 at 1:58 PM Jordan West <jw...@apache.org
>>> <mailto:jw...@apache.org>> wrote:
>>> I’m generally supportive. The concept is one that I can see the benefits of
>>> and I also think the current implementation adds a lot of complexity to the
>>> codebase for being stuck in experimental mode. It will be great to have a
>>> more robust version built on a better approach.
>>>
>>> On Sun, May 4, 2025 at 00:27 Benedict <bened...@apache.org
>>> <mailto:bened...@apache.org>> wrote:
>>> +1
>>>
>>> This is an obviously good feature for operators that are storage-bound in
>>> multi-DC deployments but want to retain their latency characteristics
>>> during node maintenance. Log replicas are the right approach.
>>>
>>> > On 3 May 2025, at 23:42, sc...@paradoxica.net
>>> > <mailto:sc...@paradoxica.net> wrote:
>>> >
>>> > Hey everybody, bumping this CEP from Ariel in case you'd like some
>>> > weekend reading.
>>> >
>>> > We’d like to finish witnesses and bring them out of “experimental” status
>>> > now that Transactional Metadata and Mutation Tracking provide the
>>> > building blocks needed to complete them.
>>> >
>>> > Witnesses are part of a family of approaches in replicated storage
>>> > systems to maintain or boost availability and durability while reducing
>>> > storage costs. Log replicas are a close relative. Both are used by
>>> > leading cloud databases – for instance, Spanner implements witness
>>> > replicas [1] while DynamoDB implements log replicas [2].
>>> >
>>> > Witness replicas are a great fit for topologies that replicate at greater
>>> > than RF=3 –– most commonly multi-DC/multi-region deployments. Today in
>>> > Cassandra, all members of a voting quorum replicate all data forever.
>>> > Witness replicas let users break this coupling. They allow one to define
>>> > voting quorums that are larger than the number of copies of data that are
>>> > stored in perpetuity.
>>> >
>>> > Take a 3× DC cluster replicated at RF=3 in each DC as an example. In this
>>> > topology, Cassandra stores 9× copies of the database forever - huge
>>> > storage amplification. Witnesses allow users to maintain a voting quorum
>>> > of 9 members (3× per DC); but reduce the durable replicas to 2× per DC –
>>> > e.g., two durable replicas and one witness. This maintains the
>>> > availability properties of an RF=3×3 topology while reducing storage
>>> > costs by 33%, going from 9× copies to 6×.
>>> >
>>> > The role of a witness is to "witness" a write and persist it until it has
>>> > been reconciled among all durable replicas; and to respond to read
>>> > requests for witnessed writes awaiting reconciliation. Note that
>>> > witnesses don't introduce a dedicated role for a node – whether a node is
>>> > a durable replica or witness for a token just depends on its position in
>>> > the ring.
>>> >
>>> > This CEP builds on CEP-45: Mutation Tracking to establish the safety
>>> > property of the witness: guaranteeing that writes have been persisted to
>>> > all durable replicas before becoming purgeable. CEP-45's journal and
>>> > reconciliation design provide a great mechanism to ensure this while
>>> > avoiding the write amplification of incremental repair and anticompaction.
>>> >
>>> > Take a look at the CEP if you're interested - happy to answer questions
>>> > and discuss further:
>>> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-45%3A+Mutation+Tracking
>>> >
>>> > – Scott
>>> >
>>> > [1] https://cloud.google.com/spanner/docs/replication
>>> > [2] https://www.usenix.org/system/files/atc22-elhemali.pdf
>>> >
>>> >> On Apr 25, 2025, at 8:21 AM, Ariel Weisberg <ar...@weisberg.ws
>>> >> <mailto:ar...@weisberg.ws>> wrote:
>>> >>
>>> >> Hi all,
>>> >>
>>> >> The CEP is available here:
>>> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=353601959
>>> >>
>>> >> We would like to propose CEP-46: Finish Transient Replication/Witnesses
>>> >> for adoption by the community. CEP-46 would rename transient replication
>>> >> to witnesses and leverage mutation tracking to implement witnesses as
>>> >> CEP-45 Mutation Tracking based Log Replicas as a replacement for
>>> >> incremental repair based witnesses.
>>> >>
>>> >> For those not familiar with transient replication it would have the
>>> >> keyspace replication settings declare some replicas as transient and
>>> >> when incremental repair runs the transient replicas would delete data
>>> >> instead of moving it into the repaired set.
>>> >>
>>> >> With log replicas nodes only materialize mutations in their local LSM
>>> >> for ranges where they are full replicas and not witnesses. For witness
>>> >> ranges a node will write mutations to their local mutation tracking log
>>> >> and participate in background and read time reconciliation. This saves
>>> >> the compaction overhead of IR based witnesses which have to materialize
>>> >> and perform compaction on all mutations even those being applied to
>>> >> witness ranges.
>>> >>
>>> >> This would address one of the biggest issues with witnesses which is the
>>> >> lack of monotonic reads. Implementation complexity wise this would
>>> >> actually delete code compared to what would be required to complete IR
>>> >> based witnesses because most of the heavy lifting is already done by
>>> >> mutation tracking.
>>> >>
>>> >> Log replicas also makes it much more practical to realize the cost
>>> >> savings of witnesses because log replicas have easier to characterize
>>> >> resource consumption requirements (write rate * recovery/reconfiguration
>>> >> time) and target a 10x improvement in write throughput. This makes
>>> >> knowing how much capacity can be omitted safer and easier.
>>> >>
>>> >> Thanks,
>>> >> Ariel
>>> >
>
