I like this suggestion of enhanced manifest + copy snapshots to local storage as an initial step. This could provide a Cassandra baseline that sidecar or other tools could use to implement more intelligent (ie. cluster-wide) backups.
Once this works for local storage, users could potentially mount an s3/nfs/ebs file system to transparently transfer snapshots to remote storage without Cassandra needing to handle this part. On Thu, 8 Oct 2026 at 14:42 Štefan Miklošovič <[email protected]> wrote: > Yep. > > So I think that we should work "backwards". First we should probably > do manifest-v2.json which is way more comfortable to work with, > contains hashes of files, whole schema, each file ... any other > metadata. > > Once this is in place then we would do "copying to a local disk" with > this deduplication in place and eventually we would abstract away the > local disk to work with anything else. > > On Thu, Oct 8, 2026 at 8:35 PM Isaac Reath <[email protected]> wrote: > > > > > We have solved this already in our tooling, each snapshot has a > > manifest containing all SSTables it logically consists of. So if you > > go to remove a snapshot, you can remove a particular SSTable only in > > case it is not referenced in any other manifest. Otherwise your other > > snapshot would be lacking sstables. > > > > This is pretty much how Medusa handles this. Medusa uploads a manifest > with each backup to S3 with the files in the backup. SSTables are only > removed from S3 when `medusa purge` runs. `medusa purge` in turn deletes > the files which are not referenced in any active backup manifest (and have > had their configurable grace period expire to ensure that purge doesn't > race with an ongoing backup). I think any tooling we build in-project for > this (cassandra, sidecar or otherwise) should consider a similar approach. > > > > On Thu, Oct 8, 2026 at 2:16 PM Štefan Miklošovič <[email protected]> > wrote: > >> > >> Regarding snapshots, I want to say there is one conceptual "problem" > >> which seems to be more complicated than it seems. > >> > >> Currently, a snapshot is a bunch of hard links. Hardlinks don't > >> translate to S3. So we need to copy. Alright, so imagine we have this > >> snapshot: > >> > >> snap1 > >> > >> SSTable1 > >> SSTable2 > >> SSTable3 > >> > >> snap2 > >> > >> SSTable3 > >> SSTable4 > >> SSTable5 > >> > >> snap1 is uploaded, then snap2 is going to be, sstable 3 is in s3 > >> already so we skip that, then upload sstable 4 and 5. > >> > >> Now you go to remove snap2. But you can not remove sstable 3. Because > >> it is logically contained in snap1. Ha. > >> > >> In other words, you would need to have some global way to track this. > >> > >> We have solved this already in our tooling, each snapshot has a > >> manifest containing all SSTables it logically consists of. So if you > >> go to remove a snapshot, you can remove a particular SSTable only in > >> case it is not referenced in any other manifest. Otherwise your other > >> snapshot would be lacking sstables. > >> > >> So this "just upload snapshots to s3" is way more involved than it > >> seems, we would need to solve stuff like this. > >> > >> Or some other approach can be taken? I have not done extensive > >> research on how other tooling deals with this. Maybe there is some > >> elegant workaround. > >> > >> On Thu, Oct 8, 2026 at 7:59 PM Štefan Miklošovič < > [email protected]> wrote: > >> > > >> > These are all valid points you raised. I think that CEP-36 can be > >> > "limited" to just providing a pluggable way for ChannelProxy. Serving > >> > from S3 directly or not, there will still need to be a way to get a > >> > cloud-specific implementation of ChannelProxy for reads / write of any > >> > kind. Later on we can decide the overall architecture of that but I > >> > would reserve it for another CEP, that work will most probably be way > >> > more involved than "just" a ChannelProxy creation. > >> > > >> > If ChannelProxy is pluggable already, we can start to use this for > >> > snapshot uploading _if somebody ever wants to go that route_, I would > >> > just need to tweak the snapshotting stuff to be ChannelProxy-aware > >> > instead of on "File"-s. The default impl would be on FileChannel as we > >> > know it. > >> > > >> > On Thu, Oct 8, 2026 at 5:02 PM Jon Haddad <[email protected]> > wrote: > >> > > > >> > > I didn't realize the CEP hadn't been updated. I think I was mixing > up the convo on the ML and what was actually in the CEP. I had remembered > it behaving much closer to clickhouse, where it keeps the data on disk and > pushes the compacted files to S3. I don't think serving directly off S3 is > going to be much use to anyone. The latency will be awful, and they issue > a backoff warning fairly quickly. In my mind it only makes sense to treat > S3 as the authority and cache SStables to local disk. I think the Astra > folks may have done something along these lines, I'd be willing to bet > there's a ton of knowledge there as they operationalized it successfully. > >> > > > >> > > Anyways - the reason I asked my question, was to frame it in the > context of a longer term project strategy. Asking more directly - If we're > going to be pushing and pulling from S3 during compaction, and it's a major > feature, would that go in Sidecar? It certainly could, but I'm concerned > about having a critical component of C* living it's own independent > lifecycle. > >> > > > >> > > Example of why I get concerned: > https://the-asf.slack.com/archives/C07V0FK0M2P/p1788964591823929 > >> > > > >> > > Imo, having 2 projects that are tightly coupled, but developed > independently, is going to be fairly high risk. I really don't like having > responsibility spread this way, and I probably wouldn't want to run it in > prod. > >> > > > >> > > Jon > >> > > > >> > > > >> > > > >> > > On Wed, Oct 7, 2026 at 7:15 AM Štefan Miklošovič < > [email protected]> wrote: > >> > >> > >> > >> I have a draft of CEP-36 here (1). S3 deps + proxy impl is meant > to be > >> > >> implemented in a different project (2) which once built is added to > >> > >> Cassandra via custom ClassLoader so it is all encapsulated and we > can > >> > >> not have any dependency clashes. > >> > >> > >> > >> There is the impl of s3 proxy with e2e tests. It shows how > Cassandra > >> > >> starts to serve reads from s3. When it comes to writes, these > SSTables > >> > >> will be created locally as they are not and then "offloaded" to S3 > for > >> > >> reads. Once SSTable is created it is immutable so we can upload it > and > >> > >> then switch SSTableReader to read via S3 proxy. > >> > >> > >> > >> At least that is the idea. Heavy demo. Dragons everywhere. > AI-driven a > >> > >> lot. It more _does not work_ than does :D But it genuinely starts > to > >> > >> read from S3 via proxy which is quite ... promising, but I think > that > >> > >> if we want to seriously consider CEP-36 and implement it fully > that is > >> > >> a serious piece of work for two quarters :D > >> > >> > >> > >> Give me some time here please, like a week or two, I would like to > >> > >> explore this in more detail. Maybe it will be better to modify > >> > >> SnapshotManager to work with ChannelProxy instead of File and when > >> > >> taking snapshots we would use S3 ChannelProxy plus we would tweak > the > >> > >> hardlinking bit. > >> > >> > >> > >> You can of course do this all in Sidecar, I just want to see where > >> > >> this gets us. You would have two ways to choose from (and CEP-36 > >> > >> building blocks would be in place to extend on further). > >> > >> > >> > >> (1) https://github.com/apache/cassandra/pull/5238/changes > >> > >> (2) https://github.com/smiklosovic/cassandra-s3-channel-proxy > >> > >> > >> > >> On Wed, Oct 7, 2026 at 3:30 PM Isaac Reath <[email protected]> > wrote: > >> > >> > > >> > >> > In fairness, CEP-36 does explicitly call out that anything that > uses a hard link will no longer work once data is written in an alternative > location: > >> > >> > > >> > >> > > Once triggered any data written to alternative locations will > not be included in the snapshotting, incremental backups or other processes > that use hard file links. > >> > >> > > >> > >> > In a world where we have CEP-36 and an S3-backed implementation > of ChannelProxy, I would imagine that a call to the > ColumnFamilyStore.takeSnapshot function would either no-op (sticking to > what is defined in the CEP), wrap a call to take an on-demand backup via > AWS Backup, or do something similar to running aws s3 sync. In this world, > if you wanted to take a backup across a whole cluster, you could still use > Sidecar to manage the coordination. Or you could just use continuous > backups on your S3 buckets and call it a day. Depends on the semantics and > use-cases of what we're looking for out of a backup solution. > >> > >> > > >> > >> > CEP-36 offers details around how to get the ChannelProxy > interface in place, but leaves the details of how to implement an object > store-backed version of the storage engine as an exercise for the reader. I > think if we were to get CEP-36 voted on and implemented, we'd need a > follow-up CEP that dives into the S3 implementation, or otherwise we'd need > to expand the current scope of CEP-36 to cover it. > >> > >> > > >> > >> > On Wed, Oct 7, 2026 at 6:31 AM Štefan Miklošovič < > [email protected]> wrote: > >> > >> >> > >> > >> >> BTW I am still not in favor of putting this to Cassandra, just > have > >> > >> >> snapshots in Sidecar with its own endpoints, it is way easier > that > >> > >> >> way. I am pretty tired at this point to go through these > discussions > >> > >> >> all over again yielding no actual resolution of what we are > going to > >> > >> >> do. "cloud-aware Cassandra" was so long on a roadmap I am > losing hope > >> > >> >> we will ever see that. The amount of work needed is enormous, > we would > >> > >> >> need to go case by case and decide what is still to be written > on a > >> > >> >> local disk and what goes to the cloud. Some functionality is > not even > >> > >> >> there: > >> > >> >> > >> > >> >> 1) How do you atomically rename a file? S3 does not have the > concept > >> > >> >> of rename. There is some "S3 Express One Zone" which seems to > support > >> > >> >> RenameObject but that seems quite niche > >> > >> >> 2) Hard links ... none, we would need to copy > >> > >> >> 3) mmap, how is this done in S3? You can not memory map an > object > >> > >> >> store, no? We would need to do disk_access_mode: standard > >> > >> >> 4) Latency ... > >> > >> >> > >> > >> >> Commit logs, hints, caches ... Accord stuff. I don't know what > else, > >> > >> >> all the local disk interaction which is illogical to move to > cloud > >> > >> >> would need to stay local. (anti)-compaction, repair ... all of > that > >> > >> >> stuff would be cloud-native too? The performance of that would > be ... > >> > >> >> tragic? > >> > >> >> > >> > >> >> I know it is something to aspire for and I do not want to be too > >> > >> >> negative about this but I think it makes sense to stop for a > while and > >> > >> >> actually see what this would mean and how much work it is. > >> > >> >> > >> > >> >> For that reason I would be doing backup / restores in Sidecar > and be > >> > >> >> done with that. It is a very well understood domain, s3 deps > already > >> > >> >> there, endpoints endpointing, a snapshot is a JMX call ... why > to > >> > >> >> complicate this more really. > >> > >> >> > >> > >> >> On Tue, Oct 6, 2026 at 9:11 PM Štefan Miklošovič < > [email protected]> wrote: > >> > >> >> > > >> > >> >> > Well ... that's quite a topic. > >> > >> >> > > >> > >> >> > The OP was asking specifically about backup / restore so I was > >> > >> >> > replying in that spirit. > >> > >> >> > > >> > >> >> > One note for backup / restores, we talk about "snapshots", > right? So > >> > >> >> > if you back them up, then one would expect that Sidecar would > also > >> > >> >> > offer a way to list them, then to clear them ... It would not > be "just > >> > >> >> > about backup and restore". There would be a family of > endpoints around > >> > >> >> > snapshot management, very likely. That means we kind of > duplicate what > >> > >> >> > is in Cassandra already, if we do not put this into Cassandra > >> > >> >> > directly, then Cassandra would not "see" the snapshots, > except those > >> > >> >> > taken locally. > >> > >> >> > > >> > >> >> > If we want to entertain CEP-36 specifically, that would > change the > >> > >> >> > picture quite significantly. In either case, snapshots would > need to > >> > >> >> > be copied over (SSTables, that is), because we can not do a > hard-link > >> > >> >> > from local storage to a S3 bucket. So we would need to copy > that, > >> > >> >> > right? If that is the case then we would need to probably > throttle how > >> > >> >> > fast it is uploaded to not interfere with other operations and > >> > >> >> > similar. > >> > >> >> > > >> > >> >> > I was completely over-hauling the snapshot management in > >> > >> >> > CASSANDRA-18111, together with Paulo Motta who did a great job > >> > >> >> > reviewing it, everything is centralized in > SnapshotManager(MBean). The > >> > >> >> > only tasks touching the disk are TakeSnapshotTask, > ClearSnapshotTask > >> > >> >> > and SnapshotLoader which loads them. Their representation / > metadata > >> > >> >> > are in memory. So when you list it is not going to S3. That > isolates > >> > >> >> > what would need to be accommodated to talk to a cloud. So if > you > >> > >> >> > wanted to start to use S3 as a storage, taking CEP-36 into > account, we > >> > >> >> > would need to: > >> > >> >> > > >> > >> >> > 1) create "cloude-aware" implementation of > >> > >> >> > org.apache.cassandra.io.util.File / java.nio.file.Path > >> > >> >> > 2) create "S3 TakeSnapshotTaks" which would need to start to > copy > >> > >> >> > instead of hardlinking etc. > >> > >> >> > 3) create another version of SnapshotLoader or modify it > heavily as > >> > >> >> > currently it is pretty-much wired to talk to local file system > >> > >> >> > 4) Find some way how to "switch" between local / cloud via > >> > >> >> > configuration in a complex manner > >> > >> >> > > >> > >> >> > I still think it holds that we would need to introduce some > >> > >> >> > "class-path modularisation" like you would say, via > configuration, > >> > >> >> > what cloud storage provider you want to use and the related > libraries > >> > >> >> > would be auto-magically put into the class path. If we had > this in > >> > >> >> > place then CEP-36 might be potentially closer to the > implementation as > >> > >> >> > these building blocks would be there already. > >> > >> >> > > >> > >> >> > On Tue, Oct 6, 2026 at 6:32 PM Jon Haddad < > [email protected]> wrote: > >> > >> >> > > > >> > >> >> > > For the folks who want to put this in sidecar, what are > your thoughts on how that would work with a potential future improvement > using s3 as part of its normal operations? I’m thinking about CEP-36 > specifically. > >> > >> >> > > > >> > >> >> > > Thanks! > >> > >> >> > > Jon > >> > >> >> > > > >> > >> >> > > > >> > >> >> > > On Tue, Oct 6, 2026 at 7:37 AM Isaac Reath < > [email protected]> wrote: > >> > >> >> > >> > >> > >> >> > >> > If I am not mistaken, Sidecar already exposes a set of > >> > >> >> > >> > endpoints related to the restoration of data from S3 > when running some > >> > >> >> > >> > Spark jobs, right? > >> > >> >> > >> > >> > >> >> > >> You are correct Stefan. > >> > >> >> > >> > >> > >> >> > >> I'll echo Stefan's concerns around the weight of adding a > cloud provider dependency into Cassandra. Especially now, with the > increasing volume of CVE discovery across all sorts of open-source > projects, keeping the dependency scope of the core Cassandra process small > is a distinct advantage. As an operator, it's much easier for me to push > out patches across thousands of Sidecars than it is to push them across > thousands of Cassandra nodes. > >> > >> >> > >> > >> > >> >> > >> I also think that from a resource isolation perspective, > running these types of maintenance operations out-of-process can create > much more predictable performance, especially when paired with OS-level > resource governance tools like cgroups. What a dream it might be if we had > out-of-process compaction or out-of-process Merkle tree calculation for > repair. > >> > >> >> > >> > >> > >> >> > >> I understand your point that adding another Sidecar just > for backups can increase operational complexity, but I'd argue that > cassandra-sidecar is shaping up to offer far more than just backups. Today > we have Sidecar acting as the entrypoint for bulk analytics, CEP-44 > bringing CDC publishing to Kafka, CEP-40 adding zero-copy online data > migration between nodes and with CEP-53 and CEP-62 we are getting rolling > restarts, configuration management and a framework for executing other > distributed jobs across a Cassandra cluster. Framed this way it's becoming > a one-stop-shop for these types of operations. I also think that building > on top of the operational job framework being added with CEP-53 could go a > long way towards making the implementation easier. > >> > >> >> > >> > >> > >> >> > >> > >> > >> >> > >> On Tue, Oct 6, 2026 at 8:56 AM Victor Araujo < > [email protected]> wrote: > >> > >> >> > >>> > >> > >> >> > >>> Hi Bernardo and Stefan, > >> > >> >> > >>> > >> > >> >> > >>> Thanks for the feedback. I agree that there are several > benefits to having backup and restore in Sidecar, especially regarding > isolation, dependencies, and being able to evolve the feature independently > from Cassandra itself. I would be happy to explore and contribute to that > direction, potentially in my free time, and I may also be able to bring > this discussion to my company later. > >> > >> >> > >>> > >> > >> >> > >>> My main concern is more about the entry point for users. > In my current environment, we are actually trying to reduce our use of > Sidecars and other additional processes to reduce the overall footprint. I > also think there may be users who would prefer having a basic built-in > solution in Cassandra rather than having to deploy another component just > to get backup and restore. > >> > >> >> > >>> > >> > >> >> > >>> Maybe there is a middle ground here. We could have an > opt-in, generic backup/restore capability in Cassandra core, similar in > spirit to the unified repair solution, without bringing any third-party > cloud dependencies into Cassandra. For example, the core could provide the > backup/restore abstractions and lifecycle, with a pluggable storage > implementation provided by the user or another component. This could serve > as a simple entry-level solution, while a more complete implementation with > cloud-specific integrations could live in Sidecar. > >> > >> >> > >>> > >> > >> >> > >>> I don't know yet whether this split makes sense from an > architectural perspective, but I think it could be worth exploring. I would > be interested in hearing what others think about this approach. > >> > >> >> > >>> > >> > >> >> > >>> Thanks, > >> > >> >> > >>> Victor > >> > >> >> > >>> > >> > >> >> > >>> > >> > >> >> > >>> Em ter., 6 de out. de 2026 às 07:49, Štefan Miklošovič < > [email protected]> escreveu: > >> > >> >> > >>>> > >> > >> >> > >>>> I also think that we should investigate how to integrate > backup / > >> > >> >> > >>>> restores in Sidecar rather than doing it in Cassandra. > >> > >> >> > >>>> Dependency-wise, I think Sidecar already depends on s3 > so that is for > >> > >> >> > >>>> free already. If I am not mistaken, Sidecar already > exposes a set of > >> > >> >> > >>>> endpoints related to the restoration of data from S3 > when running some > >> > >> >> > >>>> Spark jobs, right? So this would be just a natural > extension of that - > >> > >> >> > >>>> a set of endpoints backuping to and restoring from an S3 > bucket. > >> > >> >> > >>>> > >> > >> >> > >>>> However, I think that we should make it generic enough > in such a way > >> > >> >> > >>>> that if somebody wants to do this against Azure or GCP > they could. We > >> > >> >> > >>>> should not "wire" S3 only. I am not sure how to do this > though. These > >> > >> >> > >>>> deps are quite "heavy", depending on their own > dependency chain ... I > >> > >> >> > >>>> do not have good experience with mixing various cloud > libraries. > >> > >> >> > >>>> > >> > >> >> > >>>> What we do in our backup / restore solution is that we > modularize it > >> > >> >> > >>>> in such a way that a user can choose what cloud they go > to deal with > >> > >> >> > >>>> and they have only that cloud-specific libraries on the > class path. > >> > >> >> > >>>> > >> > >> >> > >>>> On Tue, Oct 6, 2026 at 12:14 PM Bernardo Botella > >> > >> >> > >>>> <[email protected]> wrote: > >> > >> >> > >>>> > > >> > >> >> > >>>> > Hi Victor! > >> > >> >> > >>>> > > >> > >> >> > >>>> > Thanks for bringing up this discussion 😊 > >> > >> >> > >>>> > > >> > >> >> > >>>> > I completely agree that a first citizen backup and > restore mechanism in Cassandra would be great. > >> > >> >> > >>>> > > >> > >> >> > >>>> > I’d like to explore a little bit more the solution > involving Sidecar. While it is true that it would “force” you to have yet > another component running, it definitely comes with a lot of benefits > besides the current other features it provides (control plane, analytics > library, CDC over Kafka, to name some). > >> > >> >> > >>>> > > >> > >> >> > >>>> > It allows for a more modular approach to a Cassandra > deployment. In my humble opinion, having smaller, more specialized > processes is the direction we should be following, as opposed to keep > making the core of the project bigger and more complex. > >> > >> >> > >>>> > It allows for a faster iteration in terms of software. > When it comes to version upgrades, it is definitely way easier to update > just the sidecar than the entire Cassandra core. It is also less risky. > >> > >> >> > >>>> > Blast radius is also smaller. It is better to have the > Sidecar affected if a backup process stalls/breaks/misbehaves than the > whole Cassandra core. > >> > >> >> > >>>> > > >> > >> >> > >>>> > > >> > >> >> > >>>> > That’s my two cents! > >> > >> >> > >>>> > Bernardo > >> > >> >> > >>>> > > >> > >> >> > >>>> > From: Victor Araujo <[email protected]> > >> > >> >> > >>>> > Date: Monday, 5 October 2026 at 13:34 > >> > >> >> > >>>> > To: [email protected] <[email protected] > > > >> > >> >> > >>>> > Subject: [DISCUSS] Backup and restore in Cassandra > >> > >> >> > >>>> > > >> > >> >> > >>>> > Hi everyone, > >> > >> >> > >>>> > > >> > >> >> > >>>> > I would like to discuss the possibility of having a > built-in backup and restore mechanism in Cassandra, including incremental > backups and support for external object storage such as S3. The main goal > would be to provide a supported solution while keeping the implementation > provider-agnostic and avoiding unnecessary operational overhead. > >> > >> >> > >>>> > > >> > >> >> > >>>> > I initially discussed this on Slack and was pointed to > CEP-36 and the previous discussions around external storage and snapshots. > We also discussed whether S3 would be too provider-specific, and whether > Cassandra Sidecar could be part of the solution. My concern with Sidecar is > that introducing another component would increase the operational > footprint, which goes against one of the goals of this work. I also have a > small PoC on top of trunk exploring some of these ideas. > >> > >> >> > >>>> > > >> > >> >> > >>>> > I'd be interested in hearing other opinions and any > previous work I should look at. If there is interest in pursuing this, > would a draft CEP be a useful next step so we can start working on the > design in parallel? > >> > >> >> > >>>> > > >> > >> >> > >>>> > Link of the first slack discussion about this topic: > https://the-asf.slack.com/archives/CK23JSY2K/p1789021792614639 > >> > >> >> > >>>> > > >> > >> >> > >>>> > Thanks, > >> > >> >> > >>>> > Victor Cavichioli >
