> We have solved this already in our tooling, each snapshot has a manifest containing all SSTables it logically consists of. So if you go to remove a snapshot, you can remove a particular SSTable only in case it is not referenced in any other manifest. Otherwise your other snapshot would be lacking sstables.
This is pretty much how Medusa handles this. Medusa uploads a manifest with each backup to S3 with the files in the backup. SSTables are only removed from S3 when `medusa purge` runs. `medusa purge` in turn deletes the files which are not referenced in any active backup manifest (and have had their configurable grace period expire to ensure that purge doesn't race with an ongoing backup). I think any tooling we build in-project for this (cassandra, sidecar or otherwise) should consider a similar approach. On Thu, Oct 8, 2026 at 2:16 PM Štefan Miklošovič <[email protected]> wrote: > Regarding snapshots, I want to say there is one conceptual "problem" > which seems to be more complicated than it seems. > > Currently, a snapshot is a bunch of hard links. Hardlinks don't > translate to S3. So we need to copy. Alright, so imagine we have this > snapshot: > > snap1 > > SSTable1 > SSTable2 > SSTable3 > > snap2 > > SSTable3 > SSTable4 > SSTable5 > > snap1 is uploaded, then snap2 is going to be, sstable 3 is in s3 > already so we skip that, then upload sstable 4 and 5. > > Now you go to remove snap2. But you can not remove sstable 3. Because > it is logically contained in snap1. Ha. > > In other words, you would need to have some global way to track this. > > We have solved this already in our tooling, each snapshot has a > manifest containing all SSTables it logically consists of. So if you > go to remove a snapshot, you can remove a particular SSTable only in > case it is not referenced in any other manifest. Otherwise your other > snapshot would be lacking sstables. > > So this "just upload snapshots to s3" is way more involved than it > seems, we would need to solve stuff like this. > > Or some other approach can be taken? I have not done extensive > research on how other tooling deals with this. Maybe there is some > elegant workaround. > > On Thu, Oct 8, 2026 at 7:59 PM Štefan Miklošovič <[email protected]> > wrote: > > > > These are all valid points you raised. I think that CEP-36 can be > > "limited" to just providing a pluggable way for ChannelProxy. Serving > > from S3 directly or not, there will still need to be a way to get a > > cloud-specific implementation of ChannelProxy for reads / write of any > > kind. Later on we can decide the overall architecture of that but I > > would reserve it for another CEP, that work will most probably be way > > more involved than "just" a ChannelProxy creation. > > > > If ChannelProxy is pluggable already, we can start to use this for > > snapshot uploading _if somebody ever wants to go that route_, I would > > just need to tweak the snapshotting stuff to be ChannelProxy-aware > > instead of on "File"-s. The default impl would be on FileChannel as we > > know it. > > > > On Thu, Oct 8, 2026 at 5:02 PM Jon Haddad <[email protected]> > wrote: > > > > > > I didn't realize the CEP hadn't been updated. I think I was mixing up > the convo on the ML and what was actually in the CEP. I had remembered it > behaving much closer to clickhouse, where it keeps the data on disk and > pushes the compacted files to S3. I don't think serving directly off S3 is > going to be much use to anyone. The latency will be awful, and they issue > a backoff warning fairly quickly. In my mind it only makes sense to treat > S3 as the authority and cache SStables to local disk. I think the Astra > folks may have done something along these lines, I'd be willing to bet > there's a ton of knowledge there as they operationalized it successfully. > > > > > > Anyways - the reason I asked my question, was to frame it in the > context of a longer term project strategy. Asking more directly - If we're > going to be pushing and pulling from S3 during compaction, and it's a major > feature, would that go in Sidecar? It certainly could, but I'm concerned > about having a critical component of C* living it's own independent > lifecycle. > > > > > > Example of why I get concerned: > https://the-asf.slack.com/archives/C07V0FK0M2P/p1788964591823929 > > > > > > Imo, having 2 projects that are tightly coupled, but developed > independently, is going to be fairly high risk. I really don't like having > responsibility spread this way, and I probably wouldn't want to run it in > prod. > > > > > > Jon > > > > > > > > > > > > On Wed, Oct 7, 2026 at 7:15 AM Štefan Miklošovič < > [email protected]> wrote: > > >> > > >> I have a draft of CEP-36 here (1). S3 deps + proxy impl is meant to be > > >> implemented in a different project (2) which once built is added to > > >> Cassandra via custom ClassLoader so it is all encapsulated and we can > > >> not have any dependency clashes. > > >> > > >> There is the impl of s3 proxy with e2e tests. It shows how Cassandra > > >> starts to serve reads from s3. When it comes to writes, these SSTables > > >> will be created locally as they are not and then "offloaded" to S3 for > > >> reads. Once SSTable is created it is immutable so we can upload it and > > >> then switch SSTableReader to read via S3 proxy. > > >> > > >> At least that is the idea. Heavy demo. Dragons everywhere. AI-driven a > > >> lot. It more _does not work_ than does :D But it genuinely starts to > > >> read from S3 via proxy which is quite ... promising, but I think that > > >> if we want to seriously consider CEP-36 and implement it fully that is > > >> a serious piece of work for two quarters :D > > >> > > >> Give me some time here please, like a week or two, I would like to > > >> explore this in more detail. Maybe it will be better to modify > > >> SnapshotManager to work with ChannelProxy instead of File and when > > >> taking snapshots we would use S3 ChannelProxy plus we would tweak the > > >> hardlinking bit. > > >> > > >> You can of course do this all in Sidecar, I just want to see where > > >> this gets us. You would have two ways to choose from (and CEP-36 > > >> building blocks would be in place to extend on further). > > >> > > >> (1) https://github.com/apache/cassandra/pull/5238/changes > > >> (2) https://github.com/smiklosovic/cassandra-s3-channel-proxy > > >> > > >> On Wed, Oct 7, 2026 at 3:30 PM Isaac Reath <[email protected]> > wrote: > > >> > > > >> > In fairness, CEP-36 does explicitly call out that anything that > uses a hard link will no longer work once data is written in an alternative > location: > > >> > > > >> > > Once triggered any data written to alternative locations will not > be included in the snapshotting, incremental backups or other processes > that use hard file links. > > >> > > > >> > In a world where we have CEP-36 and an S3-backed implementation of > ChannelProxy, I would imagine that a call to the > ColumnFamilyStore.takeSnapshot function would either no-op (sticking to > what is defined in the CEP), wrap a call to take an on-demand backup via > AWS Backup, or do something similar to running aws s3 sync. In this world, > if you wanted to take a backup across a whole cluster, you could still use > Sidecar to manage the coordination. Or you could just use continuous > backups on your S3 buckets and call it a day. Depends on the semantics and > use-cases of what we're looking for out of a backup solution. > > >> > > > >> > CEP-36 offers details around how to get the ChannelProxy interface > in place, but leaves the details of how to implement an object store-backed > version of the storage engine as an exercise for the reader. I think if we > were to get CEP-36 voted on and implemented, we'd need a follow-up CEP that > dives into the S3 implementation, or otherwise we'd need to expand the > current scope of CEP-36 to cover it. > > >> > > > >> > On Wed, Oct 7, 2026 at 6:31 AM Štefan Miklošovič < > [email protected]> wrote: > > >> >> > > >> >> BTW I am still not in favor of putting this to Cassandra, just have > > >> >> snapshots in Sidecar with its own endpoints, it is way easier that > > >> >> way. I am pretty tired at this point to go through these > discussions > > >> >> all over again yielding no actual resolution of what we are going > to > > >> >> do. "cloud-aware Cassandra" was so long on a roadmap I am losing > hope > > >> >> we will ever see that. The amount of work needed is enormous, we > would > > >> >> need to go case by case and decide what is still to be written on a > > >> >> local disk and what goes to the cloud. Some functionality is not > even > > >> >> there: > > >> >> > > >> >> 1) How do you atomically rename a file? S3 does not have the > concept > > >> >> of rename. There is some "S3 Express One Zone" which seems to > support > > >> >> RenameObject but that seems quite niche > > >> >> 2) Hard links ... none, we would need to copy > > >> >> 3) mmap, how is this done in S3? You can not memory map an object > > >> >> store, no? We would need to do disk_access_mode: standard > > >> >> 4) Latency ... > > >> >> > > >> >> Commit logs, hints, caches ... Accord stuff. I don't know what > else, > > >> >> all the local disk interaction which is illogical to move to cloud > > >> >> would need to stay local. (anti)-compaction, repair ... all of that > > >> >> stuff would be cloud-native too? The performance of that would be > ... > > >> >> tragic? > > >> >> > > >> >> I know it is something to aspire for and I do not want to be too > > >> >> negative about this but I think it makes sense to stop for a while > and > > >> >> actually see what this would mean and how much work it is. > > >> >> > > >> >> For that reason I would be doing backup / restores in Sidecar and > be > > >> >> done with that. It is a very well understood domain, s3 deps > already > > >> >> there, endpoints endpointing, a snapshot is a JMX call ... why to > > >> >> complicate this more really. > > >> >> > > >> >> On Tue, Oct 6, 2026 at 9:11 PM Štefan Miklošovič < > [email protected]> wrote: > > >> >> > > > >> >> > Well ... that's quite a topic. > > >> >> > > > >> >> > The OP was asking specifically about backup / restore so I was > > >> >> > replying in that spirit. > > >> >> > > > >> >> > One note for backup / restores, we talk about "snapshots", > right? So > > >> >> > if you back them up, then one would expect that Sidecar would > also > > >> >> > offer a way to list them, then to clear them ... It would not be > "just > > >> >> > about backup and restore". There would be a family of endpoints > around > > >> >> > snapshot management, very likely. That means we kind of > duplicate what > > >> >> > is in Cassandra already, if we do not put this into Cassandra > > >> >> > directly, then Cassandra would not "see" the snapshots, except > those > > >> >> > taken locally. > > >> >> > > > >> >> > If we want to entertain CEP-36 specifically, that would change > the > > >> >> > picture quite significantly. In either case, snapshots would > need to > > >> >> > be copied over (SSTables, that is), because we can not do a > hard-link > > >> >> > from local storage to a S3 bucket. So we would need to copy that, > > >> >> > right? If that is the case then we would need to probably > throttle how > > >> >> > fast it is uploaded to not interfere with other operations and > > >> >> > similar. > > >> >> > > > >> >> > I was completely over-hauling the snapshot management in > > >> >> > CASSANDRA-18111, together with Paulo Motta who did a great job > > >> >> > reviewing it, everything is centralized in > SnapshotManager(MBean). The > > >> >> > only tasks touching the disk are TakeSnapshotTask, > ClearSnapshotTask > > >> >> > and SnapshotLoader which loads them. Their representation / > metadata > > >> >> > are in memory. So when you list it is not going to S3. That > isolates > > >> >> > what would need to be accommodated to talk to a cloud. So if you > > >> >> > wanted to start to use S3 as a storage, taking CEP-36 into > account, we > > >> >> > would need to: > > >> >> > > > >> >> > 1) create "cloude-aware" implementation of > > >> >> > org.apache.cassandra.io.util.File / java.nio.file.Path > > >> >> > 2) create "S3 TakeSnapshotTaks" which would need to start to copy > > >> >> > instead of hardlinking etc. > > >> >> > 3) create another version of SnapshotLoader or modify it heavily > as > > >> >> > currently it is pretty-much wired to talk to local file system > > >> >> > 4) Find some way how to "switch" between local / cloud via > > >> >> > configuration in a complex manner > > >> >> > > > >> >> > I still think it holds that we would need to introduce some > > >> >> > "class-path modularisation" like you would say, via > configuration, > > >> >> > what cloud storage provider you want to use and the related > libraries > > >> >> > would be auto-magically put into the class path. If we had this > in > > >> >> > place then CEP-36 might be potentially closer to the > implementation as > > >> >> > these building blocks would be there already. > > >> >> > > > >> >> > On Tue, Oct 6, 2026 at 6:32 PM Jon Haddad < > [email protected]> wrote: > > >> >> > > > > >> >> > > For the folks who want to put this in sidecar, what are your > thoughts on how that would work with a potential future improvement using > s3 as part of its normal operations? I’m thinking about CEP-36 > specifically. > > >> >> > > > > >> >> > > Thanks! > > >> >> > > Jon > > >> >> > > > > >> >> > > > > >> >> > > On Tue, Oct 6, 2026 at 7:37 AM Isaac Reath < > [email protected]> wrote: > > >> >> > >> > > >> >> > >> > If I am not mistaken, Sidecar already exposes a set of > > >> >> > >> > endpoints related to the restoration of data from S3 when > running some > > >> >> > >> > Spark jobs, right? > > >> >> > >> > > >> >> > >> You are correct Stefan. > > >> >> > >> > > >> >> > >> I'll echo Stefan's concerns around the weight of adding a > cloud provider dependency into Cassandra. Especially now, with the > increasing volume of CVE discovery across all sorts of open-source > projects, keeping the dependency scope of the core Cassandra process small > is a distinct advantage. As an operator, it's much easier for me to push > out patches across thousands of Sidecars than it is to push them across > thousands of Cassandra nodes. > > >> >> > >> > > >> >> > >> I also think that from a resource isolation perspective, > running these types of maintenance operations out-of-process can create > much more predictable performance, especially when paired with OS-level > resource governance tools like cgroups. What a dream it might be if we had > out-of-process compaction or out-of-process Merkle tree calculation for > repair. > > >> >> > >> > > >> >> > >> I understand your point that adding another Sidecar just for > backups can increase operational complexity, but I'd argue that > cassandra-sidecar is shaping up to offer far more than just backups. Today > we have Sidecar acting as the entrypoint for bulk analytics, CEP-44 > bringing CDC publishing to Kafka, CEP-40 adding zero-copy online data > migration between nodes and with CEP-53 and CEP-62 we are getting rolling > restarts, configuration management and a framework for executing other > distributed jobs across a Cassandra cluster. Framed this way it's becoming > a one-stop-shop for these types of operations. I also think that building > on top of the operational job framework being added with CEP-53 could go a > long way towards making the implementation easier. > > >> >> > >> > > >> >> > >> > > >> >> > >> On Tue, Oct 6, 2026 at 8:56 AM Victor Araujo < > [email protected]> wrote: > > >> >> > >>> > > >> >> > >>> Hi Bernardo and Stefan, > > >> >> > >>> > > >> >> > >>> Thanks for the feedback. I agree that there are several > benefits to having backup and restore in Sidecar, especially regarding > isolation, dependencies, and being able to evolve the feature independently > from Cassandra itself. I would be happy to explore and contribute to that > direction, potentially in my free time, and I may also be able to bring > this discussion to my company later. > > >> >> > >>> > > >> >> > >>> My main concern is more about the entry point for users. In > my current environment, we are actually trying to reduce our use of > Sidecars and other additional processes to reduce the overall footprint. I > also think there may be users who would prefer having a basic built-in > solution in Cassandra rather than having to deploy another component just > to get backup and restore. > > >> >> > >>> > > >> >> > >>> Maybe there is a middle ground here. We could have an > opt-in, generic backup/restore capability in Cassandra core, similar in > spirit to the unified repair solution, without bringing any third-party > cloud dependencies into Cassandra. For example, the core could provide the > backup/restore abstractions and lifecycle, with a pluggable storage > implementation provided by the user or another component. This could serve > as a simple entry-level solution, while a more complete implementation with > cloud-specific integrations could live in Sidecar. > > >> >> > >>> > > >> >> > >>> I don't know yet whether this split makes sense from an > architectural perspective, but I think it could be worth exploring. I would > be interested in hearing what others think about this approach. > > >> >> > >>> > > >> >> > >>> Thanks, > > >> >> > >>> Victor > > >> >> > >>> > > >> >> > >>> > > >> >> > >>> Em ter., 6 de out. de 2026 às 07:49, Štefan Miklošovič < > [email protected]> escreveu: > > >> >> > >>>> > > >> >> > >>>> I also think that we should investigate how to integrate > backup / > > >> >> > >>>> restores in Sidecar rather than doing it in Cassandra. > > >> >> > >>>> Dependency-wise, I think Sidecar already depends on s3 so > that is for > > >> >> > >>>> free already. If I am not mistaken, Sidecar already exposes > a set of > > >> >> > >>>> endpoints related to the restoration of data from S3 when > running some > > >> >> > >>>> Spark jobs, right? So this would be just a natural > extension of that - > > >> >> > >>>> a set of endpoints backuping to and restoring from an S3 > bucket. > > >> >> > >>>> > > >> >> > >>>> However, I think that we should make it generic enough in > such a way > > >> >> > >>>> that if somebody wants to do this against Azure or GCP they > could. We > > >> >> > >>>> should not "wire" S3 only. I am not sure how to do this > though. These > > >> >> > >>>> deps are quite "heavy", depending on their own dependency > chain ... I > > >> >> > >>>> do not have good experience with mixing various cloud > libraries. > > >> >> > >>>> > > >> >> > >>>> What we do in our backup / restore solution is that we > modularize it > > >> >> > >>>> in such a way that a user can choose what cloud they go to > deal with > > >> >> > >>>> and they have only that cloud-specific libraries on the > class path. > > >> >> > >>>> > > >> >> > >>>> On Tue, Oct 6, 2026 at 12:14 PM Bernardo Botella > > >> >> > >>>> <[email protected]> wrote: > > >> >> > >>>> > > > >> >> > >>>> > Hi Victor! > > >> >> > >>>> > > > >> >> > >>>> > Thanks for bringing up this discussion 😊 > > >> >> > >>>> > > > >> >> > >>>> > I completely agree that a first citizen backup and > restore mechanism in Cassandra would be great. > > >> >> > >>>> > > > >> >> > >>>> > I’d like to explore a little bit more the solution > involving Sidecar. While it is true that it would “force” you to have yet > another component running, it definitely comes with a lot of benefits > besides the current other features it provides (control plane, analytics > library, CDC over Kafka, to name some). > > >> >> > >>>> > > > >> >> > >>>> > It allows for a more modular approach to a Cassandra > deployment. In my humble opinion, having smaller, more specialized > processes is the direction we should be following, as opposed to keep > making the core of the project bigger and more complex. > > >> >> > >>>> > It allows for a faster iteration in terms of software. > When it comes to version upgrades, it is definitely way easier to update > just the sidecar than the entire Cassandra core. It is also less risky. > > >> >> > >>>> > Blast radius is also smaller. It is better to have the > Sidecar affected if a backup process stalls/breaks/misbehaves than the > whole Cassandra core. > > >> >> > >>>> > > > >> >> > >>>> > > > >> >> > >>>> > That’s my two cents! > > >> >> > >>>> > Bernardo > > >> >> > >>>> > > > >> >> > >>>> > From: Victor Araujo <[email protected]> > > >> >> > >>>> > Date: Monday, 5 October 2026 at 13:34 > > >> >> > >>>> > To: [email protected] <[email protected]> > > >> >> > >>>> > Subject: [DISCUSS] Backup and restore in Cassandra > > >> >> > >>>> > > > >> >> > >>>> > Hi everyone, > > >> >> > >>>> > > > >> >> > >>>> > I would like to discuss the possibility of having a > built-in backup and restore mechanism in Cassandra, including incremental > backups and support for external object storage such as S3. The main goal > would be to provide a supported solution while keeping the implementation > provider-agnostic and avoiding unnecessary operational overhead. > > >> >> > >>>> > > > >> >> > >>>> > I initially discussed this on Slack and was pointed to > CEP-36 and the previous discussions around external storage and snapshots. > We also discussed whether S3 would be too provider-specific, and whether > Cassandra Sidecar could be part of the solution. My concern with Sidecar is > that introducing another component would increase the operational > footprint, which goes against one of the goals of this work. I also have a > small PoC on top of trunk exploring some of these ideas. > > >> >> > >>>> > > > >> >> > >>>> > I'd be interested in hearing other opinions and any > previous work I should look at. If there is interest in pursuing this, > would a draft CEP be a useful next step so we can start working on the > design in parallel? > > >> >> > >>>> > > > >> >> > >>>> > Link of the first slack discussion about this topic: > https://the-asf.slack.com/archives/CK23JSY2K/p1789021792614639 > > >> >> > >>>> > > > >> >> > >>>> > Thanks, > > >> >> > >>>> > Victor Cavichioli >
