Well ... that's quite a topic. The OP was asking specifically about backup / restore so I was replying in that spirit.
One note for backup / restores, we talk about "snapshots", right? So if you back them up, then one would expect that Sidecar would also offer a way to list them, then to clear them ... It would not be "just about backup and restore". There would be a family of endpoints around snapshot management, very likely. That means we kind of duplicate what is in Cassandra already, if we do not put this into Cassandra directly, then Cassandra would not "see" the snapshots, except those taken locally. If we want to entertain CEP-36 specifically, that would change the picture quite significantly. In either case, snapshots would need to be copied over (SSTables, that is), because we can not do a hard-link from local storage to a S3 bucket. So we would need to copy that, right? If that is the case then we would need to probably throttle how fast it is uploaded to not interfere with other operations and similar. I was completely over-hauling the snapshot management in CASSANDRA-18111, together with Paulo Motta who did a great job reviewing it, everything is centralized in SnapshotManager(MBean). The only tasks touching the disk are TakeSnapshotTask, ClearSnapshotTask and SnapshotLoader which loads them. Their representation / metadata are in memory. So when you list it is not going to S3. That isolates what would need to be accommodated to talk to a cloud. So if you wanted to start to use S3 as a storage, taking CEP-36 into account, we would need to: 1) create "cloude-aware" implementation of org.apache.cassandra.io.util.File / java.nio.file.Path 2) create "S3 TakeSnapshotTaks" which would need to start to copy instead of hardlinking etc. 3) create another version of SnapshotLoader or modify it heavily as currently it is pretty-much wired to talk to local file system 4) Find some way how to "switch" between local / cloud via configuration in a complex manner I still think it holds that we would need to introduce some "class-path modularisation" like you would say, via configuration, what cloud storage provider you want to use and the related libraries would be auto-magically put into the class path. If we had this in place then CEP-36 might be potentially closer to the implementation as these building blocks would be there already. On Tue, Oct 6, 2026 at 6:32 PM Jon Haddad <[email protected]> wrote: > > For the folks who want to put this in sidecar, what are your thoughts on how > that would work with a potential future improvement using s3 as part of its > normal operations? I’m thinking about CEP-36 specifically. > > Thanks! > Jon > > > On Tue, Oct 6, 2026 at 7:37 AM Isaac Reath <[email protected]> wrote: >> >> > If I am not mistaken, Sidecar already exposes a set of >> > endpoints related to the restoration of data from S3 when running some >> > Spark jobs, right? >> >> You are correct Stefan. >> >> I'll echo Stefan's concerns around the weight of adding a cloud provider >> dependency into Cassandra. Especially now, with the increasing volume of CVE >> discovery across all sorts of open-source projects, keeping the dependency >> scope of the core Cassandra process small is a distinct advantage. As an >> operator, it's much easier for me to push out patches across thousands of >> Sidecars than it is to push them across thousands of Cassandra nodes. >> >> I also think that from a resource isolation perspective, running these types >> of maintenance operations out-of-process can create much more predictable >> performance, especially when paired with OS-level resource governance tools >> like cgroups. What a dream it might be if we had out-of-process compaction >> or out-of-process Merkle tree calculation for repair. >> >> I understand your point that adding another Sidecar just for backups can >> increase operational complexity, but I'd argue that cassandra-sidecar is >> shaping up to offer far more than just backups. Today we have Sidecar acting >> as the entrypoint for bulk analytics, CEP-44 bringing CDC publishing to >> Kafka, CEP-40 adding zero-copy online data migration between nodes and with >> CEP-53 and CEP-62 we are getting rolling restarts, configuration management >> and a framework for executing other distributed jobs across a Cassandra >> cluster. Framed this way it's becoming a one-stop-shop for these types of >> operations. I also think that building on top of the operational job >> framework being added with CEP-53 could go a long way towards making the >> implementation easier. >> >> >> On Tue, Oct 6, 2026 at 8:56 AM Victor Araujo <[email protected]> wrote: >>> >>> Hi Bernardo and Stefan, >>> >>> Thanks for the feedback. I agree that there are several benefits to having >>> backup and restore in Sidecar, especially regarding isolation, >>> dependencies, and being able to evolve the feature independently from >>> Cassandra itself. I would be happy to explore and contribute to that >>> direction, potentially in my free time, and I may also be able to bring >>> this discussion to my company later. >>> >>> My main concern is more about the entry point for users. In my current >>> environment, we are actually trying to reduce our use of Sidecars and other >>> additional processes to reduce the overall footprint. I also think there >>> may be users who would prefer having a basic built-in solution in Cassandra >>> rather than having to deploy another component just to get backup and >>> restore. >>> >>> Maybe there is a middle ground here. We could have an opt-in, generic >>> backup/restore capability in Cassandra core, similar in spirit to the >>> unified repair solution, without bringing any third-party cloud >>> dependencies into Cassandra. For example, the core could provide the >>> backup/restore abstractions and lifecycle, with a pluggable storage >>> implementation provided by the user or another component. This could serve >>> as a simple entry-level solution, while a more complete implementation with >>> cloud-specific integrations could live in Sidecar. >>> >>> I don't know yet whether this split makes sense from an architectural >>> perspective, but I think it could be worth exploring. I would be interested >>> in hearing what others think about this approach. >>> >>> Thanks, >>> Victor >>> >>> >>> Em ter., 6 de out. de 2026 às 07:49, Štefan Miklošovič >>> <[email protected]> escreveu: >>>> >>>> I also think that we should investigate how to integrate backup / >>>> restores in Sidecar rather than doing it in Cassandra. >>>> Dependency-wise, I think Sidecar already depends on s3 so that is for >>>> free already. If I am not mistaken, Sidecar already exposes a set of >>>> endpoints related to the restoration of data from S3 when running some >>>> Spark jobs, right? So this would be just a natural extension of that - >>>> a set of endpoints backuping to and restoring from an S3 bucket. >>>> >>>> However, I think that we should make it generic enough in such a way >>>> that if somebody wants to do this against Azure or GCP they could. We >>>> should not "wire" S3 only. I am not sure how to do this though. These >>>> deps are quite "heavy", depending on their own dependency chain ... I >>>> do not have good experience with mixing various cloud libraries. >>>> >>>> What we do in our backup / restore solution is that we modularize it >>>> in such a way that a user can choose what cloud they go to deal with >>>> and they have only that cloud-specific libraries on the class path. >>>> >>>> On Tue, Oct 6, 2026 at 12:14 PM Bernardo Botella >>>> <[email protected]> wrote: >>>> > >>>> > Hi Victor! >>>> > >>>> > Thanks for bringing up this discussion 😊 >>>> > >>>> > I completely agree that a first citizen backup and restore mechanism in >>>> > Cassandra would be great. >>>> > >>>> > I’d like to explore a little bit more the solution involving Sidecar. >>>> > While it is true that it would “force” you to have yet another component >>>> > running, it definitely comes with a lot of benefits besides the current >>>> > other features it provides (control plane, analytics library, CDC over >>>> > Kafka, to name some). >>>> > >>>> > It allows for a more modular approach to a Cassandra deployment. In my >>>> > humble opinion, having smaller, more specialized processes is the >>>> > direction we should be following, as opposed to keep making the core of >>>> > the project bigger and more complex. >>>> > It allows for a faster iteration in terms of software. When it comes to >>>> > version upgrades, it is definitely way easier to update just the sidecar >>>> > than the entire Cassandra core. It is also less risky. >>>> > Blast radius is also smaller. It is better to have the Sidecar affected >>>> > if a backup process stalls/breaks/misbehaves than the whole Cassandra >>>> > core. >>>> > >>>> > >>>> > That’s my two cents! >>>> > Bernardo >>>> > >>>> > From: Victor Araujo <[email protected]> >>>> > Date: Monday, 5 October 2026 at 13:34 >>>> > To: [email protected] <[email protected]> >>>> > Subject: [DISCUSS] Backup and restore in Cassandra >>>> > >>>> > Hi everyone, >>>> > >>>> > I would like to discuss the possibility of having a built-in backup and >>>> > restore mechanism in Cassandra, including incremental backups and >>>> > support for external object storage such as S3. The main goal would be >>>> > to provide a supported solution while keeping the implementation >>>> > provider-agnostic and avoiding unnecessary operational overhead. >>>> > >>>> > I initially discussed this on Slack and was pointed to CEP-36 and the >>>> > previous discussions around external storage and snapshots. We also >>>> > discussed whether S3 would be too provider-specific, and whether >>>> > Cassandra Sidecar could be part of the solution. My concern with Sidecar >>>> > is that introducing another component would increase the operational >>>> > footprint, which goes against one of the goals of this work. I also have >>>> > a small PoC on top of trunk exploring some of these ideas. >>>> > >>>> > I'd be interested in hearing other opinions and any previous work I >>>> > should look at. If there is interest in pursuing this, would a draft CEP >>>> > be a useful next step so we can start working on the design in parallel? >>>> > >>>> > Link of the first slack discussion about this topic: >>>> > https://the-asf.slack.com/archives/CK23JSY2K/p1789021792614639 >>>> > >>>> > Thanks, >>>> > Victor Cavichioli
