BTW I am still not in favor of putting this to Cassandra, just have snapshots in Sidecar with its own endpoints, it is way easier that way. I am pretty tired at this point to go through these discussions all over again yielding no actual resolution of what we are going to do. "cloud-aware Cassandra" was so long on a roadmap I am losing hope we will ever see that. The amount of work needed is enormous, we would need to go case by case and decide what is still to be written on a local disk and what goes to the cloud. Some functionality is not even there:
1) How do you atomically rename a file? S3 does not have the concept of rename. There is some "S3 Express One Zone" which seems to support RenameObject but that seems quite niche 2) Hard links ... none, we would need to copy 3) mmap, how is this done in S3? You can not memory map an object store, no? We would need to do disk_access_mode: standard 4) Latency ... Commit logs, hints, caches ... Accord stuff. I don't know what else, all the local disk interaction which is illogical to move to cloud would need to stay local. (anti)-compaction, repair ... all of that stuff would be cloud-native too? The performance of that would be ... tragic? I know it is something to aspire for and I do not want to be too negative about this but I think it makes sense to stop for a while and actually see what this would mean and how much work it is. For that reason I would be doing backup / restores in Sidecar and be done with that. It is a very well understood domain, s3 deps already there, endpoints endpointing, a snapshot is a JMX call ... why to complicate this more really. On Tue, Oct 6, 2026 at 9:11 PM Štefan Miklošovič <[email protected]> wrote: > > Well ... that's quite a topic. > > The OP was asking specifically about backup / restore so I was > replying in that spirit. > > One note for backup / restores, we talk about "snapshots", right? So > if you back them up, then one would expect that Sidecar would also > offer a way to list them, then to clear them ... It would not be "just > about backup and restore". There would be a family of endpoints around > snapshot management, very likely. That means we kind of duplicate what > is in Cassandra already, if we do not put this into Cassandra > directly, then Cassandra would not "see" the snapshots, except those > taken locally. > > If we want to entertain CEP-36 specifically, that would change the > picture quite significantly. In either case, snapshots would need to > be copied over (SSTables, that is), because we can not do a hard-link > from local storage to a S3 bucket. So we would need to copy that, > right? If that is the case then we would need to probably throttle how > fast it is uploaded to not interfere with other operations and > similar. > > I was completely over-hauling the snapshot management in > CASSANDRA-18111, together with Paulo Motta who did a great job > reviewing it, everything is centralized in SnapshotManager(MBean). The > only tasks touching the disk are TakeSnapshotTask, ClearSnapshotTask > and SnapshotLoader which loads them. Their representation / metadata > are in memory. So when you list it is not going to S3. That isolates > what would need to be accommodated to talk to a cloud. So if you > wanted to start to use S3 as a storage, taking CEP-36 into account, we > would need to: > > 1) create "cloude-aware" implementation of > org.apache.cassandra.io.util.File / java.nio.file.Path > 2) create "S3 TakeSnapshotTaks" which would need to start to copy > instead of hardlinking etc. > 3) create another version of SnapshotLoader or modify it heavily as > currently it is pretty-much wired to talk to local file system > 4) Find some way how to "switch" between local / cloud via > configuration in a complex manner > > I still think it holds that we would need to introduce some > "class-path modularisation" like you would say, via configuration, > what cloud storage provider you want to use and the related libraries > would be auto-magically put into the class path. If we had this in > place then CEP-36 might be potentially closer to the implementation as > these building blocks would be there already. > > On Tue, Oct 6, 2026 at 6:32 PM Jon Haddad <[email protected]> wrote: > > > > For the folks who want to put this in sidecar, what are your thoughts on > > how that would work with a potential future improvement using s3 as part of > > its normal operations? I’m thinking about CEP-36 specifically. > > > > Thanks! > > Jon > > > > > > On Tue, Oct 6, 2026 at 7:37 AM Isaac Reath <[email protected]> wrote: > >> > >> > If I am not mistaken, Sidecar already exposes a set of > >> > endpoints related to the restoration of data from S3 when running some > >> > Spark jobs, right? > >> > >> You are correct Stefan. > >> > >> I'll echo Stefan's concerns around the weight of adding a cloud provider > >> dependency into Cassandra. Especially now, with the increasing volume of > >> CVE discovery across all sorts of open-source projects, keeping the > >> dependency scope of the core Cassandra process small is a distinct > >> advantage. As an operator, it's much easier for me to push out patches > >> across thousands of Sidecars than it is to push them across thousands of > >> Cassandra nodes. > >> > >> I also think that from a resource isolation perspective, running these > >> types of maintenance operations out-of-process can create much more > >> predictable performance, especially when paired with OS-level resource > >> governance tools like cgroups. What a dream it might be if we had > >> out-of-process compaction or out-of-process Merkle tree calculation for > >> repair. > >> > >> I understand your point that adding another Sidecar just for backups can > >> increase operational complexity, but I'd argue that cassandra-sidecar is > >> shaping up to offer far more than just backups. Today we have Sidecar > >> acting as the entrypoint for bulk analytics, CEP-44 bringing CDC > >> publishing to Kafka, CEP-40 adding zero-copy online data migration between > >> nodes and with CEP-53 and CEP-62 we are getting rolling restarts, > >> configuration management and a framework for executing other distributed > >> jobs across a Cassandra cluster. Framed this way it's becoming a > >> one-stop-shop for these types of operations. I also think that building on > >> top of the operational job framework being added with CEP-53 could go a > >> long way towards making the implementation easier. > >> > >> > >> On Tue, Oct 6, 2026 at 8:56 AM Victor Araujo <[email protected]> > >> wrote: > >>> > >>> Hi Bernardo and Stefan, > >>> > >>> Thanks for the feedback. I agree that there are several benefits to > >>> having backup and restore in Sidecar, especially regarding isolation, > >>> dependencies, and being able to evolve the feature independently from > >>> Cassandra itself. I would be happy to explore and contribute to that > >>> direction, potentially in my free time, and I may also be able to bring > >>> this discussion to my company later. > >>> > >>> My main concern is more about the entry point for users. In my current > >>> environment, we are actually trying to reduce our use of Sidecars and > >>> other additional processes to reduce the overall footprint. I also think > >>> there may be users who would prefer having a basic built-in solution in > >>> Cassandra rather than having to deploy another component just to get > >>> backup and restore. > >>> > >>> Maybe there is a middle ground here. We could have an opt-in, generic > >>> backup/restore capability in Cassandra core, similar in spirit to the > >>> unified repair solution, without bringing any third-party cloud > >>> dependencies into Cassandra. For example, the core could provide the > >>> backup/restore abstractions and lifecycle, with a pluggable storage > >>> implementation provided by the user or another component. This could > >>> serve as a simple entry-level solution, while a more complete > >>> implementation with cloud-specific integrations could live in Sidecar. > >>> > >>> I don't know yet whether this split makes sense from an architectural > >>> perspective, but I think it could be worth exploring. I would be > >>> interested in hearing what others think about this approach. > >>> > >>> Thanks, > >>> Victor > >>> > >>> > >>> Em ter., 6 de out. de 2026 às 07:49, Štefan Miklošovič > >>> <[email protected]> escreveu: > >>>> > >>>> I also think that we should investigate how to integrate backup / > >>>> restores in Sidecar rather than doing it in Cassandra. > >>>> Dependency-wise, I think Sidecar already depends on s3 so that is for > >>>> free already. If I am not mistaken, Sidecar already exposes a set of > >>>> endpoints related to the restoration of data from S3 when running some > >>>> Spark jobs, right? So this would be just a natural extension of that - > >>>> a set of endpoints backuping to and restoring from an S3 bucket. > >>>> > >>>> However, I think that we should make it generic enough in such a way > >>>> that if somebody wants to do this against Azure or GCP they could. We > >>>> should not "wire" S3 only. I am not sure how to do this though. These > >>>> deps are quite "heavy", depending on their own dependency chain ... I > >>>> do not have good experience with mixing various cloud libraries. > >>>> > >>>> What we do in our backup / restore solution is that we modularize it > >>>> in such a way that a user can choose what cloud they go to deal with > >>>> and they have only that cloud-specific libraries on the class path. > >>>> > >>>> On Tue, Oct 6, 2026 at 12:14 PM Bernardo Botella > >>>> <[email protected]> wrote: > >>>> > > >>>> > Hi Victor! > >>>> > > >>>> > Thanks for bringing up this discussion 😊 > >>>> > > >>>> > I completely agree that a first citizen backup and restore mechanism > >>>> > in Cassandra would be great. > >>>> > > >>>> > I’d like to explore a little bit more the solution involving Sidecar. > >>>> > While it is true that it would “force” you to have yet another > >>>> > component running, it definitely comes with a lot of benefits besides > >>>> > the current other features it provides (control plane, analytics > >>>> > library, CDC over Kafka, to name some). > >>>> > > >>>> > It allows for a more modular approach to a Cassandra deployment. In my > >>>> > humble opinion, having smaller, more specialized processes is the > >>>> > direction we should be following, as opposed to keep making the core > >>>> > of the project bigger and more complex. > >>>> > It allows for a faster iteration in terms of software. When it comes > >>>> > to version upgrades, it is definitely way easier to update just the > >>>> > sidecar than the entire Cassandra core. It is also less risky. > >>>> > Blast radius is also smaller. It is better to have the Sidecar > >>>> > affected if a backup process stalls/breaks/misbehaves than the whole > >>>> > Cassandra core. > >>>> > > >>>> > > >>>> > That’s my two cents! > >>>> > Bernardo > >>>> > > >>>> > From: Victor Araujo <[email protected]> > >>>> > Date: Monday, 5 October 2026 at 13:34 > >>>> > To: [email protected] <[email protected]> > >>>> > Subject: [DISCUSS] Backup and restore in Cassandra > >>>> > > >>>> > Hi everyone, > >>>> > > >>>> > I would like to discuss the possibility of having a built-in backup > >>>> > and restore mechanism in Cassandra, including incremental backups and > >>>> > support for external object storage such as S3. The main goal would be > >>>> > to provide a supported solution while keeping the implementation > >>>> > provider-agnostic and avoiding unnecessary operational overhead. > >>>> > > >>>> > I initially discussed this on Slack and was pointed to CEP-36 and the > >>>> > previous discussions around external storage and snapshots. We also > >>>> > discussed whether S3 would be too provider-specific, and whether > >>>> > Cassandra Sidecar could be part of the solution. My concern with > >>>> > Sidecar is that introducing another component would increase the > >>>> > operational footprint, which goes against one of the goals of this > >>>> > work. I also have a small PoC on top of trunk exploring some of these > >>>> > ideas. > >>>> > > >>>> > I'd be interested in hearing other opinions and any previous work I > >>>> > should look at. If there is interest in pursuing this, would a draft > >>>> > CEP be a useful next step so we can start working on the design in > >>>> > parallel? > >>>> > > >>>> > Link of the first slack discussion about this topic: > >>>> > https://the-asf.slack.com/archives/CK23JSY2K/p1789021792614639 > >>>> > > >>>> > Thanks, > >>>> > Victor Cavichioli
