In fairness, CEP-36 does explicitly call out that anything that uses a hard link will no longer work once data is written in an alternative location:
> Once triggered any data written to alternative locations will not be included in the snapshotting, incremental backups or other processes that use hard file links. In a world where we have CEP-36 and an S3-backed implementation of ChannelProxy, I would imagine that a call to the ColumnFamilyStore.takeSnapshot function would either no-op (sticking to what is defined in the CEP), wrap a call to take an on-demand backup via AWS Backup, or do something similar to running aws s3 sync. In this world, if you wanted to take a backup across a whole cluster, you could still use Sidecar to manage the coordination. Or you could just use continuous backups on your S3 buckets and call it a day. Depends on the semantics and use-cases of what we're looking for out of a backup solution. CEP-36 offers details around how to get the ChannelProxy interface in place, but leaves the details of how to implement an object store-backed version of the storage engine as an exercise for the reader. I think if we were to get CEP-36 voted on and implemented, we'd need a follow-up CEP that dives into the S3 implementation, or otherwise we'd need to expand the current scope of CEP-36 to cover it. On Wed, Oct 7, 2026 at 6:31 AM Štefan Miklošovič <[email protected]> wrote: > BTW I am still not in favor of putting this to Cassandra, just have > snapshots in Sidecar with its own endpoints, it is way easier that > way. I am pretty tired at this point to go through these discussions > all over again yielding no actual resolution of what we are going to > do. "cloud-aware Cassandra" was so long on a roadmap I am losing hope > we will ever see that. The amount of work needed is enormous, we would > need to go case by case and decide what is still to be written on a > local disk and what goes to the cloud. Some functionality is not even > there: > > 1) How do you atomically rename a file? S3 does not have the concept > of rename. There is some "S3 Express One Zone" which seems to support > RenameObject but that seems quite niche > 2) Hard links ... none, we would need to copy > 3) mmap, how is this done in S3? You can not memory map an object > store, no? We would need to do disk_access_mode: standard > 4) Latency ... > > Commit logs, hints, caches ... Accord stuff. I don't know what else, > all the local disk interaction which is illogical to move to cloud > would need to stay local. (anti)-compaction, repair ... all of that > stuff would be cloud-native too? The performance of that would be ... > tragic? > > I know it is something to aspire for and I do not want to be too > negative about this but I think it makes sense to stop for a while and > actually see what this would mean and how much work it is. > > For that reason I would be doing backup / restores in Sidecar and be > done with that. It is a very well understood domain, s3 deps already > there, endpoints endpointing, a snapshot is a JMX call ... why to > complicate this more really. > > On Tue, Oct 6, 2026 at 9:11 PM Štefan Miklošovič <[email protected]> > wrote: > > > > Well ... that's quite a topic. > > > > The OP was asking specifically about backup / restore so I was > > replying in that spirit. > > > > One note for backup / restores, we talk about "snapshots", right? So > > if you back them up, then one would expect that Sidecar would also > > offer a way to list them, then to clear them ... It would not be "just > > about backup and restore". There would be a family of endpoints around > > snapshot management, very likely. That means we kind of duplicate what > > is in Cassandra already, if we do not put this into Cassandra > > directly, then Cassandra would not "see" the snapshots, except those > > taken locally. > > > > If we want to entertain CEP-36 specifically, that would change the > > picture quite significantly. In either case, snapshots would need to > > be copied over (SSTables, that is), because we can not do a hard-link > > from local storage to a S3 bucket. So we would need to copy that, > > right? If that is the case then we would need to probably throttle how > > fast it is uploaded to not interfere with other operations and > > similar. > > > > I was completely over-hauling the snapshot management in > > CASSANDRA-18111, together with Paulo Motta who did a great job > > reviewing it, everything is centralized in SnapshotManager(MBean). The > > only tasks touching the disk are TakeSnapshotTask, ClearSnapshotTask > > and SnapshotLoader which loads them. Their representation / metadata > > are in memory. So when you list it is not going to S3. That isolates > > what would need to be accommodated to talk to a cloud. So if you > > wanted to start to use S3 as a storage, taking CEP-36 into account, we > > would need to: > > > > 1) create "cloude-aware" implementation of > > org.apache.cassandra.io.util.File / java.nio.file.Path > > 2) create "S3 TakeSnapshotTaks" which would need to start to copy > > instead of hardlinking etc. > > 3) create another version of SnapshotLoader or modify it heavily as > > currently it is pretty-much wired to talk to local file system > > 4) Find some way how to "switch" between local / cloud via > > configuration in a complex manner > > > > I still think it holds that we would need to introduce some > > "class-path modularisation" like you would say, via configuration, > > what cloud storage provider you want to use and the related libraries > > would be auto-magically put into the class path. If we had this in > > place then CEP-36 might be potentially closer to the implementation as > > these building blocks would be there already. > > > > On Tue, Oct 6, 2026 at 6:32 PM Jon Haddad <[email protected]> > wrote: > > > > > > For the folks who want to put this in sidecar, what are your thoughts > on how that would work with a potential future improvement using s3 as part > of its normal operations? I’m thinking about CEP-36 specifically. > > > > > > Thanks! > > > Jon > > > > > > > > > On Tue, Oct 6, 2026 at 7:37 AM Isaac Reath <[email protected]> > wrote: > > >> > > >> > If I am not mistaken, Sidecar already exposes a set of > > >> > endpoints related to the restoration of data from S3 when running > some > > >> > Spark jobs, right? > > >> > > >> You are correct Stefan. > > >> > > >> I'll echo Stefan's concerns around the weight of adding a cloud > provider dependency into Cassandra. Especially now, with the increasing > volume of CVE discovery across all sorts of open-source projects, keeping > the dependency scope of the core Cassandra process small is a distinct > advantage. As an operator, it's much easier for me to push out patches > across thousands of Sidecars than it is to push them across thousands of > Cassandra nodes. > > >> > > >> I also think that from a resource isolation perspective, running > these types of maintenance operations out-of-process can create much more > predictable performance, especially when paired with OS-level resource > governance tools like cgroups. What a dream it might be if we had > out-of-process compaction or out-of-process Merkle tree calculation for > repair. > > >> > > >> I understand your point that adding another Sidecar just for backups > can increase operational complexity, but I'd argue that cassandra-sidecar > is shaping up to offer far more than just backups. Today we have Sidecar > acting as the entrypoint for bulk analytics, CEP-44 bringing CDC publishing > to Kafka, CEP-40 adding zero-copy online data migration between nodes and > with CEP-53 and CEP-62 we are getting rolling restarts, configuration > management and a framework for executing other distributed jobs across a > Cassandra cluster. Framed this way it's becoming a one-stop-shop for these > types of operations. I also think that building on top of the operational > job framework being added with CEP-53 could go a long way towards making > the implementation easier. > > >> > > >> > > >> On Tue, Oct 6, 2026 at 8:56 AM Victor Araujo <[email protected]> > wrote: > > >>> > > >>> Hi Bernardo and Stefan, > > >>> > > >>> Thanks for the feedback. I agree that there are several benefits to > having backup and restore in Sidecar, especially regarding isolation, > dependencies, and being able to evolve the feature independently from > Cassandra itself. I would be happy to explore and contribute to that > direction, potentially in my free time, and I may also be able to bring > this discussion to my company later. > > >>> > > >>> My main concern is more about the entry point for users. In my > current environment, we are actually trying to reduce our use of Sidecars > and other additional processes to reduce the overall footprint. I also > think there may be users who would prefer having a basic built-in solution > in Cassandra rather than having to deploy another component just to get > backup and restore. > > >>> > > >>> Maybe there is a middle ground here. We could have an opt-in, > generic backup/restore capability in Cassandra core, similar in spirit to > the unified repair solution, without bringing any third-party cloud > dependencies into Cassandra. For example, the core could provide the > backup/restore abstractions and lifecycle, with a pluggable storage > implementation provided by the user or another component. This could serve > as a simple entry-level solution, while a more complete implementation with > cloud-specific integrations could live in Sidecar. > > >>> > > >>> I don't know yet whether this split makes sense from an > architectural perspective, but I think it could be worth exploring. I would > be interested in hearing what others think about this approach. > > >>> > > >>> Thanks, > > >>> Victor > > >>> > > >>> > > >>> Em ter., 6 de out. de 2026 às 07:49, Štefan Miklošovič < > [email protected]> escreveu: > > >>>> > > >>>> I also think that we should investigate how to integrate backup / > > >>>> restores in Sidecar rather than doing it in Cassandra. > > >>>> Dependency-wise, I think Sidecar already depends on s3 so that is > for > > >>>> free already. If I am not mistaken, Sidecar already exposes a set of > > >>>> endpoints related to the restoration of data from S3 when running > some > > >>>> Spark jobs, right? So this would be just a natural extension of > that - > > >>>> a set of endpoints backuping to and restoring from an S3 bucket. > > >>>> > > >>>> However, I think that we should make it generic enough in such a way > > >>>> that if somebody wants to do this against Azure or GCP they could. > We > > >>>> should not "wire" S3 only. I am not sure how to do this though. > These > > >>>> deps are quite "heavy", depending on their own dependency chain ... > I > > >>>> do not have good experience with mixing various cloud libraries. > > >>>> > > >>>> What we do in our backup / restore solution is that we modularize it > > >>>> in such a way that a user can choose what cloud they go to deal with > > >>>> and they have only that cloud-specific libraries on the class path. > > >>>> > > >>>> On Tue, Oct 6, 2026 at 12:14 PM Bernardo Botella > > >>>> <[email protected]> wrote: > > >>>> > > > >>>> > Hi Victor! > > >>>> > > > >>>> > Thanks for bringing up this discussion 😊 > > >>>> > > > >>>> > I completely agree that a first citizen backup and restore > mechanism in Cassandra would be great. > > >>>> > > > >>>> > I’d like to explore a little bit more the solution involving > Sidecar. While it is true that it would “force” you to have yet another > component running, it definitely comes with a lot of benefits besides the > current other features it provides (control plane, analytics library, CDC > over Kafka, to name some). > > >>>> > > > >>>> > It allows for a more modular approach to a Cassandra deployment. > In my humble opinion, having smaller, more specialized processes is the > direction we should be following, as opposed to keep making the core of the > project bigger and more complex. > > >>>> > It allows for a faster iteration in terms of software. When it > comes to version upgrades, it is definitely way easier to update just the > sidecar than the entire Cassandra core. It is also less risky. > > >>>> > Blast radius is also smaller. It is better to have the Sidecar > affected if a backup process stalls/breaks/misbehaves than the whole > Cassandra core. > > >>>> > > > >>>> > > > >>>> > That’s my two cents! > > >>>> > Bernardo > > >>>> > > > >>>> > From: Victor Araujo <[email protected]> > > >>>> > Date: Monday, 5 October 2026 at 13:34 > > >>>> > To: [email protected] <[email protected]> > > >>>> > Subject: [DISCUSS] Backup and restore in Cassandra > > >>>> > > > >>>> > Hi everyone, > > >>>> > > > >>>> > I would like to discuss the possibility of having a built-in > backup and restore mechanism in Cassandra, including incremental backups > and support for external object storage such as S3. The main goal would be > to provide a supported solution while keeping the implementation > provider-agnostic and avoiding unnecessary operational overhead. > > >>>> > > > >>>> > I initially discussed this on Slack and was pointed to CEP-36 and > the previous discussions around external storage and snapshots. We also > discussed whether S3 would be too provider-specific, and whether Cassandra > Sidecar could be part of the solution. My concern with Sidecar is that > introducing another component would increase the operational footprint, > which goes against one of the goals of this work. I also have a small PoC > on top of trunk exploring some of these ideas. > > >>>> > > > >>>> > I'd be interested in hearing other opinions and any previous work > I should look at. If there is interest in pursuing this, would a draft CEP > be a useful next step so we can start working on the design in parallel? > > >>>> > > > >>>> > Link of the first slack discussion about this topic: > https://the-asf.slack.com/archives/CK23JSY2K/p1789021792614639 > > >>>> > > > >>>> > Thanks, > > >>>> > Victor Cavichioli >
