I think we should avoid adding more low grade operational resource consuming things to the coordinator/storage hybrid mega-JDK we already have. So +1 for doing this out-of-core-process.
No sense in having backup / restore fighting with critical path I/O in the JDK space; let the kernel scheduler work it out. Tends to be way smarter than our "throw it into a ThreadPool and pray" approach. :) On Fri, Oct 9, 2026, at 9:35 PM, Paulo Motta wrote: > +1 to do backups via sidecar! The only issue is this requires users wanting > to use this feature to adopt sidecar who otherwise wouldn't due to the > increased operational complexity of managing multiple processes. Ideally we > should make sidecar easier to deploy/configure and more modular, so users can > easily adopt it to enable specific functions (ie. backup). > > On Fri, Oct 9, 2026 at 3:34 AM Štefan Miklošovič <[email protected]> > wrote: >> Paulo, what do you think about approach: since snapshots are cheap >> (because hardlinks), we can keep snapshotting like that in Cassandra >> while Sidecar would be responsible for uploading them to s3? Once all >> is uploaded then local hardlinked snapshot can be removed. >> >> I think it is safer to do it like that because what if the upload from >> Cassandra fails in the middle (if it was Cassandra's job to upload >> it)? I do not want to put additional logic and stress Cassandra even >> more with stuff like this. However, for Sidecar it would be >> justifiable. >> >> Cassandra would only provide enhanced manifests and JMX methods for >> retrieving them from Sidecar and all logic around this will be in >> Sidecar, since it sees local disks anyway. >> >> On Fri, Oct 9, 2026 at 1:22 AM Paulo Motta <[email protected]> wrote: >> > >> > I like this suggestion of enhanced manifest + copy snapshots to local >> > storage as an initial step. This could provide a Cassandra baseline that >> > sidecar or other tools could use to implement more intelligent (ie. >> > cluster-wide) backups. >> > >> > Once this works for local storage, users could potentially mount an >> > s3/nfs/ebs file system to transparently transfer snapshots to remote >> > storage without Cassandra needing to handle this part. >> > >> > On Thu, 8 Oct 2026 at 14:42 Štefan Miklošovič <[email protected]> >> > wrote: >> >> >> >> Yep. >> >> >> >> So I think that we should work "backwards". First we should probably >> >> do manifest-v2.json which is way more comfortable to work with, >> >> contains hashes of files, whole schema, each file ... any other >> >> metadata. >> >> >> >> Once this is in place then we would do "copying to a local disk" with >> >> this deduplication in place and eventually we would abstract away the >> >> local disk to work with anything else. >> >> >> >> On Thu, Oct 8, 2026 at 8:35 PM Isaac Reath <[email protected]> wrote: >> >> > >> >> > > We have solved this already in our tooling, each snapshot has a >> >> > manifest containing all SSTables it logically consists of. So if you >> >> > go to remove a snapshot, you can remove a particular SSTable only in >> >> > case it is not referenced in any other manifest. Otherwise your other >> >> > snapshot would be lacking sstables. >> >> > >> >> > This is pretty much how Medusa handles this. Medusa uploads a manifest >> >> > with each backup to S3 with the files in the backup. SSTables are only >> >> > removed from S3 when `medusa purge` runs. `medusa purge` in turn >> >> > deletes the files which are not referenced in any active backup >> >> > manifest (and have had their configurable grace period expire to ensure >> >> > that purge doesn't race with an ongoing backup). I think any tooling we >> >> > build in-project for this (cassandra, sidecar or otherwise) should >> >> > consider a similar approach. >> >> > >> >> > On Thu, Oct 8, 2026 at 2:16 PM Štefan Miklošovič >> >> > <[email protected]> wrote: >> >> >> >> >> >> Regarding snapshots, I want to say there is one conceptual "problem" >> >> >> which seems to be more complicated than it seems. >> >> >> >> >> >> Currently, a snapshot is a bunch of hard links. Hardlinks don't >> >> >> translate to S3. So we need to copy. Alright, so imagine we have this >> >> >> snapshot: >> >> >> >> >> >> snap1 >> >> >> >> >> >> SSTable1 >> >> >> SSTable2 >> >> >> SSTable3 >> >> >> >> >> >> snap2 >> >> >> >> >> >> SSTable3 >> >> >> SSTable4 >> >> >> SSTable5 >> >> >> >> >> >> snap1 is uploaded, then snap2 is going to be, sstable 3 is in s3 >> >> >> already so we skip that, then upload sstable 4 and 5. >> >> >> >> >> >> Now you go to remove snap2. But you can not remove sstable 3. Because >> >> >> it is logically contained in snap1. Ha. >> >> >> >> >> >> In other words, you would need to have some global way to track this. >> >> >> >> >> >> We have solved this already in our tooling, each snapshot has a >> >> >> manifest containing all SSTables it logically consists of. So if you >> >> >> go to remove a snapshot, you can remove a particular SSTable only in >> >> >> case it is not referenced in any other manifest. Otherwise your other >> >> >> snapshot would be lacking sstables. >> >> >> >> >> >> So this "just upload snapshots to s3" is way more involved than it >> >> >> seems, we would need to solve stuff like this. >> >> >> >> >> >> Or some other approach can be taken? I have not done extensive >> >> >> research on how other tooling deals with this. Maybe there is some >> >> >> elegant workaround. >> >> >> >> >> >> On Thu, Oct 8, 2026 at 7:59 PM Štefan Miklošovič >> >> >> <[email protected]> wrote: >> >> >> > >> >> >> > These are all valid points you raised. I think that CEP-36 can be >> >> >> > "limited" to just providing a pluggable way for ChannelProxy. Serving >> >> >> > from S3 directly or not, there will still need to be a way to get a >> >> >> > cloud-specific implementation of ChannelProxy for reads / write of >> >> >> > any >> >> >> > kind. Later on we can decide the overall architecture of that but I >> >> >> > would reserve it for another CEP, that work will most probably be way >> >> >> > more involved than "just" a ChannelProxy creation. >> >> >> > >> >> >> > If ChannelProxy is pluggable already, we can start to use this for >> >> >> > snapshot uploading _if somebody ever wants to go that route_, I would >> >> >> > just need to tweak the snapshotting stuff to be ChannelProxy-aware >> >> >> > instead of on "File"-s. The default impl would be on FileChannel as >> >> >> > we >> >> >> > know it. >> >> >> > >> >> >> > On Thu, Oct 8, 2026 at 5:02 PM Jon Haddad <[email protected]> >> >> >> > wrote: >> >> >> > > >> >> >> > > I didn't realize the CEP hadn't been updated. I think I was >> >> >> > > mixing up the convo on the ML and what was actually in the CEP. I >> >> >> > > had remembered it behaving much closer to clickhouse, where it >> >> >> > > keeps the data on disk and pushes the compacted files to S3. I >> >> >> > > don't think serving directly off S3 is going to be much use to >> >> >> > > anyone. The latency will be awful, and they issue a backoff >> >> >> > > warning fairly quickly. In my mind it only makes sense to treat >> >> >> > > S3 as the authority and cache SStables to local disk. I think the >> >> >> > > Astra folks may have done something along these lines, I'd be >> >> >> > > willing to bet there's a ton of knowledge there as they >> >> >> > > operationalized it successfully. >> >> >> > > >> >> >> > > Anyways - the reason I asked my question, was to frame it in the >> >> >> > > context of a longer term project strategy. Asking more directly - >> >> >> > > If we're going to be pushing and pulling from S3 during >> >> >> > > compaction, and it's a major feature, would that go in Sidecar? >> >> >> > > It certainly could, but I'm concerned about having a critical >> >> >> > > component of C* living it's own independent lifecycle. >> >> >> > > >> >> >> > > Example of why I get concerned: >> >> >> > > https://the-asf.slack.com/archives/C07V0FK0M2P/p1788964591823929 >> >> >> > > >> >> >> > > Imo, having 2 projects that are tightly coupled, but developed >> >> >> > > independently, is going to be fairly high risk. I really don't >> >> >> > > like having responsibility spread this way, and I probably >> >> >> > > wouldn't want to run it in prod. >> >> >> > > >> >> >> > > Jon >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > On Wed, Oct 7, 2026 at 7:15 AM Štefan Miklošovič >> >> >> > > <[email protected]> wrote: >> >> >> > >> >> >> >> > >> I have a draft of CEP-36 here (1). S3 deps + proxy impl is meant >> >> >> > >> to be >> >> >> > >> implemented in a different project (2) which once built is added >> >> >> > >> to >> >> >> > >> Cassandra via custom ClassLoader so it is all encapsulated and we >> >> >> > >> can >> >> >> > >> not have any dependency clashes. >> >> >> > >> >> >> >> > >> There is the impl of s3 proxy with e2e tests. It shows how >> >> >> > >> Cassandra >> >> >> > >> starts to serve reads from s3. When it comes to writes, these >> >> >> > >> SSTables >> >> >> > >> will be created locally as they are not and then "offloaded" to >> >> >> > >> S3 for >> >> >> > >> reads. Once SSTable is created it is immutable so we can upload >> >> >> > >> it and >> >> >> > >> then switch SSTableReader to read via S3 proxy. >> >> >> > >> >> >> >> > >> At least that is the idea. Heavy demo. Dragons everywhere. >> >> >> > >> AI-driven a >> >> >> > >> lot. It more _does not work_ than does :D But it genuinely starts >> >> >> > >> to >> >> >> > >> read from S3 via proxy which is quite ... promising, but I think >> >> >> > >> that >> >> >> > >> if we want to seriously consider CEP-36 and implement it fully >> >> >> > >> that is >> >> >> > >> a serious piece of work for two quarters :D >> >> >> > >> >> >> >> > >> Give me some time here please, like a week or two, I would like to >> >> >> > >> explore this in more detail. Maybe it will be better to modify >> >> >> > >> SnapshotManager to work with ChannelProxy instead of File and when >> >> >> > >> taking snapshots we would use S3 ChannelProxy plus we would tweak >> >> >> > >> the >> >> >> > >> hardlinking bit. >> >> >> > >> >> >> >> > >> You can of course do this all in Sidecar, I just want to see where >> >> >> > >> this gets us. You would have two ways to choose from (and CEP-36 >> >> >> > >> building blocks would be in place to extend on further). >> >> >> > >> >> >> >> > >> (1) https://github.com/apache/cassandra/pull/5238/changes >> >> >> > >> (2) https://github.com/smiklosovic/cassandra-s3-channel-proxy >> >> >> > >> >> >> >> > >> On Wed, Oct 7, 2026 at 3:30 PM Isaac Reath <[email protected]> >> >> >> > >> wrote: >> >> >> > >> > >> >> >> > >> > In fairness, CEP-36 does explicitly call out that anything that >> >> >> > >> > uses a hard link will no longer work once data is written in an >> >> >> > >> > alternative location: >> >> >> > >> > >> >> >> > >> > > Once triggered any data written to alternative locations will >> >> >> > >> > > not be included in the snapshotting, incremental backups or >> >> >> > >> > > other processes that use hard file links. >> >> >> > >> > >> >> >> > >> > In a world where we have CEP-36 and an S3-backed implementation >> >> >> > >> > of ChannelProxy, I would imagine that a call to the >> >> >> > >> > ColumnFamilyStore.takeSnapshot function would either no-op >> >> >> > >> > (sticking to what is defined in the CEP), wrap a call to take >> >> >> > >> > an on-demand backup via AWS Backup, or do something similar to >> >> >> > >> > running aws s3 sync. In this world, if you wanted to take a >> >> >> > >> > backup across a whole cluster, you could still use Sidecar to >> >> >> > >> > manage the coordination. Or you could just use continuous >> >> >> > >> > backups on your S3 buckets and call it a day. Depends on the >> >> >> > >> > semantics and use-cases of what we're looking for out of a >> >> >> > >> > backup solution. >> >> >> > >> > >> >> >> > >> > CEP-36 offers details around how to get the ChannelProxy >> >> >> > >> > interface in place, but leaves the details of how to implement >> >> >> > >> > an object store-backed version of the storage engine as an >> >> >> > >> > exercise for the reader. I think if we were to get CEP-36 voted >> >> >> > >> > on and implemented, we'd need a follow-up CEP that dives into >> >> >> > >> > the S3 implementation, or otherwise we'd need to expand the >> >> >> > >> > current scope of CEP-36 to cover it. >> >> >> > >> > >> >> >> > >> > On Wed, Oct 7, 2026 at 6:31 AM Štefan Miklošovič >> >> >> > >> > <[email protected]> wrote: >> >> >> > >> >> >> >> >> > >> >> BTW I am still not in favor of putting this to Cassandra, just >> >> >> > >> >> have >> >> >> > >> >> snapshots in Sidecar with its own endpoints, it is way easier >> >> >> > >> >> that >> >> >> > >> >> way. I am pretty tired at this point to go through these >> >> >> > >> >> discussions >> >> >> > >> >> all over again yielding no actual resolution of what we are >> >> >> > >> >> going to >> >> >> > >> >> do. "cloud-aware Cassandra" was so long on a roadmap I am >> >> >> > >> >> losing hope >> >> >> > >> >> we will ever see that. The amount of work needed is enormous, >> >> >> > >> >> we would >> >> >> > >> >> need to go case by case and decide what is still to be written >> >> >> > >> >> on a >> >> >> > >> >> local disk and what goes to the cloud. Some functionality is >> >> >> > >> >> not even >> >> >> > >> >> there: >> >> >> > >> >> >> >> >> > >> >> 1) How do you atomically rename a file? S3 does not have the >> >> >> > >> >> concept >> >> >> > >> >> of rename. There is some "S3 Express One Zone" which seems to >> >> >> > >> >> support >> >> >> > >> >> RenameObject but that seems quite niche >> >> >> > >> >> 2) Hard links ... none, we would need to copy >> >> >> > >> >> 3) mmap, how is this done in S3? You can not memory map an >> >> >> > >> >> object >> >> >> > >> >> store, no? We would need to do disk_access_mode: standard >> >> >> > >> >> 4) Latency ... >> >> >> > >> >> >> >> >> > >> >> Commit logs, hints, caches ... Accord stuff. I don't know what >> >> >> > >> >> else, >> >> >> > >> >> all the local disk interaction which is illogical to move to >> >> >> > >> >> cloud >> >> >> > >> >> would need to stay local. (anti)-compaction, repair ... all of >> >> >> > >> >> that >> >> >> > >> >> stuff would be cloud-native too? The performance of that would >> >> >> > >> >> be ... >> >> >> > >> >> tragic? >> >> >> > >> >> >> >> >> > >> >> I know it is something to aspire for and I do not want to be >> >> >> > >> >> too >> >> >> > >> >> negative about this but I think it makes sense to stop for a >> >> >> > >> >> while and >> >> >> > >> >> actually see what this would mean and how much work it is. >> >> >> > >> >> >> >> >> > >> >> For that reason I would be doing backup / restores in Sidecar >> >> >> > >> >> and be >> >> >> > >> >> done with that. It is a very well understood domain, s3 deps >> >> >> > >> >> already >> >> >> > >> >> there, endpoints endpointing, a snapshot is a JMX call ... why >> >> >> > >> >> to >> >> >> > >> >> complicate this more really. >> >> >> > >> >> >> >> >> > >> >> On Tue, Oct 6, 2026 at 9:11 PM Štefan Miklošovič >> >> >> > >> >> <[email protected]> wrote: >> >> >> > >> >> > >> >> >> > >> >> > Well ... that's quite a topic. >> >> >> > >> >> > >> >> >> > >> >> > The OP was asking specifically about backup / restore so I >> >> >> > >> >> > was >> >> >> > >> >> > replying in that spirit. >> >> >> > >> >> > >> >> >> > >> >> > One note for backup / restores, we talk about "snapshots", >> >> >> > >> >> > right? So >> >> >> > >> >> > if you back them up, then one would expect that Sidecar >> >> >> > >> >> > would also >> >> >> > >> >> > offer a way to list them, then to clear them ... It would >> >> >> > >> >> > not be "just >> >> >> > >> >> > about backup and restore". There would be a family of >> >> >> > >> >> > endpoints around >> >> >> > >> >> > snapshot management, very likely. That means we kind of >> >> >> > >> >> > duplicate what >> >> >> > >> >> > is in Cassandra already, if we do not put this into Cassandra >> >> >> > >> >> > directly, then Cassandra would not "see" the snapshots, >> >> >> > >> >> > except those >> >> >> > >> >> > taken locally. >> >> >> > >> >> > >> >> >> > >> >> > If we want to entertain CEP-36 specifically, that would >> >> >> > >> >> > change the >> >> >> > >> >> > picture quite significantly. In either case, snapshots would >> >> >> > >> >> > need to >> >> >> > >> >> > be copied over (SSTables, that is), because we can not do a >> >> >> > >> >> > hard-link >> >> >> > >> >> > from local storage to a S3 bucket. So we would need to copy >> >> >> > >> >> > that, >> >> >> > >> >> > right? If that is the case then we would need to probably >> >> >> > >> >> > throttle how >> >> >> > >> >> > fast it is uploaded to not interfere with other operations >> >> >> > >> >> > and >> >> >> > >> >> > similar. >> >> >> > >> >> > >> >> >> > >> >> > I was completely over-hauling the snapshot management in >> >> >> > >> >> > CASSANDRA-18111, together with Paulo Motta who did a great >> >> >> > >> >> > job >> >> >> > >> >> > reviewing it, everything is centralized in >> >> >> > >> >> > SnapshotManager(MBean). The >> >> >> > >> >> > only tasks touching the disk are TakeSnapshotTask, >> >> >> > >> >> > ClearSnapshotTask >> >> >> > >> >> > and SnapshotLoader which loads them. Their representation / >> >> >> > >> >> > metadata >> >> >> > >> >> > are in memory. So when you list it is not going to S3. That >> >> >> > >> >> > isolates >> >> >> > >> >> > what would need to be accommodated to talk to a cloud. So if >> >> >> > >> >> > you >> >> >> > >> >> > wanted to start to use S3 as a storage, taking CEP-36 into >> >> >> > >> >> > account, we >> >> >> > >> >> > would need to: >> >> >> > >> >> > >> >> >> > >> >> > 1) create "cloude-aware" implementation of >> >> >> > >> >> > org.apache.cassandra.io.util.File / java.nio.file.Path >> >> >> > >> >> > 2) create "S3 TakeSnapshotTaks" which would need to start to >> >> >> > >> >> > copy >> >> >> > >> >> > instead of hardlinking etc. >> >> >> > >> >> > 3) create another version of SnapshotLoader or modify it >> >> >> > >> >> > heavily as >> >> >> > >> >> > currently it is pretty-much wired to talk to local file >> >> >> > >> >> > system >> >> >> > >> >> > 4) Find some way how to "switch" between local / cloud via >> >> >> > >> >> > configuration in a complex manner >> >> >> > >> >> > >> >> >> > >> >> > I still think it holds that we would need to introduce some >> >> >> > >> >> > "class-path modularisation" like you would say, via >> >> >> > >> >> > configuration, >> >> >> > >> >> > what cloud storage provider you want to use and the related >> >> >> > >> >> > libraries >> >> >> > >> >> > would be auto-magically put into the class path. If we had >> >> >> > >> >> > this in >> >> >> > >> >> > place then CEP-36 might be potentially closer to the >> >> >> > >> >> > implementation as >> >> >> > >> >> > these building blocks would be there already. >> >> >> > >> >> > >> >> >> > >> >> > On Tue, Oct 6, 2026 at 6:32 PM Jon Haddad >> >> >> > >> >> > <[email protected]> wrote: >> >> >> > >> >> > > >> >> >> > >> >> > > For the folks who want to put this in sidecar, what are >> >> >> > >> >> > > your thoughts on how that would work with a potential >> >> >> > >> >> > > future improvement using s3 as part of its normal >> >> >> > >> >> > > operations? I’m thinking about CEP-36 specifically. >> >> >> > >> >> > > >> >> >> > >> >> > > Thanks! >> >> >> > >> >> > > Jon >> >> >> > >> >> > > >> >> >> > >> >> > > >> >> >> > >> >> > > On Tue, Oct 6, 2026 at 7:37 AM Isaac Reath >> >> >> > >> >> > > <[email protected]> wrote: >> >> >> > >> >> > >> >> >> >> > >> >> > >> > If I am not mistaken, Sidecar already exposes a set of >> >> >> > >> >> > >> > endpoints related to the restoration of data from S3 >> >> >> > >> >> > >> > when running some >> >> >> > >> >> > >> > Spark jobs, right? >> >> >> > >> >> > >> >> >> >> > >> >> > >> You are correct Stefan. >> >> >> > >> >> > >> >> >> >> > >> >> > >> I'll echo Stefan's concerns around the weight of adding a >> >> >> > >> >> > >> cloud provider dependency into Cassandra. Especially now, >> >> >> > >> >> > >> with the increasing volume of CVE discovery across all >> >> >> > >> >> > >> sorts of open-source projects, keeping the dependency >> >> >> > >> >> > >> scope of the core Cassandra process small is a distinct >> >> >> > >> >> > >> advantage. As an operator, it's much easier for me to >> >> >> > >> >> > >> push out patches across thousands of Sidecars than it is >> >> >> > >> >> > >> to push them across thousands of Cassandra nodes. >> >> >> > >> >> > >> >> >> >> > >> >> > >> I also think that from a resource isolation perspective, >> >> >> > >> >> > >> running these types of maintenance operations >> >> >> > >> >> > >> out-of-process can create much more predictable >> >> >> > >> >> > >> performance, especially when paired with OS-level >> >> >> > >> >> > >> resource governance tools like cgroups. What a dream it >> >> >> > >> >> > >> might be if we had out-of-process compaction or >> >> >> > >> >> > >> out-of-process Merkle tree calculation for repair. >> >> >> > >> >> > >> >> >> >> > >> >> > >> I understand your point that adding another Sidecar just >> >> >> > >> >> > >> for backups can increase operational complexity, but I'd >> >> >> > >> >> > >> argue that cassandra-sidecar is shaping up to offer far >> >> >> > >> >> > >> more than just backups. Today we have Sidecar acting as >> >> >> > >> >> > >> the entrypoint for bulk analytics, CEP-44 bringing CDC >> >> >> > >> >> > >> publishing to Kafka, CEP-40 adding zero-copy online data >> >> >> > >> >> > >> migration between nodes and with CEP-53 and CEP-62 we are >> >> >> > >> >> > >> getting rolling restarts, configuration management and a >> >> >> > >> >> > >> framework for executing other distributed jobs across a >> >> >> > >> >> > >> Cassandra cluster. Framed this way it's becoming a >> >> >> > >> >> > >> one-stop-shop for these types of operations. I also think >> >> >> > >> >> > >> that building on top of the operational job framework >> >> >> > >> >> > >> being added with CEP-53 could go a long way towards >> >> >> > >> >> > >> making the implementation easier. >> >> >> > >> >> > >> >> >> >> > >> >> > >> >> >> >> > >> >> > >> On Tue, Oct 6, 2026 at 8:56 AM Victor Araujo >> >> >> > >> >> > >> <[email protected]> wrote: >> >> >> > >> >> > >>> >> >> >> > >> >> > >>> Hi Bernardo and Stefan, >> >> >> > >> >> > >>> >> >> >> > >> >> > >>> Thanks for the feedback. I agree that there are several >> >> >> > >> >> > >>> benefits to having backup and restore in Sidecar, >> >> >> > >> >> > >>> especially regarding isolation, dependencies, and being >> >> >> > >> >> > >>> able to evolve the feature independently from Cassandra >> >> >> > >> >> > >>> itself. I would be happy to explore and contribute to >> >> >> > >> >> > >>> that direction, potentially in my free time, and I may >> >> >> > >> >> > >>> also be able to bring this discussion to my company >> >> >> > >> >> > >>> later. >> >> >> > >> >> > >>> >> >> >> > >> >> > >>> My main concern is more about the entry point for users. >> >> >> > >> >> > >>> In my current environment, we are actually trying to >> >> >> > >> >> > >>> reduce our use of Sidecars and other additional >> >> >> > >> >> > >>> processes to reduce the overall footprint. I also think >> >> >> > >> >> > >>> there may be users who would prefer having a basic >> >> >> > >> >> > >>> built-in solution in Cassandra rather than having to >> >> >> > >> >> > >>> deploy another component just to get backup and restore. >> >> >> > >> >> > >>> >> >> >> > >> >> > >>> Maybe there is a middle ground here. We could have an >> >> >> > >> >> > >>> opt-in, generic backup/restore capability in Cassandra >> >> >> > >> >> > >>> core, similar in spirit to the unified repair solution, >> >> >> > >> >> > >>> without bringing any third-party cloud dependencies into >> >> >> > >> >> > >>> Cassandra. For example, the core could provide the >> >> >> > >> >> > >>> backup/restore abstractions and lifecycle, with a >> >> >> > >> >> > >>> pluggable storage implementation provided by the user or >> >> >> > >> >> > >>> another component. This could serve as a simple >> >> >> > >> >> > >>> entry-level solution, while a more complete >> >> >> > >> >> > >>> implementation with cloud-specific integrations could >> >> >> > >> >> > >>> live in Sidecar. >> >> >> > >> >> > >>> >> >> >> > >> >> > >>> I don't know yet whether this split makes sense from an >> >> >> > >> >> > >>> architectural perspective, but I think it could be worth >> >> >> > >> >> > >>> exploring. I would be interested in hearing what others >> >> >> > >> >> > >>> think about this approach. >> >> >> > >> >> > >>> >> >> >> > >> >> > >>> Thanks, >> >> >> > >> >> > >>> Victor >> >> >> > >> >> > >>> >> >> >> > >> >> > >>> >> >> >> > >> >> > >>> Em ter., 6 de out. de 2026 às 07:49, Štefan Miklošovič >> >> >> > >> >> > >>> <[email protected]> escreveu: >> >> >> > >> >> > >>>> >> >> >> > >> >> > >>>> I also think that we should investigate how to >> >> >> > >> >> > >>>> integrate backup / >> >> >> > >> >> > >>>> restores in Sidecar rather than doing it in Cassandra. >> >> >> > >> >> > >>>> Dependency-wise, I think Sidecar already depends on s3 >> >> >> > >> >> > >>>> so that is for >> >> >> > >> >> > >>>> free already. If I am not mistaken, Sidecar already >> >> >> > >> >> > >>>> exposes a set of >> >> >> > >> >> > >>>> endpoints related to the restoration of data from S3 >> >> >> > >> >> > >>>> when running some >> >> >> > >> >> > >>>> Spark jobs, right? So this would be just a natural >> >> >> > >> >> > >>>> extension of that - >> >> >> > >> >> > >>>> a set of endpoints backuping to and restoring from an >> >> >> > >> >> > >>>> S3 bucket. >> >> >> > >> >> > >>>> >> >> >> > >> >> > >>>> However, I think that we should make it generic enough >> >> >> > >> >> > >>>> in such a way >> >> >> > >> >> > >>>> that if somebody wants to do this against Azure or GCP >> >> >> > >> >> > >>>> they could. We >> >> >> > >> >> > >>>> should not "wire" S3 only. I am not sure how to do this >> >> >> > >> >> > >>>> though. These >> >> >> > >> >> > >>>> deps are quite "heavy", depending on their own >> >> >> > >> >> > >>>> dependency chain ... I >> >> >> > >> >> > >>>> do not have good experience with mixing various cloud >> >> >> > >> >> > >>>> libraries. >> >> >> > >> >> > >>>> >> >> >> > >> >> > >>>> What we do in our backup / restore solution is that we >> >> >> > >> >> > >>>> modularize it >> >> >> > >> >> > >>>> in such a way that a user can choose what cloud they go >> >> >> > >> >> > >>>> to deal with >> >> >> > >> >> > >>>> and they have only that cloud-specific libraries on the >> >> >> > >> >> > >>>> class path. >> >> >> > >> >> > >>>> >> >> >> > >> >> > >>>> On Tue, Oct 6, 2026 at 12:14 PM Bernardo Botella >> >> >> > >> >> > >>>> <[email protected]> wrote: >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > Hi Victor! >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > Thanks for bringing up this discussion 😊 >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > I completely agree that a first citizen backup and >> >> >> > >> >> > >>>> > restore mechanism in Cassandra would be great. >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > I’d like to explore a little bit more the solution >> >> >> > >> >> > >>>> > involving Sidecar. While it is true that it would >> >> >> > >> >> > >>>> > “force” you to have yet another component running, it >> >> >> > >> >> > >>>> > definitely comes with a lot of benefits besides the >> >> >> > >> >> > >>>> > current other features it provides (control plane, >> >> >> > >> >> > >>>> > analytics library, CDC over Kafka, to name some). >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > It allows for a more modular approach to a Cassandra >> >> >> > >> >> > >>>> > deployment. In my humble opinion, having smaller, >> >> >> > >> >> > >>>> > more specialized processes is the direction we should >> >> >> > >> >> > >>>> > be following, as opposed to keep making the core of >> >> >> > >> >> > >>>> > the project bigger and more complex. >> >> >> > >> >> > >>>> > It allows for a faster iteration in terms of >> >> >> > >> >> > >>>> > software. When it comes to version upgrades, it is >> >> >> > >> >> > >>>> > definitely way easier to update just the sidecar than >> >> >> > >> >> > >>>> > the entire Cassandra core. It is also less risky. >> >> >> > >> >> > >>>> > Blast radius is also smaller. It is better to have >> >> >> > >> >> > >>>> > the Sidecar affected if a backup process >> >> >> > >> >> > >>>> > stalls/breaks/misbehaves than the whole Cassandra >> >> >> > >> >> > >>>> > core. >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > That’s my two cents! >> >> >> > >> >> > >>>> > Bernardo >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > From: Victor Araujo <[email protected]> >> >> >> > >> >> > >>>> > Date: Monday, 5 October 2026 at 13:34 >> >> >> > >> >> > >>>> > To: [email protected] >> >> >> > >> >> > >>>> > <[email protected]> >> >> >> > >> >> > >>>> > Subject: [DISCUSS] Backup and restore in Cassandra >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > Hi everyone, >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > I would like to discuss the possibility of having a >> >> >> > >> >> > >>>> > built-in backup and restore mechanism in Cassandra, >> >> >> > >> >> > >>>> > including incremental backups and support for >> >> >> > >> >> > >>>> > external object storage such as S3. The main goal >> >> >> > >> >> > >>>> > would be to provide a supported solution while >> >> >> > >> >> > >>>> > keeping the implementation provider-agnostic and >> >> >> > >> >> > >>>> > avoiding unnecessary operational overhead. >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > I initially discussed this on Slack and was pointed >> >> >> > >> >> > >>>> > to CEP-36 and the previous discussions around >> >> >> > >> >> > >>>> > external storage and snapshots. We also discussed >> >> >> > >> >> > >>>> > whether S3 would be too provider-specific, and >> >> >> > >> >> > >>>> > whether Cassandra Sidecar could be part of the >> >> >> > >> >> > >>>> > solution. My concern with Sidecar is that introducing >> >> >> > >> >> > >>>> > another component would increase the operational >> >> >> > >> >> > >>>> > footprint, which goes against one of the goals of >> >> >> > >> >> > >>>> > this work. I also have a small PoC on top of trunk >> >> >> > >> >> > >>>> > exploring some of these ideas. >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > I'd be interested in hearing other opinions and any >> >> >> > >> >> > >>>> > previous work I should look at. If there is interest >> >> >> > >> >> > >>>> > in pursuing this, would a draft CEP be a useful next >> >> >> > >> >> > >>>> > step so we can start working on the design in >> >> >> > >> >> > >>>> > parallel? >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > Link of the first slack discussion about this topic: >> >> >> > >> >> > >>>> > https://the-asf.slack.com/archives/CK23JSY2K/p1789021792614639 >> >> >> > >> >> > >>>> > >> >> >> > >> >> > >>>> > Thanks, >> >> >> > >> >> > >>>> > Victor Cavichioli
