These are all valid points you raised. I think that CEP-36 can be "limited" to just providing a pluggable way for ChannelProxy. Serving from S3 directly or not, there will still need to be a way to get a cloud-specific implementation of ChannelProxy for reads / write of any kind. Later on we can decide the overall architecture of that but I would reserve it for another CEP, that work will most probably be way more involved than "just" a ChannelProxy creation.
If ChannelProxy is pluggable already, we can start to use this for snapshot uploading _if somebody ever wants to go that route_, I would just need to tweak the snapshotting stuff to be ChannelProxy-aware instead of on "File"-s. The default impl would be on FileChannel as we know it. On Thu, Oct 8, 2026 at 5:02 PM Jon Haddad <[email protected]> wrote: > > I didn't realize the CEP hadn't been updated. I think I was mixing up the > convo on the ML and what was actually in the CEP. I had remembered it > behaving much closer to clickhouse, where it keeps the data on disk and > pushes the compacted files to S3. I don't think serving directly off S3 is > going to be much use to anyone. The latency will be awful, and they issue a > backoff warning fairly quickly. In my mind it only makes sense to treat S3 > as the authority and cache SStables to local disk. I think the Astra folks > may have done something along these lines, I'd be willing to bet there's a > ton of knowledge there as they operationalized it successfully. > > Anyways - the reason I asked my question, was to frame it in the context of a > longer term project strategy. Asking more directly - If we're going to be > pushing and pulling from S3 during compaction, and it's a major feature, > would that go in Sidecar? It certainly could, but I'm concerned about having > a critical component of C* living it's own independent lifecycle. > > Example of why I get concerned: > https://the-asf.slack.com/archives/C07V0FK0M2P/p1788964591823929 > > Imo, having 2 projects that are tightly coupled, but developed independently, > is going to be fairly high risk. I really don't like having responsibility > spread this way, and I probably wouldn't want to run it in prod. > > Jon > > > > On Wed, Oct 7, 2026 at 7:15 AM Štefan Miklošovič <[email protected]> > wrote: >> >> I have a draft of CEP-36 here (1). S3 deps + proxy impl is meant to be >> implemented in a different project (2) which once built is added to >> Cassandra via custom ClassLoader so it is all encapsulated and we can >> not have any dependency clashes. >> >> There is the impl of s3 proxy with e2e tests. It shows how Cassandra >> starts to serve reads from s3. When it comes to writes, these SSTables >> will be created locally as they are not and then "offloaded" to S3 for >> reads. Once SSTable is created it is immutable so we can upload it and >> then switch SSTableReader to read via S3 proxy. >> >> At least that is the idea. Heavy demo. Dragons everywhere. AI-driven a >> lot. It more _does not work_ than does :D But it genuinely starts to >> read from S3 via proxy which is quite ... promising, but I think that >> if we want to seriously consider CEP-36 and implement it fully that is >> a serious piece of work for two quarters :D >> >> Give me some time here please, like a week or two, I would like to >> explore this in more detail. Maybe it will be better to modify >> SnapshotManager to work with ChannelProxy instead of File and when >> taking snapshots we would use S3 ChannelProxy plus we would tweak the >> hardlinking bit. >> >> You can of course do this all in Sidecar, I just want to see where >> this gets us. You would have two ways to choose from (and CEP-36 >> building blocks would be in place to extend on further). >> >> (1) https://github.com/apache/cassandra/pull/5238/changes >> (2) https://github.com/smiklosovic/cassandra-s3-channel-proxy >> >> On Wed, Oct 7, 2026 at 3:30 PM Isaac Reath <[email protected]> wrote: >> > >> > In fairness, CEP-36 does explicitly call out that anything that uses a >> > hard link will no longer work once data is written in an alternative >> > location: >> > >> > > Once triggered any data written to alternative locations will not be >> > > included in the snapshotting, incremental backups or other processes >> > > that use hard file links. >> > >> > In a world where we have CEP-36 and an S3-backed implementation of >> > ChannelProxy, I would imagine that a call to the >> > ColumnFamilyStore.takeSnapshot function would either no-op (sticking to >> > what is defined in the CEP), wrap a call to take an on-demand backup via >> > AWS Backup, or do something similar to running aws s3 sync. In this world, >> > if you wanted to take a backup across a whole cluster, you could still use >> > Sidecar to manage the coordination. Or you could just use continuous >> > backups on your S3 buckets and call it a day. Depends on the semantics and >> > use-cases of what we're looking for out of a backup solution. >> > >> > CEP-36 offers details around how to get the ChannelProxy interface in >> > place, but leaves the details of how to implement an object store-backed >> > version of the storage engine as an exercise for the reader. I think if we >> > were to get CEP-36 voted on and implemented, we'd need a follow-up CEP >> > that dives into the S3 implementation, or otherwise we'd need to expand >> > the current scope of CEP-36 to cover it. >> > >> > On Wed, Oct 7, 2026 at 6:31 AM Štefan Miklošovič <[email protected]> >> > wrote: >> >> >> >> BTW I am still not in favor of putting this to Cassandra, just have >> >> snapshots in Sidecar with its own endpoints, it is way easier that >> >> way. I am pretty tired at this point to go through these discussions >> >> all over again yielding no actual resolution of what we are going to >> >> do. "cloud-aware Cassandra" was so long on a roadmap I am losing hope >> >> we will ever see that. The amount of work needed is enormous, we would >> >> need to go case by case and decide what is still to be written on a >> >> local disk and what goes to the cloud. Some functionality is not even >> >> there: >> >> >> >> 1) How do you atomically rename a file? S3 does not have the concept >> >> of rename. There is some "S3 Express One Zone" which seems to support >> >> RenameObject but that seems quite niche >> >> 2) Hard links ... none, we would need to copy >> >> 3) mmap, how is this done in S3? You can not memory map an object >> >> store, no? We would need to do disk_access_mode: standard >> >> 4) Latency ... >> >> >> >> Commit logs, hints, caches ... Accord stuff. I don't know what else, >> >> all the local disk interaction which is illogical to move to cloud >> >> would need to stay local. (anti)-compaction, repair ... all of that >> >> stuff would be cloud-native too? The performance of that would be ... >> >> tragic? >> >> >> >> I know it is something to aspire for and I do not want to be too >> >> negative about this but I think it makes sense to stop for a while and >> >> actually see what this would mean and how much work it is. >> >> >> >> For that reason I would be doing backup / restores in Sidecar and be >> >> done with that. It is a very well understood domain, s3 deps already >> >> there, endpoints endpointing, a snapshot is a JMX call ... why to >> >> complicate this more really. >> >> >> >> On Tue, Oct 6, 2026 at 9:11 PM Štefan Miklošovič <[email protected]> >> >> wrote: >> >> > >> >> > Well ... that's quite a topic. >> >> > >> >> > The OP was asking specifically about backup / restore so I was >> >> > replying in that spirit. >> >> > >> >> > One note for backup / restores, we talk about "snapshots", right? So >> >> > if you back them up, then one would expect that Sidecar would also >> >> > offer a way to list them, then to clear them ... It would not be "just >> >> > about backup and restore". There would be a family of endpoints around >> >> > snapshot management, very likely. That means we kind of duplicate what >> >> > is in Cassandra already, if we do not put this into Cassandra >> >> > directly, then Cassandra would not "see" the snapshots, except those >> >> > taken locally. >> >> > >> >> > If we want to entertain CEP-36 specifically, that would change the >> >> > picture quite significantly. In either case, snapshots would need to >> >> > be copied over (SSTables, that is), because we can not do a hard-link >> >> > from local storage to a S3 bucket. So we would need to copy that, >> >> > right? If that is the case then we would need to probably throttle how >> >> > fast it is uploaded to not interfere with other operations and >> >> > similar. >> >> > >> >> > I was completely over-hauling the snapshot management in >> >> > CASSANDRA-18111, together with Paulo Motta who did a great job >> >> > reviewing it, everything is centralized in SnapshotManager(MBean). The >> >> > only tasks touching the disk are TakeSnapshotTask, ClearSnapshotTask >> >> > and SnapshotLoader which loads them. Their representation / metadata >> >> > are in memory. So when you list it is not going to S3. That isolates >> >> > what would need to be accommodated to talk to a cloud. So if you >> >> > wanted to start to use S3 as a storage, taking CEP-36 into account, we >> >> > would need to: >> >> > >> >> > 1) create "cloude-aware" implementation of >> >> > org.apache.cassandra.io.util.File / java.nio.file.Path >> >> > 2) create "S3 TakeSnapshotTaks" which would need to start to copy >> >> > instead of hardlinking etc. >> >> > 3) create another version of SnapshotLoader or modify it heavily as >> >> > currently it is pretty-much wired to talk to local file system >> >> > 4) Find some way how to "switch" between local / cloud via >> >> > configuration in a complex manner >> >> > >> >> > I still think it holds that we would need to introduce some >> >> > "class-path modularisation" like you would say, via configuration, >> >> > what cloud storage provider you want to use and the related libraries >> >> > would be auto-magically put into the class path. If we had this in >> >> > place then CEP-36 might be potentially closer to the implementation as >> >> > these building blocks would be there already. >> >> > >> >> > On Tue, Oct 6, 2026 at 6:32 PM Jon Haddad <[email protected]> >> >> > wrote: >> >> > > >> >> > > For the folks who want to put this in sidecar, what are your thoughts >> >> > > on how that would work with a potential future improvement using s3 >> >> > > as part of its normal operations? I’m thinking about CEP-36 >> >> > > specifically. >> >> > > >> >> > > Thanks! >> >> > > Jon >> >> > > >> >> > > >> >> > > On Tue, Oct 6, 2026 at 7:37 AM Isaac Reath <[email protected]> >> >> > > wrote: >> >> > >> >> >> > >> > If I am not mistaken, Sidecar already exposes a set of >> >> > >> > endpoints related to the restoration of data from S3 when running >> >> > >> > some >> >> > >> > Spark jobs, right? >> >> > >> >> >> > >> You are correct Stefan. >> >> > >> >> >> > >> I'll echo Stefan's concerns around the weight of adding a cloud >> >> > >> provider dependency into Cassandra. Especially now, with the >> >> > >> increasing volume of CVE discovery across all sorts of open-source >> >> > >> projects, keeping the dependency scope of the core Cassandra process >> >> > >> small is a distinct advantage. As an operator, it's much easier for >> >> > >> me to push out patches across thousands of Sidecars than it is to >> >> > >> push them across thousands of Cassandra nodes. >> >> > >> >> >> > >> I also think that from a resource isolation perspective, running >> >> > >> these types of maintenance operations out-of-process can create much >> >> > >> more predictable performance, especially when paired with OS-level >> >> > >> resource governance tools like cgroups. What a dream it might be if >> >> > >> we had out-of-process compaction or out-of-process Merkle tree >> >> > >> calculation for repair. >> >> > >> >> >> > >> I understand your point that adding another Sidecar just for backups >> >> > >> can increase operational complexity, but I'd argue that >> >> > >> cassandra-sidecar is shaping up to offer far more than just backups. >> >> > >> Today we have Sidecar acting as the entrypoint for bulk analytics, >> >> > >> CEP-44 bringing CDC publishing to Kafka, CEP-40 adding zero-copy >> >> > >> online data migration between nodes and with CEP-53 and CEP-62 we >> >> > >> are getting rolling restarts, configuration management and a >> >> > >> framework for executing other distributed jobs across a Cassandra >> >> > >> cluster. Framed this way it's becoming a one-stop-shop for these >> >> > >> types of operations. I also think that building on top of the >> >> > >> operational job framework being added with CEP-53 could go a long >> >> > >> way towards making the implementation easier. >> >> > >> >> >> > >> >> >> > >> On Tue, Oct 6, 2026 at 8:56 AM Victor Araujo >> >> > >> <[email protected]> wrote: >> >> > >>> >> >> > >>> Hi Bernardo and Stefan, >> >> > >>> >> >> > >>> Thanks for the feedback. I agree that there are several benefits to >> >> > >>> having backup and restore in Sidecar, especially regarding >> >> > >>> isolation, dependencies, and being able to evolve the feature >> >> > >>> independently from Cassandra itself. I would be happy to explore >> >> > >>> and contribute to that direction, potentially in my free time, and >> >> > >>> I may also be able to bring this discussion to my company later. >> >> > >>> >> >> > >>> My main concern is more about the entry point for users. In my >> >> > >>> current environment, we are actually trying to reduce our use of >> >> > >>> Sidecars and other additional processes to reduce the overall >> >> > >>> footprint. I also think there may be users who would prefer having >> >> > >>> a basic built-in solution in Cassandra rather than having to deploy >> >> > >>> another component just to get backup and restore. >> >> > >>> >> >> > >>> Maybe there is a middle ground here. We could have an opt-in, >> >> > >>> generic backup/restore capability in Cassandra core, similar in >> >> > >>> spirit to the unified repair solution, without bringing any >> >> > >>> third-party cloud dependencies into Cassandra. For example, the >> >> > >>> core could provide the backup/restore abstractions and lifecycle, >> >> > >>> with a pluggable storage implementation provided by the user or >> >> > >>> another component. This could serve as a simple entry-level >> >> > >>> solution, while a more complete implementation with cloud-specific >> >> > >>> integrations could live in Sidecar. >> >> > >>> >> >> > >>> I don't know yet whether this split makes sense from an >> >> > >>> architectural perspective, but I think it could be worth exploring. >> >> > >>> I would be interested in hearing what others think about this >> >> > >>> approach. >> >> > >>> >> >> > >>> Thanks, >> >> > >>> Victor >> >> > >>> >> >> > >>> >> >> > >>> Em ter., 6 de out. de 2026 às 07:49, Štefan Miklošovič >> >> > >>> <[email protected]> escreveu: >> >> > >>>> >> >> > >>>> I also think that we should investigate how to integrate backup / >> >> > >>>> restores in Sidecar rather than doing it in Cassandra. >> >> > >>>> Dependency-wise, I think Sidecar already depends on s3 so that is >> >> > >>>> for >> >> > >>>> free already. If I am not mistaken, Sidecar already exposes a set >> >> > >>>> of >> >> > >>>> endpoints related to the restoration of data from S3 when running >> >> > >>>> some >> >> > >>>> Spark jobs, right? So this would be just a natural extension of >> >> > >>>> that - >> >> > >>>> a set of endpoints backuping to and restoring from an S3 bucket. >> >> > >>>> >> >> > >>>> However, I think that we should make it generic enough in such a >> >> > >>>> way >> >> > >>>> that if somebody wants to do this against Azure or GCP they could. >> >> > >>>> We >> >> > >>>> should not "wire" S3 only. I am not sure how to do this though. >> >> > >>>> These >> >> > >>>> deps are quite "heavy", depending on their own dependency chain >> >> > >>>> ... I >> >> > >>>> do not have good experience with mixing various cloud libraries. >> >> > >>>> >> >> > >>>> What we do in our backup / restore solution is that we modularize >> >> > >>>> it >> >> > >>>> in such a way that a user can choose what cloud they go to deal >> >> > >>>> with >> >> > >>>> and they have only that cloud-specific libraries on the class path. >> >> > >>>> >> >> > >>>> On Tue, Oct 6, 2026 at 12:14 PM Bernardo Botella >> >> > >>>> <[email protected]> wrote: >> >> > >>>> > >> >> > >>>> > Hi Victor! >> >> > >>>> > >> >> > >>>> > Thanks for bringing up this discussion 😊 >> >> > >>>> > >> >> > >>>> > I completely agree that a first citizen backup and restore >> >> > >>>> > mechanism in Cassandra would be great. >> >> > >>>> > >> >> > >>>> > I’d like to explore a little bit more the solution involving >> >> > >>>> > Sidecar. While it is true that it would “force” you to have yet >> >> > >>>> > another component running, it definitely comes with a lot of >> >> > >>>> > benefits besides the current other features it provides (control >> >> > >>>> > plane, analytics library, CDC over Kafka, to name some). >> >> > >>>> > >> >> > >>>> > It allows for a more modular approach to a Cassandra deployment. >> >> > >>>> > In my humble opinion, having smaller, more specialized processes >> >> > >>>> > is the direction we should be following, as opposed to keep >> >> > >>>> > making the core of the project bigger and more complex. >> >> > >>>> > It allows for a faster iteration in terms of software. When it >> >> > >>>> > comes to version upgrades, it is definitely way easier to update >> >> > >>>> > just the sidecar than the entire Cassandra core. It is also less >> >> > >>>> > risky. >> >> > >>>> > Blast radius is also smaller. It is better to have the Sidecar >> >> > >>>> > affected if a backup process stalls/breaks/misbehaves than the >> >> > >>>> > whole Cassandra core. >> >> > >>>> > >> >> > >>>> > >> >> > >>>> > That’s my two cents! >> >> > >>>> > Bernardo >> >> > >>>> > >> >> > >>>> > From: Victor Araujo <[email protected]> >> >> > >>>> > Date: Monday, 5 October 2026 at 13:34 >> >> > >>>> > To: [email protected] <[email protected]> >> >> > >>>> > Subject: [DISCUSS] Backup and restore in Cassandra >> >> > >>>> > >> >> > >>>> > Hi everyone, >> >> > >>>> > >> >> > >>>> > I would like to discuss the possibility of having a built-in >> >> > >>>> > backup and restore mechanism in Cassandra, including incremental >> >> > >>>> > backups and support for external object storage such as S3. The >> >> > >>>> > main goal would be to provide a supported solution while keeping >> >> > >>>> > the implementation provider-agnostic and avoiding unnecessary >> >> > >>>> > operational overhead. >> >> > >>>> > >> >> > >>>> > I initially discussed this on Slack and was pointed to CEP-36 >> >> > >>>> > and the previous discussions around external storage and >> >> > >>>> > snapshots. We also discussed whether S3 would be too >> >> > >>>> > provider-specific, and whether Cassandra Sidecar could be part >> >> > >>>> > of the solution. My concern with Sidecar is that introducing >> >> > >>>> > another component would increase the operational footprint, >> >> > >>>> > which goes against one of the goals of this work. I also have a >> >> > >>>> > small PoC on top of trunk exploring some of these ideas. >> >> > >>>> > >> >> > >>>> > I'd be interested in hearing other opinions and any previous >> >> > >>>> > work I should look at. If there is interest in pursuing this, >> >> > >>>> > would a draft CEP be a useful next step so we can start working >> >> > >>>> > on the design in parallel? >> >> > >>>> > >> >> > >>>> > Link of the first slack discussion about this topic: >> >> > >>>> > https://the-asf.slack.com/archives/CK23JSY2K/p1789021792614639 >> >> > >>>> > >> >> > >>>> > Thanks, >> >> > >>>> > Victor Cavichioli
