For what it's worth, as it might come to somebody I am rejecting this altogether (which is not the case, all I am trying to say is that we should just think about it more) - it would be cool to know more about the experience of others when it comes to this, maybe somebody already tried to mount and it did not work as expected?
On the other hand, there is this "snapshots outside data dir" effort I am doing and if we did it with this, then I can imagine that we could say "and if you deal with snapshots, use this proxy instead" which would transparently upload it to s3. Then we would not need to do anything at all, code-wise. We would not need to store snapshots "outside of data dir" just to be able to place it on a directory which is mounted as an s3 bucket. I don't know if it is possible to do it like that. Worth to explore I guess. I like mounted dirs for its simplicity and I guess that for copying files it might be just enough. Plus we would not need to add all s3 jars on CP either etc ... On Tue, Mar 4, 2025 at 2:46 PM Štefan Miklošovič <smikloso...@apache.org> wrote: > I don't say that using remote object storage is useless. > > I am just saying that I don't see the difference. I have not measured that > but I can imagine that s3 mounted would use, under the hood, the same calls > to s3 api. How else would it be done? You need to talk to remote s3 storage > eventually anyway. So why does it matter if we call s3 api from Java or by > other means from some "s3 driver"? It is eventually using same thing, no? > > On Tue, Mar 4, 2025 at 12:47 PM Jeff Jirsa <jji...@gmail.com> wrote: > >> Mounting an s3 bucket as a directory is an easy but poor implementation >> of object backed storage for databases >> >> Object storage is durable (most data loss is due to bugs not concurrent >> hardware failures), cheap (can 5-10x cheaper) and ubiquitous. A huge >> number of modern systems are object-storage-only because the approximately >> infinite scale / cost / throughput tradeoffs often make up for the latency. >> >> Outright dismissing object storage for Cassandra is short sighted - it >> needs to be done in a way that makes sense, not just blindly copying over >> the block access patterns to object. >> >> >> On Mar 4, 2025, at 11:19 AM, Štefan Miklošovič <smikloso...@apache.org> >> wrote: >> >> >> I do not think we need this CEP, honestly. I don't want to diss this >> unnecessarily but if you mount a remote storage locally (e.g. mounting s3 >> bucket as if it was any other directory on node's machine), then what is >> this CEP good for? >> >> Not talking about the necessity to put all dependencies to be able to >> talk to respective remote storage to Cassandra's class path, introducing >> potential problems with dependencies and their possible incompatibilities / >> different versions etc ... >> >> On Thu, Feb 27, 2025 at 6:21 AM C. Scott Andreas <sc...@paradoxica.net> >> wrote: >> >>> I’d love to see this implemented — where “this” is a proxy for some >>> notion of support for remote object storage, perhaps usable by compaction >>> strategies like TWCS to migrate data older than a threshold from a local >>> filesystem to remote object. >>> >>> It’s not an area where I can currently dedicate engineering effort. But >>> if others are interested in contributing a feature like this, I’d see it as >>> valuable for the project and would be happy to collaborate on >>> design/architecture/goals. >>> >>> – Scott >>> >>> On Feb 26, 2025, at 6:56 AM, guo Maxwell <cclive1...@gmail.com> wrote: >>> >>> >>> Is anyone else interested in continuing to discuss this topic? >>> >>> guo Maxwell <cclive1...@gmail.com> 于2024年9月20日周五 09:44写道： >>> >>>> I discussed this offline with Claude, he is no longer working on this. >>>> >>>> It's a pity. I think this is a very valuable thing. Commitlog's >>>> archiving and restore may be able to use the relevant code if it is >>>> completed. >>>> >>>> Patrick McFadin <pmcfa...@gmail.com>于2024年9月20日 周五上午2:01写道： >>>> >>>>> Thanks for reviving this one! >>>>> >>>>> On Wed, Sep 18, 2024 at 12:06 AM guo Maxwell <cclive1...@gmail.com> >>>>> wrote: >>>>> >>>>>> Is there any update on this topic? It seems that things can make a >>>>>> big progress if Jake Luciani can find someone who can make the >>>>>> FileSystemProvider code accessible. >>>>>> >>>>>> Jon Haddad <j...@jonhaddad.com> 于2023年12月16日周六 05:29写道： >>>>>> >>>>>>> At a high level I really like the idea of being able to better >>>>>>> leverage cheaper storage especially object stores like S3. >>>>>>> >>>>>>> One important thing though - I feel pretty strongly that there's a >>>>>>> big, deal breaking downside. Backups, disk failure policies, snapshots >>>>>>> and possibly repairs would get more complicated which haven't been >>>>>>> particularly great in the past, and of course there's the issue of >>>>>>> failure >>>>>>> recovery being only partially possible if you're looking at a durable >>>>>>> block >>>>>>> store paired with an ephemeral one with some of your data not >>>>>>> replicated to >>>>>>> the cold side. That introduces a failure case that's unacceptable for >>>>>>> most >>>>>>> teams, which results in needing to implement potentially 2 different >>>>>>> backup >>>>>>> solutions. This is operationally complex with a lot of surface area for >>>>>>> headaches. I think a lot of teams would probably have an issue with the >>>>>>> big question mark around durability and I probably would avoid it >>>>>>> myself. >>>>>>> >>>>>>> On the other hand, I'm +1 if we approach it something slightly >>>>>>> differently - where _all_ the data is located on the cold storage, with >>>>>>> the >>>>>>> local hot storage used as a cache. This means we can use the cold >>>>>>> directories for the complete dataset, simplifying backups and node >>>>>>> replacements. >>>>>>> >>>>>>> For a little background, we had a ticket several years ago where I >>>>>>> pointed out it was possible to do this *today* at the operating system >>>>>>> level as long as you're using block devices (vs an object store) and LVM >>>>>>> [1]. For example, this works well with GP3 EBS w/ low IOPS >>>>>>> provisioning + >>>>>>> local NVMe to get a nice balance of great read performance without going >>>>>>> nuts on the cost for IOPS. I also wrote about this in a little more >>>>>>> detail >>>>>>> in my blog [2]. There's also the new mount point tech in AWS which >>>>>>> pretty >>>>>>> much does exactly what I've suggested above [3] that's probably worth >>>>>>> evaluating just to get a feel for it. >>>>>>> >>>>>>> I'm not insisting we require LVM or the AWS S3 fs, since that would >>>>>>> rule out other cloud providers, but I am pretty confident that the >>>>>>> entire >>>>>>> dataset should reside in the "cold" side of things for the practical and >>>>>>> technical reasons I listed above. I don't think it massively changes >>>>>>> the >>>>>>> proposal, and should simplify things for everyone. >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> [1] https://rustyrazorblade.com/post/2018/2018-04-24-intro-to-lvm/ >>>>>>> [2] https://issues.apache.org/jira/browse/CASSANDRA-8460 >>>>>>> [3] >>>>>>> https://aws.amazon.com/about-aws/whats-new/2023/03/mountpoint-amazon-s3/ >>>>>>> >>>>>>> >>>>>>> On Thu, Dec 14, 2023 at 1:56 AM Claude Warren <cla...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Is there still interest in this? Can we get some points down on >>>>>>>> electrons so that we all understand the issues? >>>>>>>> >>>>>>>> While it is fairly simple to redirect the read/write to something >>>>>>>> other than the local system for a single node this will not solve the >>>>>>>> problem for tiered storage. >>>>>>>> >>>>>>>> Tiered storage will require that on read/write the primary key be >>>>>>>> assessed and determine if the read/write should be redirected. My >>>>>>>> reasoning for this statement is that in a cluster with a replication >>>>>>>> factor >>>>>>>> greater than 1 the node will store data for the keys that would be >>>>>>>> allocated to it in a cluster with a replication factor = 1, as well as >>>>>>>> some >>>>>>>> keys from nodes earlier in the ring. >>>>>>>> >>>>>>>> Even if we can get the primary keys for all the data we want to >>>>>>>> write to "cold storage" to map to a single node a replication factor > >>>>>>>> 1 >>>>>>>> means that data will also be placed in "normal storage" on subsequent >>>>>>>> nodes. >>>>>>>> >>>>>>>> To overcome this, we have to explore ways to route data to >>>>>>>> different storage based on the keys and that different storage may >>>>>>>> have to >>>>>>>> be available on _all_ the nodes. >>>>>>>> >>>>>>>> Have any of the partial solutions mentioned in this email chain (or >>>>>>>> others) solved this problem? >>>>>>>> >>>>>>>> Claude >>>>>>>> >>>>>>>
