GitHub user spetz added a comment to the discussion: Some ideas about cloud storage basd message system based on Apache Iggy
Hey Evan, Thank you for the very detailed input. As of now, we do have the basic support for S3 backups/archiver, and although, we did consider implementing some sort of tiered storage already in the past, actually quite similar to your use case (persisting the data onto the disk to achieve low latency, and then flushing it into the object storage in the cloud), we decided to halt it, until we complete the most important features, such as ongoing zero-copy (de)serialization, io_uring along with the shared-nothing design, and finally, clustering support based on VSR. One of the main reasons that makes Iggy stand out across the other tools within the message streaming infrastructure space, is its performance, low latency, and the overall efficiency. We're already able to achieve the microsecond range for p999+ tail latencies (depending on the use case), and this will only get better with the upcoming improvements. Speaking on behalf of the core team, we do believe, that the need for the extremely fast message streaming infrastructure, where pushing the gigabytes of data per second, within the microsecond tail latency range, will matter a lot in the near future. If we were to focus mostly on the cloud storage at this point (which is actually quite a challenge to implement properly and would require at least the partial redesign of the existing storage layer), I can't think of many reasons why someone would someone use Iggy instead of the existing Kafka API-compatible tools (where some of these solely focus on this particular feature). The tiered/diskless storage will most likely happen at some point, but at least for now, it's not on our roadmap, as it doesn't give us much of an advantage. For example, flushing the data could be even done with the usage of [connectors](https://github.com/apache/iggy/discussions/1670), but then being able to read the data, would be a different story. One would have to redesign the storage to support the remote reads, metadata, efficient batching, etc. GitHub link: https://github.com/apache/iggy/discussions/1707#discussioncomment-12877279 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
