On 4/28/23 11:33, Ilan Ginzburg wrote:
Salesforce has been working for a while on separating compute from storage in SolrCloud, see presentation at Activate 2019 SolrCloud in Public Cloud: Scaling Compute Independently from Storage <https://youtu.be/6fE5KvOfb6A>. In a nutshell, the idea is that all SolrCloud nodes are stateless, have a local disk cache of the cores they're hosting but no persistent volumes (no persistent indexes nor transaction logs), and shard level persistence is done on S3.
This is a very intriguing idea! I think it would be particularly useful for containerized setups that can add or remove nodes to meet changing demands.
My primary concern when I first looked at this was that with network-based storage there would be little opportunity for caching, and caching is SUPER critical for Solr performance. Then when I began writing this reply, I saw above that you're talking about a local disk cache... so maybe that is not something to worry about.
Bandwidth and latency limitations to/from the shared storage are another concern, especially with big indexes that have segments up to 5GB. Increasing the merge tier sizes and reducing the max segment size is probably a very good idea.
Another challenge: Ensuring that switching leaders happens reasonably quickly while making sure that there cannot be multiple replicas thinking they are leader at the same time. Making the leader fencing bulletproof is a critical piece of this. I suspect that the existing leader fencing could use some work, affecting SolrCloud in general.
I don't want to get too deep in technical weeds, mostly because I do not understand all the details ... but I am curious about something that might affect this: Are ephemeral znodes created by one Solr node visible to other Solr nodes? If they are, then I think ZK would provide all the fencing needed, and could also keep track of the segments that exist in the remote storage so follower replicas can quickly keep up with the leader.
There could also be implementations for more mundane shared storage like SMB or NFS.
Thanks, Shawn --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
