> > The wisdom of putting tens of thousands of cores on a single node aside
It doesn't make sense to put thousands of (mostly unused) cores on a normal SolrCloud node. No matter how fast and big the disks are, these cores consume resources, and if they are not used, it's a waste of resources. If the nodes have a huge amount of RAM, it should be used to do useful work like serving a huge number of queries, not to store data that is not used. Having an accessible and centralized source for the content of these unused cores allows ignoring them completely on all cluster nodes until the moment they're needed. There's then a latency penalty for fetching them over the network, but once they're copied into the node's local disk, they're accessed locally. No penalty for queries, but a penalty for indexing: need to commit locally write over the network to the shared storage - this is the main drawback of the approach for guaranteeing that the shared storage is the source of truth and any or all Solr nodes can disappear at any time with no data loss. Ilan On Sat, Nov 29, 2025 at 9:22 PM Dmitri Maziuk <[email protected]> wrote: > On 11/29/25 13:36, Ilan Ginzburg wrote: > > > > Fat data nodes with fast disks and current SolrCloud code would not > support > > thousands or tens of thousands cores per node (unless they also have very > > fat memory). > > > > The wisdom of putting tens of thousands of cores on a single node aside, > I strongly suspect putting data on ZFS with a good PCIe x.N SSD for > L2ARC will give you better bang for the buck than any storage that has > to fetch stuff over the network, and over the "fast disks" too. And with > its incremental snapshot transfers you could probably keep your data > replicas in sync no worse than most distributed FSes do. Yes, you will > want more RAM. You always want more RAM, what else's new. > > Of course that only works if you're running on bare metal and are in > charge of your purchasing decisions. If you're running on virtual/cloud > infra, you get to live with the latencies your provider provides. > > Dima > >
