And if you have control over machine placement, split them across racks so that a power outage on one rack does not take out your search cluster.
François On Apr 5, 2011, at 3:19 AM, Ephraim Ofir wrote: > I'm not sure about the scale you're aiming for, but you probably want to > do both sharding and replication. There's no central server which would > be the bottleneck. The guidelines should probably be something like: > 1. Split your index to enough shards so it can keep up with the update > rate. > 2. Have enough replicates of each shard master to keep up with the rate > of queries. > 3. Have enough aggregators in front of the shard replicates so the > aggregation doesn't become a bottleneck. > 4. Make sure you have good load balancing across your system. > > Attached is a diagram of the setup we have. You might want to look into > SolrCloud as well. > > Ephraim Ofir > > > -----Original Message----- > From: Jens Mueller [mailto:supidupi...@googlemail.com] > Sent: Tuesday, April 05, 2011 4:25 AM > To: solr-user@lucene.apache.org > Subject: Very very large scale Solr Deployment = how to do (Expert > Question)? > > Hello Experts, > > > > I am a Solr newbie but read quite a lot of docs. I still do not > understand > what would be the best way to setup very large scale deployments: > > > > Goal (threoretical): > > A.) Index-Size: 1 Petabyte (1 Document is about 5 KB in Size) > > B) Queries: 100000 Queries/ per Second > > C) Updates: 100000 Updates / per Second > > > > > Solr offers: > > 1.) Replication => Scales Well for B) BUT A) and C) are not > satisfied > > > 2.) Sharding => Scales well for A) BUT B) and C) are not satisfied > (=> As > I understand the Sharding approach all goes through a central server, > that > dispatches the updates and assembles the quries retrieved from the > different > shards. But this central server has also some capacity limits...) > > > > > What is the right approach to handle such large deployments? I would be > thankfull for just a rough sketch of the concepts so I can > experiment/search > further... > > > Maybe I am missing something very trivial as I think some of the "Solr > Users/Use Cases" on the homepage are that kind of large deployments. How > are > they implemented? > > > > Thanky very much!!! > > Jens
