Well, SolrCloud is not yet fully specified for the indexing side - more work remains. But my point is that the architecture for should be ZK based.
I added a new jira issue to flesh out a strategy for SolrCloud controlled distributed indexing in SOLR-2293 Perhaps you should open a JIRA issue for indexer failover as well. The simplest model would be to promote one of the search slaves to master indexer, as each slave will have an (almost up-to-date) copy of the index. The client should then have a means of getting alerted about the failover and from what timestamp he will need to re-feed content (based on slave index date). In my opinion it is extremely hard to try to solve some kind of always-in-sync instant failover, and most will not need it either. -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com On 19. des. 2010, at 19.29, Upayavira wrote: > Jan, > > I'd appreciate a little more explanation here. I've explored SolrCloud > somewhat, but there's some bits of this architecture I don't yet get. > > You say, next time an "indexer slave" pings ZK. What is an "indexer > slave"? Is that the external entity that is doing posting indexing > content? If this app that posts to Solr, you imply it must check with ZK > before it can do an HTTP post to Solr? Also, once you do this leader > election to switch to an alternative master, are you implying that this > new master was once a slave of the original master, and thus has a valid > index? > > Find this interesting, but still not quite sure on how it works exactly. > > Upayavira > > On Fri, 17 Dec 2010 10:09 +0100, "Jan Høydahl / Cominvent" > <jan....@cominvent.com> wrote: >> Hi, >> >> I believe the way to go is through ZooKeeper[1], not property files or >> local hacks. We've already started on this route and it makes sense to >> let ZK do what it is designed for, such as leader election. When a node >> starts up, it asks ZK what role it should have and fetches corresponding >> configuration. Then it polls ZK regularly to know if the world has >> changed. So if a master indexer goes down, ZK will register that as a >> state change condition, and next time one of the indexer slaves pings ZK, >> it may be elected as new master, and config in ZK is changed >> correspondingly, causing all adds to flow to the new master... >> >> Then, when the slaves cannot contact their old master, they ask ZK for an >> update, and retrieve a new value for master URL. >> >> Note also that SolrCloud is implementing load-balancing and sharding as >> part of the arcitecture so often we can skip dedicated LBs. >> >> [1] : http://wiki.apache.org/solr/SolrCloud >> >> -- >> Jan Høydahl, search solution architect >> Cominvent AS - www.cominvent.com >> >> On 15. des. 2010, at 18.50, Tommaso Teofili wrote: >> >>> Hi all, >>> me, Upayavira and other guys at Sourcesense have collected some Solr >>> architectural views inside the presentation at [1]. >>> For sure one can set up an architecture for failover and resiliency on the >>> "search face" (search slaves with coordinators and distributed search) but >>> I'd like to ask how would you reach transparent redundancy in Solr on the >>> "index face". >>> On slide 13 we put 2 slave backup masters and so if one of the main masters >>> goes down you can switch slaves' replication on the backup master. >>> First question if how could it be made automatic? >>> In a previous thread [2] I talked about a possible solution writing the >>> master url of slaves in a properties file so when you have to switch you >>> change that url to the backup master and reload the slave's core but that is >>> not automatic :-) Any more advanced ideas? >>> Second question: when main master comes up how can it be automatically >>> considered as the backup master (since hopefully the backup master has >>> received some indexing requests in the meantime)? Also consider that its >>> index should be wiped out and replicated from the new master to ensure index >>> integrity. >>> Looking forward for your feedback, >>> Cheers, >>> Tommaso >>> >>> [1] : http://www.slideshare.net/sourcesense/sharded-solr-setup-with-master >>> [2] : http://markmail.org/thread/vjj5jovbg6evpmpp >>
