The remainder of an arithmetic division http://en.wikipedia.org/wiki/Modulo_operation -----Original message----- From: Dennis Gearon <gear...@sbcglobal.net> Sent: Mon 06-09-2010 22:04 To: solr-user@lucene.apache.org; Subject: Re: SolrCloud distributed indexing (Re: anyone use hadoop+solr?)
What is a 'simple MOD'? Dennis Gearon Signature Warning ---------------- EARTH has a Right To Life, otherwise we all die. Read 'Hot, Flat, and Crowded' Laugh at http://www.yert.com/film.php --- On Mon, 9/6/10, Andrzej Bialecki <a...@getopt.org> wrote: > From: Andrzej Bialecki <a...@getopt.org> > Subject: Re: SolrCloud distributed indexing (Re: anyone use hadoop+solr?) > To: solr-user@lucene.apache.org > Date: Monday, September 6, 2010, 11:30 AM > On 2010-09-06 16:41, Yonik Seeley > wrote: > > On Mon, Sep 6, 2010 at 10:18 AM, MitchK<mitc...@web.de> > wrote: > > [...consistent hashing...] > >> But it doesn't solve the problem at all, correct > me if I am wrong, but: If > >> you add a new server, let's call him IP3-1, and > IP3-1 is nearer to the > >> current ressource X, than doc x will be indexed at > IP3-1 - even if IP2-1 > >> holds the older version. > >> Am I right? > > > > Right. You still need code to handle migration. > > > > Consistent hashing is a way for everyone to be able to > agree on the > > mapping, and for the mapping to change > incrementally. i.e. you add a > > node and it only changes the docid->node mapping of > a limited percent > > of the mappings, rather than changing the mappings of > potentially > > everything, as a simple MOD would do. > > Another strategy to avoid excessive reindexing is to keep > splitting the largest shards, and then your mapping becomes > a regular MOD plus a list of these additional splits. > Really, there's an infinite number of ways you could > implement this... > > > > > For SolrCloud, I don't think we'll end up using > consistent hashing - > > we don't need it (although some of the concepts may > still be useful). > > I imagine there could be situations where a simple MOD > won't do ;) so I think it would be good to hide this > strategy behind an interface/abstract class. It costs > nothing, and gives you flexibility in how you implement this > mapping. > > -- Best regards, > Andrzej Bialecki <>< > ___. ___ ___ ___ _ > _ __________________________________ > [__ || __|__/|__||\/| Information Retrieval, Semantic > Web > ___|||__|| \| || | Embedded Unix, > System Integration > http://www.sigram.com Contact: info at sigram dot > com > >
