On Fri, 25 Mar 2011 14:26 +0200, "Dmitry Kan" <dmitry....@gmail.com> wrote: > Hi, Upayavira > > Probably I'm confusing the terms here. When I say "distributed faceting" > I'm > more into SOLR on the cloud (e.g. HDFS + MR + cloud of commodity > machines) > rather than into traditional multicore/sharded SOLR on a single or > multiple > servers with non-distributed file systems (is that what you mean when you > refer to "distribution of facet requests across hosts"?)
I mean the latter I am afraid. I'm very interested in how the former might be implemented, but as far as I understand it, Zookeeper does not take you all the way there. It co-ordinates nodes (e.g. telling a slave where its master is), but if you have to distribute an index over multiple hosts, it will be sharded between multiple solr hosts, with each of those hosts having a local index. You are presumably talking about a scenario where you effectively have one index, spanning multiple hosts (we have code to distribute queries across multiple segments, why can't we do it across multiple hosts?). I've heard of work to do this with Infinispan underneath, but not within the core Lucene/Solr. Upayavira > On Fri, Mar 25, 2011 at 1:57 PM, Upayavira <u...@odoko.co.uk> wrote: > > > > > > > On Fri, 25 Mar 2011 13:44 +0200, "Dmitry Kan" <dmitry....@gmail.com> > > wrote: > > > Hi Yonik, > > > > > > Oh, this is great. Is distributed faceting available in the trunk? What > > > is > > > the basic server setup needed for trying this out, is it cloud with HDFS > > > and > > > SOLR with zookepers? > > > Any chance to see the related documentation? :) > > > > Distributed faceting has been available for a long time, and is > > available in the 1.4.1 release. > > > > The distribution of facet requests across hosts happens in the > > background. There's no real difference (in query syntax) between a > > standard facet query and a distributed one. > > > > i.e. you don't need SolrCloud nor Zookeeper for it. (they may provide > > other benefits, but you don't need them for distributed faceting). > > > > Upayavira > > > > > On Fri, Mar 25, 2011 at 1:35 PM, Yonik Seeley > > > <yo...@lucidimagination.com>wrote: > > > > > > > On Tue, Mar 22, 2011 at 7:51 AM, Dmitry Kan <dmitry....@gmail.com> > > wrote: > > > > > Basically, of high interest is checking out the Map-Reduce for > > > > distributed > > > > > faceting, is it even possible with the trunk? > > > > > > > > Solr already has distributed faceting, and it's much more performant > > > > than a map-reduce implementation would be. > > > > > > > > I've also seen a product use the term "map reduce" incorrectly... as > > in, > > > > we "map" the request to each shard, and then "reduce" the results to a > > > > single list (of course, that's not actually map-reduce at all ;-) > > > > > > > > > > > :) this sounds pretty strange to me as well. It was only my guess, that > > > if > > > you have MR as computational model and a cloud beneath it, you could > > > naturally map facet fields to their counts inside single documents (no > > > matter, where they are, be it shards or "single" index) and pass them > > > onto > > > reducers. > > > > > > > > > > -Yonik > > > > http://www.lucenerevolution.org -- Lucene/Solr User Conference, May > > > > 25-26, San Francisco > > > > > > > > > > > > > > > > -- > > > Regards, > > > > > > Dmitry Kan > > > > > --- > > Enterprise Search Consultant at Sourcesense UK, > > Making Sense of Open Source > > > > > > > -- > Regards, > > Dmitry Kan > --- Enterprise Search Consultant at Sourcesense UK, Making Sense of Open Source
