How can you store Solr shards in hadoop? Is each data node running a Solr server? If so - is the reducer doing a trick to write to a local fs?
Sent from my iPad On Oct 11, 2012, at 12:04 AM, "M. C. Srivas" <[email protected]> wrote: > Interestingly, a few MapR customers have gone the other way, deliberately > having the indexer put the Solr shards directly into MapR and letting it > distribute it. Has made index-management a cinch. > > Otherwise they do run into what Tim alludes to. > > On Wed, Oct 10, 2012 at 7:27 PM, Tim Williams <[email protected]> wrote: > On Wed, Oct 10, 2012 at 10:15 PM, Lance Norskog <[email protected]> wrote: > > In the LucidWorks Big Data product, we handle this with a reducer that > > sends documents to a SolrCloud cluster. This way the index files are not > > managed by Hadoop. > > Hi Lance, > I'm curious if you've gotten that to work with a decent-sized (e.g. > > 250 node) cluster? Even a trivial cluster seems to crush SolrCloud > from a few months ago at least... > > Thanks, > --tim > > > ----- Original Message ----- > > | From: "Ted Dunning" <[email protected]> > > | To: [email protected] > > | Cc: "Hadoop User" <[email protected]> > > | Sent: Wednesday, October 10, 2012 7:58:57 AM > > | Subject: Re: Hadoop/Lucene + Solr architecture suggestions? > > | > > | I prefer to create indexes in the reducer personally. > > | > > | Also you can avoid the copies if you use an advanced hadoop-derived > > | distro. Email me off list for details. > > | > > | Sent from my iPhone > > | > > | On Oct 9, 2012, at 7:47 PM, Mark Kerzner <[email protected]> > > | wrote: > > | > > | > Hi, > > | > > > | > if I create a Lucene index in each mapper, locally, then copy them > > | > to under /jobid/mapid1, /jodid/mapid2, and then in the reducers > > | > copy them to some Solr machine (perhaps even merging), does such > > | > architecture makes sense, to create a searchable index with > > | > Hadoop? > > | > > > | > Are there links for similar architectures and questions? > > | > > > | > Thank you. Sincerely, > > | > Mark > > | >
