Hi, Currently, I index to solrCloud using nutch-1.6 and nutch-2.1, without any additional patches. I run solrindex and point it to one of the solr instances in the cloud.
In doing this it seems to me that the solr instance that the documents are sent gets overloaded. Let say solrindex command sends 1000 documents per sec to two shards. In case without partitoning all 1000 docs is sent to one shard, then partitoned by solr and send to other shards. However, in case if partitioning is used before sending to solr then each shard gets only 500 docs, which avoids overloading of solr servers. Any thoughts about this are welcome. Thanks. Alex. -----Original Message----- From: Tuğcem Oral <[email protected]> To: user <[email protected]> Sent: Mon, Jul 8, 2013 3:16 am Subject: Re: Indexing from nutch 1.6 to solr 4.3.1 cloud OK then. I generated the corresponding patch. If someone also needs it till nutch 1.8 is released, I'd be happy to share. Best, Tugcem. On Mon, Jul 8, 2013 at 12:10 PM, Markus Jelsma <[email protected]>wrote: > First we need to upgrade to Solr >= 4.3 (NUTCH-1486). Then we'll have to > add an option to index via CloudSolrServer (NUTCH-1377) where you input > your Zookeeper ensemble vs. a target host. Then we can do NUTCH-1480 and > write to multiple individual servers and/or multiple cloud clusters. > > The upgrade to 4.3 is almost finished. The rest isn't very hard and will > be available in Nutch 1.8 if i can help it :) > > -----Original message----- > > From:Tuğcem Oral <[email protected]> > > Sent: Monday 8th July 2013 11:06 > > To: [email protected] > > Subject: Re: Indexing from nutch 1.6 to solr 4.3.1 cloud > > > > You're right, but nutch 1.6 comes along with solrj v3.4 which doesn't > > include CloudSolrServer. That's why we wrote such a patch. We already use > > CloudSolrServer for querying solr shards. > > > > BTW: I changed the solrj version of nutch 1.6 to 4.3.1 but they're not > > working well together while indexing > > > > So do you prefer any other solution for partitioning and indexing from > > nutch 1.6 to solr cloud? > > > > Best. > > > -- TO
