Just to add...my informal tests show that batching has waaaaay more effect than solrj vs json.
I haven't look at CUSC in a while, last time I looked it was impossible to do anything smart about error handling, so check that out before you get too deeply into it. We use a strategy of sending a batch of json documents, and if it returns an error sending each record one at a time until we find the bad one and can log something useful. On Mon, Oct 5, 2015 at 12:07 PM, Alessandro Benedetti < benedetti.ale...@gmail.com> wrote: > Thanks Erick, > you confirmed my impressions! > Thank you very much for the insights, an other opinion is welcome :) > > Cheers > > 2015-10-05 14:55 GMT+01:00 Erick Erickson <erickerick...@gmail.com>: > > > SolrJ tends to be faster for several reasons, not the least of which > > is that it sends packets to Solr in a more efficient binary format. > > > > Batching is critical. I did some rough tests using SolrJ and sending > > docs one at a time gave a throughput of < 400 docs/second. > > Sending 10 gave 2,300 or so. Sending 100 at a time gave > > over 5,300 docs/second. Curiously, 1,000 at a time gave only > > marginal improvement over 100. This was with a single thread. > > YMMV of course. > > > > CloudSolrClient is definitely the better way to go with SolrCloud, > > it routes the docs to the correct leader instead of having the > > node you send the docs to do the routing. > > > > Best, > > Erick > > > > On Mon, Oct 5, 2015 at 4:57 AM, Alessandro Benedetti > > <abenede...@apache.org> wrote: > > > I was doing some studies and analysis, just wondering in your opinion > > which > > > one is the best approach to use to index in Solr to reach the best > > > throughput possible. > > > I know that a lot of factor are affecting Indexing time, so let's only > > > focus in the feeding approach. > > > Let's isolate different scenarios : > > > > > > *Single Solr Infrastructure* > > > > > > 1) Xml/Json batch request to /update IndexHandler (xml/json) > > > > > > 2) SolrJ ConcurrentUpdateSolrClient ( javabin) > > > I was thinking this to be the fastest approach for a multi threaded > > > indexing application. > > > Posting batch of docs if possible per request. > > > > > > *Solr Cloud* > > > > > > 1) Xml/Json batch request to /update IndexHandler(xml/json) > > > > > > 2) SolrJ ConcurrentUpdateSolrClient ( javabin) > > > > > > 3) CloudSolrClient ( javabin) > > > it seems the best approach accordingly to this improvements [1] > > > > > > What are your opinions ? > > > > > > A bonus observation should be for using some Map/Reduce big data > indexer, > > > but let's assume we don't have a big cluster of cpus, but the average > > > Indexer server. > > > > > > > > > [1] > > > > > > https://lucidworks.com/blog/indexing-performance-solr-5-2-now-twice-fast/ > > > > > > > > > Cheers > > > > > > > > > -- > > > -------------------------- > > > > > > Benedetti Alessandro > > > Visiting card : http://about.me/alessandro_benedetti > > > > > > "Tyger, tyger burning bright > > > In the forests of the night, > > > What immortal hand or eye > > > Could frame thy fearful symmetry?" > > > > > > William Blake - Songs of Experience -1794 England > > > > > > -- > -------------------------- > > Benedetti Alessandro > Visiting card - http://about.me/alessandro_benedetti > Blog - http://alexbenedetti.blogspot.co.uk > > "Tyger, tyger burning bright > In the forests of the night, > What immortal hand or eye > Could frame thy fearful symmetry?" > > William Blake - Songs of Experience -1794 England > -- Bill Dueber Library Systems Programmer University of Michigan Library
