Hi Robert, You could look at pageDoc & pageScore to improve things for deep paging ( http://wiki.apache.org/solr/CommonQueryParameters#pageDoc_and_pageScore).
Regards, Aloke On Sat, Dec 8, 2012 at 8:08 AM, Upayavira <u...@odoko.co.uk> wrote: > Yes, expected. > > When it does a search for the first, say, 10 results, it must scan > through all docs, recording just the highest ten scoring ones. > > To find documents 1000 to 1010, it must scan through all docs, recording > the best scoring 1010 documents, and then discard the first 1000. This > is much more expensive. Try it on google, they won't let you go beyond > around 900 pages or such (or is it 900 results?) > > Upayavira > > On Sat, Dec 8, 2012, at 01:10 AM, Petersen, Robert wrote: > > Hi guys, > > > > > > Sometimes we get a bot crawling our search function on our retail web > > site. The ebay crawler loves to do this (Request.UserAgent: > > Terapeakbot). They just do a star search and then iterate through page > > after page. I've noticed that when they get to higher page numbers like > > page 9000, the searches are taking more than 20 seconds. Is this > > expected behavior? We're requesting standard facets with the search as > > well as incorporating boosting by function query. Our index is almost 15 > > million docs now and we're on Solr 3.6.1, this isn't causing any errors > > to occur at the solr layer but our web layer times out the search after > > 20 seconds and logs the exception. > > > > > > > > Thanks > > > > Robi > > >
