This is similar to a problem I've been having. It's pretty easy to just
limit the user to 200 pages when the results are sorted by relevance,
but if they are sorted alphabetically, then that doesn't work. It would
be nice if there were a "limit=2000" parameter to the solr call that is
applied to the relevance before sorting.
[It also made me jump through hoops when I wrote some unit tests for the
indexing.]
-----Original Message-----
From: Walter Underwood [mailto:wun...@wunderwood.org]
Sent: December-24-09 1:51 PM
To: solr-user@lucene.apache.org
Subject: Re: SOLR Performance Tuning: Pagination
Some bots will do that, too. Maybe badly written ones, but we saw that at
Netflix. It was causing search timeouts just before a peak traffic period,
so we set a page limit in the front end, something like 200 pages.
It makes sense for that to be very slow, because a request for hit
28838540 means that Solr has to calculate the relevance for 28838540 + 10
documents.
Fuad: Why are you benchmarking this? What user is looking at 20M
documents?
wunder
On Dec 24, 2009, at 10:44 AM, Erik Hatcher wrote:
On Dec 24, 2009, at 11:36 AM, Walter Underwood wrote:
When do users do a query like that? --wunder
Well, SolrEntityProcessor "users" do :)
http://issues.apache.org/jira/browse/SOLR-1499
(which by the way I plan on polishing and committing over the holidays)
Erik
On Dec 24, 2009, at 8:09 AM, Fuad Efendi wrote:
I used pagination for a while till found this...
I have filtered query ID:[* TO *] returning 20 millions results (no
faceting), and pagination always seemed to be fast. However, fast only
with
low values for start=12345. Queries like start=28838540 take 40-60
seconds,
and even cause OutOfMemoryException.
I use highlight, faceting on nontokenized "Country" field, standard
handler.
It even seems to be a bug...
Fuad Efendi
+1 416-993-2060
http://www.linkedin.com/in/liferay
Tokenizer Inc.
http://www.tokenizer.ca/
Data Mining, Vertical Search
