Getting a large number of documents by id

2013-07-18 Thread Brian Hurt
I have a situation which is common in our current use case, where I need to get a large number (many hundreds) of documents by id. What I'm doing currently is creating a large query of the form id:12345 OR id:23456 OR ... and sending it off. Unfortunately, this query is taking a long time,

Re: Getting a large number of documents by id

2013-07-18 Thread Alexandre Rafalovitch
You could start from doing id:(12345 23456) to reduce the query length and possibly speed up parsing. You could also move the query from 'q' parameter to 'fq' parameter, since you probably don't care about ranking ('fq' does not rank). If these are unique every time, you could probably look at not

Re: Getting a large number of documents by id

2013-07-18 Thread Jack Krupansky
Solr really isn't designed for that kind of use case. If it happens to work well for your particular situation, great, but don't complain when you are well outside the normal usage for a search engine (10, 20, 50, 100 results paged at a time, with modest sized query strings.) If you must get

Re: Getting a large number of documents by id

2013-07-18 Thread Michael Della Bitta
Brian, Have you tried the realtime get handler? It supports multiple documents. http://wiki.apache.org/solr/RealTimeGet Michael Della Bitta Applications Developer o: +1 646 532 3062 | c: +1 917 477 7906 appinions inc. “The Science of Influence Marketing” 18 East 41st Street New York, NY

Re: Getting a large number of documents by id

2013-07-18 Thread Roman Chyla
Look at speed of reading the data - likely, it takes long time to assemble a big response, especially if there are many long fields - you may want to try SSD disks, if you have that option. Also, to gain better understanding: Start your solr, start jvisualvm and attach to your running solr. Start

Re: Getting a large number of documents by id

2013-07-18 Thread Alexandre Rafalovitch
And I guess, if only a subset of fields is being requested but there are other large fields present, there could be the cost of loading those extra fields into memory before discarding them. In which case, using enableLazyFieldLoading may help. Regards, Alex. Personal website:

Re: Getting a large number of documents by id

2013-07-18 Thread Brian Hurt
Thanks everyone for the response. On Thu, Jul 18, 2013 at 11:22 AM, Alexandre Rafalovitch arafa...@gmail.comwrote: You could start from doing id:(12345 23456) to reduce the query length and possibly speed up parsing. I didn't know about this syntax- it looks useful. You could also move