This seems a bit odd, as I've had SOLR (which uses Lucene) deal with far more than that and sorted without problem.
Open question - for those of you using Lucene with the Zend Framework library, what's been the largest set you've dealt with, and what was your performance like (sorting, etc)? I've only personally known people using the ZFLucene stuff with under 20k, and they'd complained about performance (which was likely slow because of large amounts of memory being used, like you're seeing). I see a couple of options: 1. Don't try to pull all your fields back at once - just query for a doc id and the one field you want to sort by, and do your query that way. Then figure out the doc ids in the range/slice you want, and then query for the full documents for each id. 2. investigate using solr as the lucene server. this might not be an option for you, but if it is, it might be worth considering. #1 might be easier to implement, but if the speed still isn't good, you may have to consider other options (like #2, or looking at sphinx, or something else). Hopefully someone can point you to an easier solution. On Wed, Jul 1, 2009 at 11:18 AM, Matt Schraeder <[email protected]> wrote: > My current memory limit is 750 MB. We're trying to do the sort WITHOUT > sending the entire result set to the user. Ideally we want a sorted result > set, then pass pages of 10, 20, 30, 40, 50, etc results back to the user. > But I can't figure out a way of getting the results returned by Lucene to be > sorted without running out of memory. > > >>> till <[email protected]> 7/1/2009 10:10:32 AM >>> > > On Wed, Jul 1, 2009 at 5:11 PM, Matt Schraeder<[email protected]> wrote: > > I'm having a rather drastic ussing with sorting using Zend Search Lucene. > > Our index has 18,000 documents, so light as far as Lucene is concerned. > I > > can perform a search that will pull back all of our records with no > problem. > > > > However, once I attempt to add a sort field (Sorting by Title for > instance), > > the script crashes with an out of memory exception. What is the best way > to > > handle sorting to avoid running out of memory? > > Not sure if it's helpful, but maybe sort on the client side with JS? > > Also, what is your current memory_limit, and how much more does it require? > > Till > -- Michael Kimsal http://jsmag.com - for javascript developers http://groovymag.com - for groovy developers 919.827.4724
