On closer inspection, it seems that foreaching through the hits alone is
enough to cause a memory usage to climb.
foreach ($hits as $hit) {
echo $hit->SomeData;
echo '<br>'.memory_get_usage().'<br>';
@ob_flush();
@flush();
}
It looks as if each time through the loop, the previous $hit is still
in memory.
>>> till <[email protected]> 7/1/2009 10:33:40 AM >>>
On Wed, Jul 1, 2009 at 5:23 PM, Michael Kimsal<[email protected]>
wrote:
> This seems a bit odd, as I've had SOLR (which uses Lucene) deal with
far
> more than that and sorted without problem.
>
> Open question - for those of you using Lucene with the Zend
Framework
> library, what's been the largest set you've dealt with, and what was
your
> performance like (sorting, etc)? I've only personally known people
using
> the ZFLucene stuff with under 20k, and they'd complained about
performance
> (which was likely slow because of large amounts of memory being used,
like
> you're seeing).
>
> I see a couple of options:
> 1. Don't try to pull all your fields back at once - just query for a
doc id
> and the one field you want to sort by, and do your query that way.
Then
> figure out the doc ids in the range/slice you want, and then query
for the
> full documents for each id.
>
> 2. investigate using solr as the lucene server. this might not be
an
> option for you, but if it is, it might be worth considering.
I can second that. We are currently using Solr with the implementation
in ezComponents.
On a side-note, there has been an article in a recent issue of
php|Architect which dealt with many caveats of Zend_Search_Lucene.
Maybe backorder the issue (or get the PDF) and check it out.
Till
