On 7/17/06, Jordan Frank <[EMAIL PROTECTED]> wrote:
> <snip>
> > > From: "David Balmain" <[EMAIL PROTECTED]>
> > > <snip>
> > > This gets even messier when you need to page through the results. A
> > > much nicer solution that this would be to add a :filter_proc to the
> > > search methods. Something like this;
> > >
> > > within_radius = lambda do |doc|
> > > return ((doc[:latitude] - latitude) ** 2 +
> > > (doc[:longitude] - longitude) ** 2) < (radius ** 2)
> > > end
> > >
> > > index.search_each(query, :filter_proc => within_radius) {|d, s| ...}
> > >
> > > Does this sound like a good idea? If so I could add it to a future
> > > version of Ferret. Please let me know if you can think of a better way
> > > to do this.
> > > <snip>
>
> This is how I'm doing it now. I guess adding the filter_proc would
> clean up my code a bit, and simplify the paging etc. My question would
> be how you'd handle the problem that I mentioned earlier, that is how
> to determine how many documents to retrieve before the filter_proc is
> evaluated in order to eventually return the desired number of
> documents. I don't know enough about the internals of ferret to know
> if I'm bringing up a valid point, but I'm guessing that if I only
> request the top 5 documents for a query, it doesn't retrieve every
> single document that satisfies the query and then take the top 5 from
> that list. Maybe it does though, as I said, I don't know enough about
> the internals of ferret, though I'd like to...
Ferret actually has to check the score of every singly document in the
index that matches the query. It keeps a priority queue of as many
documents as it needs to return the result set. So if :num_docs is 50,
and :first_doc is 200 Ferret will need to keep a priority queue of 250
documents.
> So if the problem that I bring up is legitimate, then the problem
> would be in coming up with some sort of heuristic based on how many
> documents are expected to satisfy the filter_proc. If only 10% of the
> documents satisfy the filter_proc, then to get the top 5 documents
> matching a query, we'd want to retrieve the top 50 documents
> internally, then pass them through the filter_proc, and hopefully we'd
> be left with at least 5 to return. For my specific application, I'm in
> a better position to determine this hit percentage, and so I'm in a
> better position to do the filtering. I don't know whether doing this
> in ferret would be efficient or even feasible.
You wouldn't need to request more documents than you need using the
:filter_proc idea. You'd just specify :num_docs as usual and you'd get
:num_docs back. So if you want 50 documents you'd get 50 documents (or
less if fewer documents matched the query and distance constraint).
> Anyways, let me know what your thoughts are on this. The filter_proc
> idea is a good one, as long as it can be implemented efficiently.
> Otherwise I'll just keep using my two phase method, retrieve the
> documents from ferret, and then do the location filtering in SQL.
The proc would just be called once for every matching document in the
result set, not every document. It shouldn't be too expensive at all
and probably a lot more efficient than filtering using the SQL method.
Cheers,
Dave
_______________________________________________
Ferret-talk mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ferret-talk
