I for one think this custom filter would be an awesome addition. Geospatial and local search is a hot area and it would be cool if ferret facilitated this type of query easily.
Would it be a significant performance hit if ferret has to cycle through every document for this search? Fine over a couple of hundred, or thousand? but hundreds of thousands? Just tossing around the idea but... This particular search (distance) can be done quite efficiently with sql. Is it at all feasible that you could 'outsource' the query to sql? Obviously sql could return the id's simply enough, but i guess then you'd need to go through each document anyway... To return a bitset, would the database need to know about the ferret document order? Or how about the reverse, use ferret to create a list of ids to pass into a sql IN query? Afraid I have no idea how efficient that would be either... Anyone in here have a best practice? > ---------- Forwarded message ---------- > From: "David Balmain" <[EMAIL PROTECTED]> > To: [email protected] > Date: Mon, 17 Jul 2006 10:54:45 +0900 > Subject: Re: [Ferret-talk] adding a custom filter to the query > On 7/17/06, Jordan Frank <[EMAIL PROTECTED]> wrote: > > On 7/16/06, David Balmain <[EMAIL PROTECTED]> wrote: > > > <snip> > > > > > > That is a perfect example of what you can't use the QueryFilter for. I > > > may even use it as an example in the documentation. Thanks and good > > > luck with the pure Ruby version. > > > > I tried out the pure ruby version, but I'm having a little bit of > > trouble wrapping my head around how to write the filter. It seems that > > some stuff has changed internally since the sample code that I found > > at tourb.us was written. I tried looking at the RangeFilter code, but > > it seems to be solving too different a problem to really be useful as > > a guide. Do you know of any other filters, or have any pointers to how > > I would go about writing this filter? It seems really simple, just > > does a calculation on two of the fields, but because it's not > > iterating through terms, the RangeFilter code doesn't offer me much > > help. If you offer some pointers and I manage to get it working, I'd > > be happy to send you a copy to use as a sample, though it seems like > > the kind of thing you'd probably be able to write in a few minutes... > > I think I slightly misunderstood your problem the first time around. > To create this filter, you actually have to iterate through every > document in the index. This will take some time but it would be worth > it if the filter gets used many times, since it gets cached. However, > I don't think this would work for you because I'm guessing the > longitude, latitude and radius change on a query by query basis. This > is not really what the current filters are designed for. Filters > should be common query restrictions that are run over and over again. > For example, a blog may have a month filter for retrieving documents > from a particular month. This is likely to be used over and over again > and RangeFilters are pretty cheap to build. > > So the current solution to your problem is to actually post-filter > your query results yourself (ie filter the results once you have them > back). So let's say you need ten results. You'd do a search for maybe > 50 and run through each result checking the distance and discarding > the ones you don't need. You'd repeat the search until you found > enough documents. Here is a quick and dirty solution (where num_docs > is the number of documents you want in your resultset); > > def search(index, query, num_docs, latitude, longitude, radius) > first_doc = 0 > results = [] > while true > count = index.search_each(query, > :first_doc => first_doc, > :num_docs => num_docs*5) do |doc_id, score| > doc = index[doc_id] > # test distance and add to resultset if ok > if ((doc[:latitude] - latitude) ** 2 + > (doc[:longitude] - longitude) ** 2) < radius ** 2 > results << doc > end > break if results.size == num_docs # have enough docs > end > break if count < (num_docs * 5) #already scanned all results > first_doc += num_docs * 5 > end > return results > end > > > This gets even messier when you need to page through the results. A > much nicer solution that this would be to add a :filter_proc to the > search methods. Something like this; > > within_radius = lambda do |doc| > return ((doc[:latitude] - latitude) ** 2 + > (doc[:longitude] - longitude) ** 2) < (radius ** 2) > end > > index.search_each(query, :filter_proc => within_radius) {|d, s| ...} > > Does this sound like a good idea? If so I could add it to a future > version of Ferret. Please let me know if you can think of a better way > to do this. > > Cheers, > Dave _______________________________________________ Ferret-talk mailing list [email protected] http://rubyforge.org/mailman/listinfo/ferret-talk
