Alright, I don't have conclusive-conclusive evidence as to the resolution of this, but I do have some strong circumstantial evidence about what was going on.
It seems that the field weights were "working," but that they didn't work the way I had expected them to. For my test case, I was using our "item" model, and trying to play with the weights of the "title" and "description" field. What I thought would happen with field weights would be that if I weighted title higher, then the results that better matched the title (i.e., the title was all alone, or the title was at the front of the phrase) would raise in the results vs. compared with those that were poorer title matches and had description matches. The way that the weighting actually works is that weighting doesn't change anything about how Sphinx interprets the quality of the match. If two results have a certain phrase anywhere in the title and anywhere in the description, their order won't be affected by field weighting. What made this more elusive to discover was that we have hardly any results that have a phrase in the title but don't have that same phrase in the description. Basically, with field weighting "working" (the way Sphinx implements it), the only way field weights could have changed our result order would have been if we searched for "Kibbles n' bits", and item A only had "Kibbles n' bits" in the title while item B only had "Kibbles n' bits" in the description, then changing field weights would reorder the two. Since that's almost never the case (at least amongst our higher matching items... our database has close to 2 million items, so almost all the top results I was looking at matched for both title and description), our results never seemed to change. For reference for future people who want to debug this themselves, our developer debugged this by connecting to Sphinx directly (via Riddle) in IRB. Riddle speaks to Sphinx in some crazy binary language, but you can still compose a query to Sphinx, and you'll be able to see the individual and cumulative weightings of the results in raw form as they return from Sphinx. If you find the right rare case where field weights matter, you should see your results change. Hopefully I'm right about this. Like I said, we couldn't do extensive testing on it cuz we had so few cases where a term was mutually exclusive amongst the weighted fields. Bill On Mar 22, 12:15 pm, wbharding <[email protected]> wrote: > Still no dice. From what I can tell, I'm going to have to dig into > how the query is being sent to Sphinx and what the actual weighting > values are that are being returned. I've spent some time in the > Sphinx forums this morning and it sounds like numerous people have > gotten this working successfully, though I haven't actually seen any > verifiable examples where TS users got this working correctly? They > must be out there, or I'm sure you would have heard about this > sooner... but I've tried this across multiple different models/indexes > with many param sets, and it seems like I'm doing something > systematically wrong+hard to debug. > > Bill > > On Mar 21, 1:25 am, Pat Allan <[email protected]> wrote: > > > Hi Bill > > > Just wondering - what match mode are you using? Perhaps it only works > > in extended mode - from the documentation it seems quite clear that no > > weighting happens for boolean mode, but not sure about the > > others:http://www.sphinxsearch.com/docs/manual-0.9.8.html#weighting > > > so: > > Item.search "bizarre", :match_mode => :extended, :field_weights=> { > > "title" => 100, > > "tag_names" => 1 > > > } > > > -- > > Pat > > > On 18/03/2009, at 6:16 PM, wbharding wrote: > > > > Same thing -- set_property has no affect on the search results. > > > > Just updated to the trunk version of TS and am continuing to get no > > > difference fromfield_weights. > > > > HMM. > > > > I looked into the riddle code, and as best I can tell, it looks like > > > it's trying to pass the field weights to Sphinx. When I do a query > > > like Item.search("bizarre", :field_weights=> { :title => > > > 1, :tag_names => 20 }). I've gone into the > > > Riddle::Client#query_message and I can see the message includes the > > > string "field_weights" (though the value of field weights appears to > > > be a bunch of binary code. > > > > It seems like a reasonable first step in debugging might be to > > > separate the TS from Sphinx itself? Is there a way I can communicate > > > with Sphinx with little/no intervention from TS? The problem feels > > > like Sphinx might be ignoring my field weights...? > > > > Thanks again, > > > Bill > > > > On Mar 15, 10:38 pm, Pat Allan <[email protected]> wrote: > > >> Hi Bill > > > >> I've no idea why this isn't working... what happens if you set the > > >> field weights using set_property in your define_index block instead? > > > >> It's annoying that we can't test this using Sphinx's commandline > > >> search tool, as it doesn't have flags for weights. > > > >> -- > > >> Pat > > > >> On 16/03/2009, at 3:51 PM, wbharding wrote: > > > >>> Hey all, I'm having a problem getting TS to acknowledge > > >>> the :field_weightsI am setting for my search. I have tried to boil > > >>> this problem down to be as simple as possible, but am having a hard > > >>> time figuring out where to dig in debugging it. Here is my model: > > > >>> class Item < ActiveRecord::Base > > >>> define_index do > > >>> indexes title, :sortable => true > > >>> indexes description > > >>> indexes item_tags.tag_name, :as => :tag_names > > >>> indexes categories.name, :as => :category_names > > >>> .... > > >>> end > > >>> ... > > >>> end > > > >>> I have a database with a bunch of items, some of which have the word > > >>> "bizarre" in their titles, some of which have been tagged as > > >>> "bizarre." But I can run either of these two searches and I get > > >>> identical results: > > > >>> Item.search("bizarre", :field_weights=> {"title" => 1, > > >>> "tag_names" => > > >>> 20 }) > > >>> Item.search("bizarre", :field_weights=> {"title" => 20, "tag_names" > > >>> => 1 }) > > > >>> I have tried a variety of ideas... increasing and decreasing the > > >>> arbitrarily chosen weight of "20", searching with different words, > > >>> setting different match modes, using symbols to name the indexes > > >>> instead of strings, and even different queries... all for naught. > > >>> No > > >>> matter what I try, I continue to get the same set of results > > >>> regardless of myfield_weights. I also tried a couple tests using > > >>> the > > >>> set_property version of field weights, and this too had no effect on > > >>> my results. > > > >>> I'm running Sphinx 0.9.8-release (r1371) with a release of TS that I > > >>> grabbed a couple months ago (new enough to have grouping working, > > >>> 0.9.9?). --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en -~----------~----~----~----~------~----~------~--~---
