Hi Bill That's some serious debugging you've been doing. It would be great to share those findings on the Sphinx forum if you've not done so already, because I agree, if a word appears in multiple fields, I'd still expect weighting to be a factor.
-- Pat On 03/04/2009, at 5:31 PM, wbharding wrote: > > Alright, I don't have conclusive-conclusive evidence as to the > resolution of this, but I do have some strong circumstantial evidence > about what was going on. > > It seems that the field weights were "working," but that they didn't > work the way I had expected them to. > > For my test case, I was using our "item" model, and trying to play > with the weights of the "title" and "description" field. What I > thought would happen with field weights would be that if I weighted > title higher, then the results that better matched the title (i.e., > the title was all alone, or the title was at the front of the phrase) > would raise in the results vs. compared with those that were poorer > title matches and had description matches. > > The way that the weighting actually works is that weighting doesn't > change anything about how Sphinx interprets the quality of the match. > If two results have a certain phrase anywhere in the title and > anywhere in the description, their order won't be affected by field > weighting. > > What made this more elusive to discover was that we have hardly any > results that have a phrase in the title but don't have that same > phrase in the description. Basically, with field weighting > "working" (the way Sphinx implements it), the only way field weights > could have changed our result order would have been if we searched for > "Kibbles n' bits", and item A only had "Kibbles n' bits" in the title > while item B only had "Kibbles n' bits" in the description, then > changing field weights would reorder the two. Since that's almost > never the case (at least amongst our higher matching items... our > database has close to 2 million items, so almost all the top results I > was looking at matched for both title and description), our results > never seemed to change. > > For reference for future people who want to debug this themselves, our > developer debugged this by connecting to Sphinx directly (via Riddle) > in IRB. Riddle speaks to Sphinx in some crazy binary language, but > you can still compose a query to Sphinx, and you'll be able to see the > individual and cumulative weightings of the results in raw form as > they return from Sphinx. If you find the right rare case where field > weights matter, you should see your results change. > > Hopefully I'm right about this. Like I said, we couldn't do extensive > testing on it cuz we had so few cases where a term was mutually > exclusive amongst the weighted fields. > > Bill > > On Mar 22, 12:15 pm, wbharding <[email protected]> wrote: >> Still no dice. From what I can tell, I'm going to have to dig into >> how the query is being sent to Sphinx and what the actual weighting >> values are that are being returned. I've spent some time in the >> Sphinx forums this morning and it sounds like numerous people have >> gotten this working successfully, though I haven't actually seen any >> verifiable examples where TS users got this working correctly? They >> must be out there, or I'm sure you would have heard about this >> sooner... but I've tried this across multiple different models/ >> indexes >> with many param sets, and it seems like I'm doing something >> systematically wrong+hard to debug. >> >> Bill >> >> On Mar 21, 1:25 am, Pat Allan <[email protected]> wrote: >> >>> Hi Bill >> >>> Just wondering - what match mode are you using? Perhaps it only >>> works >>> in extended mode - from the documentation it seems quite clear >>> that no >>> weighting happens for boolean mode, but not sure about the >>> others:http://www.sphinxsearch.com/docs/manual-0.9.8.html#weighting >> >>> so: >>> Item.search "bizarre", :match_mode => :extended, :field_weights=> { >>> "title" => 100, >>> "tag_names" => 1 >> >>> } >> >>> -- >>> Pat >> >>> On 18/03/2009, at 6:16 PM, wbharding wrote: >> >>>> Same thing -- set_property has no affect on the search results. >> >>>> Just updated to the trunk version of TS and am continuing to get no >>>> difference fromfield_weights. >> >>>> HMM. >> >>>> I looked into the riddle code, and as best I can tell, it looks >>>> like >>>> it's trying to pass the field weights to Sphinx. When I do a query >>>> like Item.search("bizarre", :field_weights=> { :title => >>>> 1, :tag_names => 20 }). I've gone into the >>>> Riddle::Client#query_message and I can see the message includes the >>>> string "field_weights" (though the value of field weights appears >>>> to >>>> be a bunch of binary code. >> >>>> It seems like a reasonable first step in debugging might be to >>>> separate the TS from Sphinx itself? Is there a way I can >>>> communicate >>>> with Sphinx with little/no intervention from TS? The problem feels >>>> like Sphinx might be ignoring my field weights...? >> >>>> Thanks again, >>>> Bill >> >>>> On Mar 15, 10:38 pm, Pat Allan <[email protected]> wrote: >>>>> Hi Bill >> >>>>> I've no idea why this isn't working... what happens if you set the >>>>> field weights using set_property in your define_index block >>>>> instead? >> >>>>> It's annoying that we can't test this using Sphinx's commandline >>>>> search tool, as it doesn't have flags for weights. >> >>>>> -- >>>>> Pat >> >>>>> On 16/03/2009, at 3:51 PM, wbharding wrote: >> >>>>>> Hey all, I'm having a problem getting TS to acknowledge >>>>>> the :field_weightsI am setting for my search. I have tried to >>>>>> boil >>>>>> this problem down to be as simple as possible, but am having a >>>>>> hard >>>>>> time figuring out where to dig in debugging it. Here is my >>>>>> model: >> >>>>>> class Item < ActiveRecord::Base >>>>>> define_index do >>>>>> indexes title, :sortable => true >>>>>> indexes description >>>>>> indexes item_tags.tag_name, :as => :tag_names >>>>>> indexes categories.name, :as => :category_names >>>>>> .... >>>>>> end >>>>>> ... >>>>>> end >> >>>>>> I have a database with a bunch of items, some of which have the >>>>>> word >>>>>> "bizarre" in their titles, some of which have been tagged as >>>>>> "bizarre." But I can run either of these two searches and I get >>>>>> identical results: >> >>>>>> Item.search("bizarre", :field_weights=> {"title" => 1, >>>>>> "tag_names" => >>>>>> 20 }) >>>>>> Item.search("bizarre", :field_weights=> {"title" => 20, >>>>>> "tag_names" >>>>>> => 1 }) >> >>>>>> I have tried a variety of ideas... increasing and decreasing the >>>>>> arbitrarily chosen weight of "20", searching with different >>>>>> words, >>>>>> setting different match modes, using symbols to name the indexes >>>>>> instead of strings, and even different queries... all for naught. >>>>>> No >>>>>> matter what I try, I continue to get the same set of results >>>>>> regardless of myfield_weights. I also tried a couple tests using >>>>>> the >>>>>> set_property version of field weights, and this too had no >>>>>> effect on >>>>>> my results. >> >>>>>> I'm running Sphinx 0.9.8-release (r1371) with a release of TS >>>>>> that I >>>>>> grabbed a couple months ago (new enough to have grouping working, >>>>>> 0.9.9?). > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en -~----------~----~----~----~------~----~------~--~---
