Well, this IDF problem has more sides. So, let's say your synonym file contains multi-token synonyms (it does, right? or perhaps you don't need it? well, some people do)
"TV, TV set, TV foo, television" if you use the default synonym expansion, when you index 'television' you have increased frequency of also 'set', 'foo', so, the IDF of 'TV' is the same as that of 'television' - but IDF of 'foo' and 'set' has changed (their frequency increased, their IDF decreased) -- TV's have in fact made 'foo' term very frequent and undesirable So, you might be sure that IDF of 'TV' and 'television' are the same, but you are not aware it has 'screwed' other (desirable) terms - so it really depends. And I wouldn't argue these cases are esoteric. And finally: there are use cases out there, where people NEED to switch off synonym expansion at will (find only these documents, that contain the word 'TV' and not that bloody 'foo'). This cannot be done if the index contains all synonym terms (unless you have a way to mark the original and the synonym in the index). roman On Wed, Dec 12, 2012 at 12:50 PM, Walter Underwood <wun...@wunderwood.org>wrote: > Query parsers cannot fix the IDF problem or make query-time synonyms > faster. Query synonym expansion makes more search terms. More search terms > are more work at query time. > > The IDF problem is real; I've run up against it. The most rare variant of > the synonym have the highest score. This probably the opposite of what you > want. For me, it was "TV" and "television". Documents with "TV" had higher > scores than those with "television". > > wunder > > On Dec 12, 2012, at 9:45 AM, Roman Chyla wrote: > > > @wunder > > It is a misconception (well, supported by that wiki description) that the > > query time synonym filter have these problems. It is actually the default > > parser, that is causing these problems. Look at this if you still think > > that index time synonyms are cure for all: > > https://issues.apache.org/jira/browse/LUCENE-4499 > > > > @joe > > If you can use the flexible query parser (as linked in by @Swati) then > all > > you need to do is to define a different field with a different tokenizer > > chain and then swap the field names before the analyzers processes the > > document (and then rewrite the field name back - for example, we have > > fields called "author" and "author_nosyn") > > > > roman > > > > On Wed, Dec 12, 2012 at 12:38 PM, Walter Underwood < > wun...@wunderwood.org>wrote: > > > >> Query time synonyms have known problems. They are slower, cause > incorrect > >> IDF, and don't work for phrase synonyms. > >> > >> Apply synonyms at index time and you will have none of those problems. > >> > >> See: > >> > http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.SynonymFilterFactory > >> > >> wunder > >> > >> On Dec 12, 2012, at 9:34 AM, Swati Swoboda wrote: > >> > >>> Query-time analyzers are still applied, even if you include a string in > >> quotes. Would you expect "foo" to not match "Foo" just because it's > >> enclosed in quotes? > >>> > >>> Also look at this, someone who had similar requirements: > >>> > >> > http://lucene.472066.n3.nabble.com/Synonym-Filter-disable-at-query-time-td2919876.html > >>> > >>> > >>> -----Original Message----- > >>> From: joe.cohe...@gmail.com [mailto:joe.cohe...@gmail.com] > >>> Sent: Wednesday, December 12, 2012 12:09 PM > >>> To: solr-user@lucene.apache.org > >>> Subject: Re: Can a field with defined synonym be searched without the > >> synonym? > >>> > >>> > >>> I'm aplying only query-time synonym, so I have the original values > >> stored and indexed. > >>> I would've expected that if I search a strin with quotations, i'll get > >> the exact match, without applying a synonym. > >>> > >>> any way to achieve that? > >>> > >>> > >>> Upayavira wrote > >>>> You can only search against terms that are stored in your index. If > >>>> you have applied index time synonyms, you can't remove them at query > >> time. > >>>> > >>>> You can, however, use copyField to clone an incoming field to another > >>>> field that doesn't use synonyms, and search against that field > instead. > >>>> > >>>> Upayavira > >>>> > >>>> On Wed, Dec 12, 2012, at 04:26 PM, > >>> > >>>> joe.cohen.m@ > >>> > >>>> wrote: > >>>>> Hi > >>>>> I hava a field type without defined synonym.txt which retrieves both > >>>>> records with "home" and "house" when I search either one of them. > >>>>> > >>>>> I want to be able to search this field on the specific value that I > >>>>> enter, without the synonym filter. > >>>>> > >>>>> is it possible? > >>>>> > >>>>> thanks. > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> View this message in context: > >>>>> > http://lucene.472066.n3.nabble.com/Can-a-field-with-defined-synonym-b > >>>>> e-searched-without-the-synonym-tp4026381.html > >>>>> Sent from the Solr - User mailing list archive at Nabble.com. > >>> > >>> > >>> > >>> > >>> > >>> -- > >>> View this message in context: > >> > http://lucene.472066.n3.nabble.com/Can-a-field-with-defined-synonym-be-searched-without-the-synonym-tp4026381p4026405.html > >>> Sent from the Solr - User mailing list archive at Nabble.com. > >> > >> -- > >> Walter Underwood > >> wun...@wunderwood.org > >> > >> > >> > >> > > -- > Walter Underwood > wun...@wunderwood.org > > > >
