[ https://issues.apache.org/jira/browse/LUCENE-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891998#action_12891998 ]
Yonik Seeley commented on LUCENE-2458: -------------------------------------- bq. Before, one ctor used the version specified, the other hardcoded LUCENE_24. Ah.... and that constructor is the one that's used everywhere in Solr (leading me to believe that leaving Solr's default alone was deliberate). bq. As i said before, this shouldnt and cannot be "per-token" and such english centric hacks do not belong in the analysis api. The ability of a filter to say "this token is actually indexed as two adjacent tokens" is fundamental and not related to any specific language. It can be *used* for language specific hacks perhaps... but it is not a hack itself. I never mentioned issues of back compat, but of changes to Solr's default behavior, which I continue to think is the best. I think the best way forward is to add a CJK field to solr that defaults to the opposite behavior (i.e. treats split tokens as completely separate). > queryparser makes all CJK queries phrase queries regardless of analyzer > ----------------------------------------------------------------------- > > Key: LUCENE-2458 > URL: https://issues.apache.org/jira/browse/LUCENE-2458 > Project: Lucene - Java > Issue Type: Bug > Components: QueryParser > Reporter: Robert Muir > Assignee: Robert Muir > Priority: Blocker > Fix For: 3.1, 4.0 > > Attachments: LUCENE-2458.patch, LUCENE-2458.patch, LUCENE-2458.patch, > LUCENE-2458.patch > > > The queryparser automatically makes *ALL* CJK, Thai, Lao, Myanmar, Tibetan, > ... queries into phrase queries, even though you didn't ask for one, and > there isn't a way to turn this off. > This completely breaks lucene for these languages, as it treats all queries > like 'grep'. > Example: if you query for f:abcd with standardanalyzer, where a,b,c,d are > chinese characters, you get a phrasequery of "a b c d". if you use cjk > analyzer, its no better, its a phrasequery of "ab bc cd", and if you use > smartchinese analyzer, you get a phrasequery like "ab cd". But the user > didn't ask for one, and they cannot turn it off. > The reason is that the code to form phrase queries is not internationally > appropriate and assumes whitespace tokenization. If more than one token comes > out of whitespace delimited text, its automatically a phrase query no matter > what. > The proposed patch fixes the core queryparser (with all backwards compat > kept) to only form phrase queries when the double quote operator is used. > Implementing subclasses can always extend the QP and auto-generate whatever > kind of queries they want that might completely break search for languages > they don't care about, but core general-purpose QPs should be language > independent. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org