Right - QueryParsers generally do a first pass, parsing incoming Strings
using their operator characters tok tokenize the input and only after that
do they pass the tokens (or phrases) to an Analyzer. I haven't checked
Dismax - not sure how it does its parsing exactly, but I doubt you can just
"turn on the right Analyzer" to get it to recognize curly quotes as phrase
operators, eg.

On Tue, Jan 22, 2019 at 10:39 AM Mikhail Khludnev <m...@apache.org> wrote:

> My impression that these quotes are ones which are part of dismax query
> syntax ie they should be handled before the analysis happens.
>
> On Mon, Jan 21, 2019 at 8:09 PM Walter Underwood <wun...@wunderwood.org>
> wrote:
>
>> First, check which transforms are already handled by Unicode
>> normalization. Put this in all of your analyzer chains:
>>
>>         <charFilter class="solr.ICUNormalizer2CharFilterFactory"/>
>>
>> Probably need this in solrconfig.xml:
>>
>>  <!-- extras for ICU-based Unicode normalization -->
>>   <lib dir="${solr.install.dir:../../../..}/contrib/analysis-extras/lib/"
>> regex=".*\.jar" />
>>   <lib
>> dir="${solr.install.dir:../../../..}/contrib/analysis-extras/lucene-libs"
>> regex=".*\.jar" />
>>
>> I really cannot think of a reason to use unnormalized Unicode in Solr.
>> That should be in all the sample files.
>>
>> For search character matching, yes, all spaces should be normalized. I
>> have too many hacks fixing non-breaking spaces spread around the code. When
>> matching, there is zero use for stuff like ideographic space (U+3000).
>>
>> I’m not sure if quotes are normalized. I did some searching around
>> without success. That might come under character folding. There was a
>> draft, now withdrawn, for standard character folding. I’d probably start
>> there for a Unicode folding char filter.
>>
>> https://www.unicode.org/reports/tr30/tr30-4.html
>>
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>>
>> On Jan 21, 2019, at 7:43 AM, Michael Sokolov <msoko...@gmail.com> wrote:
>>
>> I think this is probably better to discuss on solr-user, or maybe
>> solr-dev, since it is dismax parser you are talking about, which really
>> lives in Solr. However, my 2c  - this seems somewhat dubious. Maybe people
>> want to include those in their terms? Also, it leads to a kind of slippery
>> slope: would you also want to convert all the various white space
>> characters (no-break space, thin space, em space, etc)  as vanilla ascii
>> 32? How about all the other "operator" characters like brackets?
>>
>> On Mon, Jan 21, 2019 at 9:50 AM John Ryan <johnryan_...@yahoo.com.invalid>
>> wrote:
>>
>>> I'm looking to create an issue to add support for Unicode Double Quotes
>>> to the dismax parser.
>>>
>>> I want to replace all types of double quotes with standard ones before
>>> they get stripped
>>>
>>> i.e.
>>>         “ ” „ “ „ « » ‟ ❝ ❞ ⹂ "
>>>
>>> With
>>>         "
>>> I presume this has been discussed before?
>>>
>>> I have a POC here:
>>> https://github.com/apache/lucene-solr/compare/branch_7x...jnyryan:branch_7x
>>>
>>> Thanks,
>>>
>>> John
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>>
>>
>
> --
> Sincerely yours
> Mikhail Khludnev
>

Reply via email to