[
https://issues.apache.org/jira/browse/LUCENE-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir updated LUCENE-3842:
--------------------------------
Attachment: LUCENE-3842.patch
messy,dirty,drinking beer on a saturday-afternoon style patch with tons of
nocommits.
particularly:
* we should make the separator optional, i dont think e.g. for the japanese
case we care so much about segmentation but readings, but for the english case
we do.
* both index-time and query-time should respect position increments (if you
dont want that for e.g. stopfilter, set your analyzer correctly). But i dont
even want to go this route until after LUCENE-3767 has landed, it adds
PositionLengthAttribute, and there is no sense in dealing with a "lossy"
tokenstream since we want the real graph, not one thats been sausaged into the
existing tokenstream model.
* duplicates on the input side dont matter, of course two suggestions could
analyze to the same thing. Though this is not particularly desirable (as they
will likely result in the same query results), I dont think we should just drop
duplicates for the highest cost? instead we can add bogus bytes to the end of
the FST input side to uniquify them, since it wont matter anyway.
> Analyzing Suggester
> -------------------
>
> Key: LUCENE-3842
> URL: https://issues.apache.org/jira/browse/LUCENE-3842
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/spellchecker
> Affects Versions: 3.6, 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3842.patch
>
>
> Since we added shortest-path wFSA search in LUCENE-3714, and generified the
> comparator in LUCENE-3801,
> I think we should look at implementing suggesters that have more capabilities
> than just basic prefix matching.
> In particular I think the most flexible approach is to integrate with
> Analyzer at both build and query time,
> such that we build a wFST with:
> input: analyzed text such as ghost0christmas0past <-- byte 0 here is an
> optional token separator
> output: surface form such as "the ghost of christmas past"
> weight: the weight of the suggestion
> we make an FST with PairOutputs<weight,output>, but only do the shortest path
> operation on the weight side (like
> the test in LUCENE-3801), at the same time accumulating the output (surface
> form), which will be the actual suggestion.
> This allows a lot of flexibility:
> * Using even standardanalyzer means you can offer suggestions that ignore
> stopwords, e.g. if you type in "ghost of chr...",
> it will suggest "the ghost of christmas past"
> * we can add support for synonyms/wdf/etc at both index and query time (there
> are tradeoffs here, and this is not implemented!)
> * this is a basis for more complicated suggesters such as Japanese
> suggesters, where the analyzed form is in fact the reading,
> so we would add a TokenFilter that copies ReadingAttribute into term text
> to support that...
> * other general things like offering suggestions that are more "fuzzy" like
> using a plural stemmer or ignoring accents or whatever.
> According to my benchmarks, suggestions are still very fast with the
> prototype (e.g. ~ 100,000 QPS), and the FST size does not
> explode (its short of twice that of a regular wFST, but this is still far
> smaller than TST or JaSpell, etc).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
