[ https://issues.apache.org/jira/browse/LUCENE-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278166#comment-13278166 ]
Sudarshan Gaikaiwari commented on LUCENE-3842: ---------------------------------------------- I was not able to apply the latest patch cleanly {quote} smg@dev21:~/lucene_trunk$ patch -p0 < ~/LUCENE-3842.patch patching file lucene/test-framework/src/java/org/apache/lucene/util/RollingBuffer.java patching file lucene/core/src/java/org/apache/lucene/util/automaton/SpecialOperations.java patching file lucene/core/src/java/org/apache/lucene/util/RollingBuffer.java Hunk #1 FAILED at 109. 1 out of 1 hunk FAILED -- saving rejects to file lucene/core/src/java/org/apache/lucene/util/RollingBuffer.java.rej patching file lucene/core/src/java/org/apache/lucene/util/fst/Util.java patching file lucene/core/src/java/org/apache/lucene/analysis/TokenStreamToAutomaton.java patching file lucene/core/src/test/org/apache/lucene/analysis/TestGraphTokenizers.java patching file lucene/core/src/test/org/apache/lucene/util/automaton/TestSpecialOperations.java patching file lucene/suggest/src/test/org/apache/lucene/search/suggest/analyzing/AnalyzingCompletionTest.java patching file lucene/suggest/src/test/org/apache/lucene/search/suggest/LookupBenchmarkTest.java patching file lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingCompletionLookup.java patching file lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/FSTUtil.java {quote} I needed to copy RollingBuffer.java to from test-framework to core for the patch to apply cleanly. {quote} cp lucene/test-framework/src/java/org/apache/lucene/util/RollingBuffer.java lucene/core/src/java/org/apache/lucene/util/ {quote} > Analyzing Suggester > ------------------- > > Key: LUCENE-3842 > URL: https://issues.apache.org/jira/browse/LUCENE-3842 > Project: Lucene - Java > Issue Type: New Feature > Components: modules/spellchecker > Affects Versions: 3.6, 4.0 > Reporter: Robert Muir > Attachments: LUCENE-3842-TokenStream_to_Automaton.patch, > LUCENE-3842.patch, LUCENE-3842.patch, LUCENE-3842.patch, LUCENE-3842.patch, > LUCENE-3842.patch, LUCENE-3842.patch, LUCENE-3842.patch > > > Since we added shortest-path wFSA search in LUCENE-3714, and generified the > comparator in LUCENE-3801, > I think we should look at implementing suggesters that have more capabilities > than just basic prefix matching. > In particular I think the most flexible approach is to integrate with > Analyzer at both build and query time, > such that we build a wFST with: > input: analyzed text such as ghost0christmas0past <-- byte 0 here is an > optional token separator > output: surface form such as "the ghost of christmas past" > weight: the weight of the suggestion > we make an FST with PairOutputs<weight,output>, but only do the shortest path > operation on the weight side (like > the test in LUCENE-3801), at the same time accumulating the output (surface > form), which will be the actual suggestion. > This allows a lot of flexibility: > * Using even standardanalyzer means you can offer suggestions that ignore > stopwords, e.g. if you type in "ghost of chr...", > it will suggest "the ghost of christmas past" > * we can add support for synonyms/wdf/etc at both index and query time (there > are tradeoffs here, and this is not implemented!) > * this is a basis for more complicated suggesters such as Japanese > suggesters, where the analyzed form is in fact the reading, > so we would add a TokenFilter that copies ReadingAttribute into term text > to support that... > * other general things like offering suggestions that are more "fuzzy" like > using a plural stemmer or ignoring accents or whatever. > According to my benchmarks, suggestions are still very fast with the > prototype (e.g. ~ 100,000 QPS), and the FST size does not > explode (its short of twice that of a regular wFST, but this is still far > smaller than TST or JaSpell, etc). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org