[
https://issues.apache.org/jira/browse/LUCENE-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226494#comment-13226494
]
Doron Cohen commented on LUCENE-3821:
-------------------------------------
{quote}
Not understanding really how SloppyPhraseScorer works now, but not trying to
add confusion to the issue, I can't help but think this problem is a variant on
LevensteinAutomata... in fact that was the motivation for the new test, i just
stole the testing methodology from there and applied it to this!
{quote}
Interesting! I was not aware of this. I went and read some about this
automaton, It is relevant.
{quote}
It seems many things are the same but with a few twists:
* fundamentally we are interleaving the streams from the subscorers into the
'index automaton'
'query automaton' is produced from the user-supplied terms
{quote}
True. In fact, the current code works hard to decide on the "correct
interleaving order" - while if we had a "Perfect Levenstein Automaton" that
took care of the computation we could just interleave, in the term position
order (forget about phrase position and all that) and let the automaton compute
the distance.
This might capture the difficulty in making the sloppy phrase scorer correct:
it started with the algorithm that was optimized for exact matching, and
adopted (hacked?) it for approximate matching.
Instead, starting with a model that fits approximate matching, might be easier
and cleaner. I like that.
{quote}
* our 'alphabet' is the terms, and holes from position increment are just an
additional symbol.
* just like the LevensteinAutomata case, repeats are problematic because they
are different characteristic vectors
* stacked terms at the same position (index or query) just make the automata
more complex (so they arent just strings)
I'm not suggesting we try to re-use any of that code at all, i don't think it
will work. But I wonder if we can re-use even
some of the math to redefine the problem more formally to figure out what
minimal state/lookahead we need for example...
{quote}
I agree. I'll think of this.
In the meantime I'll commit this partial fix.
> SloppyPhraseScorer sometimes misses documents that ExactPhraseScorer finds.
> ---------------------------------------------------------------------------
>
> Key: LUCENE-3821
> URL: https://issues.apache.org/jira/browse/LUCENE-3821
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 3.5, 4.0
> Reporter: Naomi Dushay
> Assignee: Doron Cohen
> Attachments: LUCENE-3821-SloppyDecays.patch, LUCENE-3821.patch,
> LUCENE-3821.patch, LUCENE-3821.patch, LUCENE-3821.patch,
> LUCENE-3821_test.patch, schema.xml, solrconfig-test.xml
>
>
> The general bug is a case where a phrase with no slop is found,
> but if you add slop its not.
> I committed a test today (TestSloppyPhraseQuery2) that actually triggers this
> case,
> jenkins just hasn't had enough time to chew on it.
> ant test -Dtestcase=TestSloppyPhraseQuery2 -Dtests.iter=100 is enough to make
> it fail on trunk or 3.x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
