[
https://issues.apache.org/jira/browse/LUCENE-8250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437157#comment-16437157
]
Robert Muir commented on LUCENE-8250:
-------------------------------------
ok I will dig into it. But i think its coming back, I think the problem is that
these bogus "holes" are incompatible with conversion of the stream to an
automaton in this case.
The idea if a "hole" is that from a position perspective, the token is removed
but we leave evidence that it was there. So its wrong to modify positionLength
when deleting a token if we are going to do holes.
An alternative is to not leave a hole at all when deleting a token. Instead
FilteringTokenFilter would adjust posinc/poslen as needed to behave as if the
token was never there in the first place. It would need some additional
buffering to do this correctly. So that's how LUCENE-4065 ties in.
> Should FilteringTokenFilter handle positionLength
> -------------------------------------------------
>
> Key: LUCENE-8250
> URL: https://issues.apache.org/jira/browse/LUCENE-8250
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Jim Ferenczi
> Priority: Major
> Attachments: LUCENE-8250.patch
>
>
> FilteringTokenFilter does not handle the position length graph attribute when
> removing a token from the stream. This doesn't work well with graph token
> stream that sets position length since removing a token from the stream can
> invalidate the position length set on the previous tokens.
> This issue was first discussed in
> https://issues.apache.org/jira/browse/LUCENE-4065 but it has a different
> purpose which is why I am opening a new issue here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
