OK, so I thought some more concrete evidence might be helpful to make the
case here and did a quick POC. To get access to precise within-token
offsets we do need to make some changes to the public API, but the profile
could be kept small. In the version I worked up, I extracted the character
offset mapping implementation from BaseCharFilter into a separate
CharOffsetMap interface/class and added these new public methods to
existing classes:
TokenStream.getCharOffsetMap()
CharFilter.uncorrect(int correctOffset) (pseudo-inverse of correct --
returns the left-most offset in the current character coordinates that
corresponds to the given original character offset)
The CharOffsetMap interface has just two methods correctOffset and
uncorrectOffset that support the offset mapping in both CharFilter and
TokenStream
To fully support setting offsets in TokenFilters we need (at least
something like) this inverse offset-correction method (uncorrect) because
OffsetAttribute's offsets are in the original "correct" character
coordinates, but token lengths in incrementToken() are in filtered ("not
correct") character space, and are not anchored to the origin so cannot be
converted directly.
I recognize the impact is not huge here, but we do have TokenFilters that
split tokens, and a currently trappy OffsetAttribute API. Personally I
think it makes sense to acknowledge that and make it a first class citizen,
but I guess another alternative (for fixing the trappiness) would be to
make OffsetAttribute unmodifiable. I know that either approach would have
saved me hours of confusion as I tried to correctly implement offsets.
On Wed, Aug 1, 2018 at 8:57 AM Michael Sokolov <msoko...@gmail.com> wrote:
> Given that character transformations do happen in TokenFilters, shouldn't
> we strive to have an API that supports correct offsets (ie highlighting)
> for any combination of token filters? Currently we can't do that. For
> example because of the current situation, WordDelimiterGraphFilter,
> decompounding filters and the like cannot assign offsets correctly, so eg
> it becomes impossible to highlight the text that exactly corresponds to the
> user query.
>
> Just one example, if I have URLs in some document text, and analysis chain
> is Whitespace tokenizer followed by WordDelimiterGraphFilter, then a query
> for "http" will end up highlighting the entire URL.
>
> Do you have an idea how we can address this without making our apis crazy?
> Or are you just saying we should live with it as it is?
>
> -Mike
>
>
> On Tue, Jul 31, 2018 at 6:36 AM Robert Muir <rcm...@gmail.com> wrote:
>
>> The problem is not a performance one, its a complexity thing. Really I
>> think only the tokenizer should be messing with the offsets...
>> They are the ones actually parsing the original content so it makes
>> sense they would produce the pointers back to them.
>> I know there are some tokenfilters out there trying to be tokenizers,
>> but we don't need to make our apis crazy to support that.
>>
>> On Mon, Jul 30, 2018 at 11:53 PM, Michael Sokolov <msoko...@gmail.com>
>> wrote:
>> > Yes, in fact Tokenizer already provides correctOffset which just
>> delegates
>> > to CharFilter. We could expand on this, moving correctOffset up to
>> > TokenStream, and also adding correct() so that TokenFilters can add to
>> the
>> > character offset data structure (two int arrays) and share it across the
>> > analysis chain.
>> >
>> > Implementation-wise this could continue to delegate to CharFilter I
>> guess,
>> > but I think it would be better to add a character-offset-map abstraction
>> > that wraps the two int arrays and provides the correct/correctOffset
>> > methods to both TokenStream and CharFilter.
>> >
>> > This would let us preserve correct offsets in the face of manipulations
>> > like replacing ellipses, ligatures (like AE, OE), trademark symbols
>> > (replaced by "tm") and the like so that we can have the invariant that
>> > correctOffset(OffsetAttribute.startOffset) + CharTermAttribute.length()
>> ==
>> > correctOffset(OffsetAttribute.endOffset), roughly speaking, and enable
>> > token-splitting with correct offsets.
>> >
>> > I can work up a proof of concept; I don't think it would be too
>> > API-intrusive or change performance in a significant way. Only
>> > TokenFilters that actually care about this (ie that insert or remove
>> > characters, or split tokens) would need to change; others would
>> continue to
>> > work as-is.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>
>>
