[
https://issues.apache.org/jira/browse/LUCENE-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559595#action_12559595
]
Grant Ingersoll commented on LUCENE-1137:
-----------------------------------------
{quote}
The information encoded by BitSet is a set of <int,boolean> tuples. These are
opaque values. In order for this to work, every tokenizer in the chain has to
be aware of every other one's use of these. This makes sharing hard.
{quote}
To some extent, though, the same is true for the current type() functionality.
One may decide to change the type, based on the value of the current type.
While I agree the sharing is hard, it is not impossible, as one need just make
sure to communicate which bits are available. I suppose I could see about
adding a isClaimed(int position) method or something like that, whereby one can
query the chain to see if anyone claims ownership on that position. I'll give
that a try. However, to some extent, I also think it is buyer beware in that
TokenFilters further down the chain just need to be aware of what is going on.
This is part of constructing an Analyzer that works.
As for the naming, I suppose we could do Flags, as Yonik suggests.
> Token type as BitSet: typeBits()
> --------------------------------
>
> Key: LUCENE-1137
> URL: https://issues.apache.org/jira/browse/LUCENE-1137
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 2.4
>
> Attachments: LUCENE-1137.patch
>
>
> It is sometimes useful to have a more compact, easy to parse, type
> representation for Token than the current type() String. This patch adds a
> BitSet onto Token, defaulting to null, with accessors for setting bit flags
> on a Token. This is useful for communicating information about a token to
> TokenFilters further down the chain.
> For example, in the WikipediaTokenizer, the possibility exists that a token
> could be both a category and bold (or many other variations), yet it is
> difficult to communicate this without adding in a lot of different Strings
> for type. Unlike using the payload information (which could serve this
> purpose), the BitSet does not get added to the index (although one could
> easily convert it to a payload.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
