[
https://issues.apache.org/jira/browse/LUCENE-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559606#action_12559606
]
Grant Ingersoll commented on LUCENE-1137:
-----------------------------------------
Never mind on the isClaimed() idea, I don't see a good way of how that would
work.
> Token type as BitSet: typeBits()
> --------------------------------
>
> Key: LUCENE-1137
> URL: https://issues.apache.org/jira/browse/LUCENE-1137
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 2.4
>
> Attachments: LUCENE-1137.patch
>
>
> It is sometimes useful to have a more compact, easy to parse, type
> representation for Token than the current type() String. This patch adds a
> BitSet onto Token, defaulting to null, with accessors for setting bit flags
> on a Token. This is useful for communicating information about a token to
> TokenFilters further down the chain.
> For example, in the WikipediaTokenizer, the possibility exists that a token
> could be both a category and bold (or many other variations), yet it is
> difficult to communicate this without adding in a lot of different Strings
> for type. Unlike using the payload information (which could serve this
> purpose), the BitSet does not get added to the index (although one could
> easily convert it to a payload.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
