[
https://issues.apache.org/jira/browse/LUCENE-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559607#action_12559607
]
Yonik Seeley commented on LUCENE-1137:
--------------------------------------
If we go with the bitset (int or long!!!), "type" could be deprecated...
there's no reason to have both.
StandardTokenizer could define constants to replace
public static final String [] TOKEN_TYPES = new String [] {
"<ALPHANUM>",
"<APOSTROPHE>",
"<ACRONYM>",
"<COMPANY>",
"<EMAIL>",
"<HOST>",
"<NUM>",
"<CJ>"
};
StandardTokenizer.ALPHANUM, etc
> Token type as BitSet: typeBits()
> --------------------------------
>
> Key: LUCENE-1137
> URL: https://issues.apache.org/jira/browse/LUCENE-1137
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 2.4
>
> Attachments: LUCENE-1137.patch
>
>
> It is sometimes useful to have a more compact, easy to parse, type
> representation for Token than the current type() String. This patch adds a
> BitSet onto Token, defaulting to null, with accessors for setting bit flags
> on a Token. This is useful for communicating information about a token to
> TokenFilters further down the chain.
> For example, in the WikipediaTokenizer, the possibility exists that a token
> could be both a category and bold (or many other variations), yet it is
> difficult to communicate this without adding in a lot of different Strings
> for type. Unlike using the payload information (which could serve this
> purpose), the BitSet does not get added to the index (although one could
> easily convert it to a payload.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
