One of the token patterns defined by the StandardTokenizer.jj is this:
NUM: (ALPHANUM P HAS_DIGIT
| HAS_DIGIT P ALPHANUM
| ALPHANUM (P HAS_DIGIT P ALPHANUM)+
| HAS_DIGIT (P ALPHANUM P HAS_DIGIT)+
| ALPHANUM P HAS_DIGIT (P ALPHANUM P HAS_DIGIT)+
| HAS_DIGIT P ALPHANUM (P HAS_DIGIT P ALPHANUM)+
)
So basically if you have some sequences of characters separated by a -
character, sequences that contain a digit will be combined with sequences
which are adjacent to it to form a single token. That explains why the WS
and YYMM sequences got separated out. You can alter this behavior this with
some simple changes to StandardTokenizer.jj.
- Original Message -
From: Iain Young [EMAIL PROTECTED]
To: 'Lucene Users List' [EMAIL PROTECTED]
Sent: Tuesday, December 16, 2003 7:46 AM
Subject: RE: Disabling modifiers?
I think it is a problem with the indexing. I've found another example...
WS-CA-PP00-PROCESS-YYMM
I've looked at the index, and it has been tokenized into 3 words...
WS
CA-PP00-PROCESS
YYMM
Looks as though I might have to use a custom tokenizer as well as an
analyzer then, but any ideas as to why the standard tokenizer would have
split the variable up like this (i.e. why didn't it split the middle bit,
only the word off either end)? The only thing I can think of is that there
are several other variables in the source beginning with WS- or ending
with
-YYMM, so could the tokenizer have seen this and be doing something clever
with them?
Thanks,
Iain
*
* Micro Focus Developer Forum 2004 *
* 3 days that will make a difference *
* www.microfocus.com/devforum *
*
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
