Could you not use a custom analyzer to inject "metadata" tokens into the index
at the same position as the source tokens?
For example, given the text:
The cat jumped over the dog
your analyzer could emit tokens:
[the] [cat,_posNoun] [jumped,_posVerb] [over] [the]
[dog,_posNoun]
where the "_pos...." tokens have a zero position increment to effectively
associate them with the term to which they relate (this is how the example
SynonymTokenizer in the highlighter package works). The "_pos" prefix is used
as a uniquefier for metadata tokens to avoid any name-clashes with any real
content tokens.
Theoretically you could then construct queries where the queries mixed both
data and your part-of-speech metadata eg you could use the position information
based queries to find out what things normally have a particular verb applied
to them:
"jumped _posNoun"~3
or what verbs are commonly associated with a dog (caution advised here):
"_posVerb the dog"~3
or to use an ambiguous word in a particular context/sense
"_posVerb track"~1
Cheers,
Mark
----- Original Message ----
From: Amit Kumar <[EMAIL PROTECTED]>
To: [email protected]
Cc: Amit Kumar <[EMAIL PROTECTED]>
Sent: Wednesday, 12 July, 2006 6:36:24 AM
Subject: Storing Part of Speech information in Lucene Indices
Hi,
A new project that I am investigating lucene for needs the Parts of
speech information for the tokens. I can get that
information using NLP techniques (GATE etc.), by pre processing the
documents but I would like to store that
information in the Indices. Something along the lines of
TermVectorOffsetInfo[?].getPartofSpeech();
I am writing to ask for your advice, you can tell me I am b o n k e r
s or let me know where I should start digging :).
Is that a good idea? Or would it be just less trouble for me to store
the offset information along with parts of speech
outside Lucene.
Has anyone else done that?
Best,
Amit
ps: Thank you for putting the LuceneInAction source online, it was a
great help to see the CategorizerTest.java.
I am ordering my copy of the book tomorrow :)
---------------------------------------------------------
Amit Kumar
Research Programmer
The Graduate School of Library and Information Science
University of Illinois, Urbana Champaign IL, 61820
phone: 217-333-4118 fax: 217-244-3302
---------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
