> What are you trying to do with the tokens? Yonik, we wanted a "tokenizer" that would tokenize the content of a document as per our requirements, and then store them in the index so that, we could retrieve those tokens at search time, for further processing in our application.
Regards, Rishabh On Dec 19, 2007 10:02 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote: > On Dec 19, 2007 10:59 AM, Rishabh Joshi <[EMAIL PROTECTED]> wrote: > > I have created my own Tokenizer and I am indexing the documents using > the > > same. > > > > I wanted to know if there is a way to retrieve the tokens (created by my > > custom tokenizer) from the index. > > If you want the tokens in the index, see the luke request handler. > > If you want the tokens for a specific document, it's more > complicated... Lucene maintains an *inverted* index... terms point to > documents, so by default there is no way to ask for all of the terms > in a certain document. One could ask lucene to store the terms for > certain fields (called term vectors), but that requires extra space in > the index, and solr doesn't yet have a way to ask that they be > retrieved. > > What are you trying to do with the tokens? > > -Yonik >
