Yonik/Erick, We are building a custome Search which is to be done in 2 parts executed at different points of time. As a result of it, the first step we want tokenize the information and store it, which we want to retrieve a later point of time for further processing and then store it back into the index. This processed information is what we want the users to be able to search on.
Regards, Eswar On Dec 20, 2007 8:15 PM, Erick Erickson <[EMAIL PROTECTED]> wrote: > I think that what Yonik wants is a higher-level response. > *Why* do you want to process the tokens later? What is the > use case you're trying to satisfy? > > Best > Erick > > On Dec 20, 2007 1:37 AM, Rishabh Joshi <[EMAIL PROTECTED]> wrote: > > > > What are you trying to do with the tokens? > > > > Yonik, we wanted a "tokenizer" that would tokenize the content of a > > document > > as per our requirements, and then store them in the index so that, we > > could > > retrieve those tokens at search time, for further processing in our > > application. > > > > Regards, > > Rishabh > > > > On Dec 19, 2007 10:02 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote: > > > > > On Dec 19, 2007 10:59 AM, Rishabh Joshi <[EMAIL PROTECTED]> wrote: > > > > I have created my own Tokenizer and I am indexing the documents > using > > > the > > > > same. > > > > > > > > I wanted to know if there is a way to retrieve the tokens (created > by > > my > > > > custom tokenizer) from the index. > > > > > > If you want the tokens in the index, see the luke request handler. > > > > > > If you want the tokens for a specific document, it's more > > > complicated... Lucene maintains an *inverted* index... terms point to > > > documents, so by default there is no way to ask for all of the terms > > > in a certain document. One could ask lucene to store the terms for > > > certain fields (called term vectors), but that requires extra space in > > > the index, and solr doesn't yet have a way to ask that they be > > > retrieved. > > > > > > What are you trying to do with the tokens? > > > > > > -Yonik > > > > > >
