subject:"\[sqlite\] \[FTS3\] Understanding the Flow of data through the tokenizer"

Re: [sqlite] [FTS3] Understanding the Flow of data through the tokenizer

2011-07-25 Thread Abhinav Upadhyay

On Mon, Jul 25, 2011 at 9:54 AM, Dan Kennedy wrote: > On 07/24/2011 08:16 PM, Abhinav Upadhyay wrote: >> Hi, >> >> I am trying to write my own custom tokenizer to filter stopwords apart >> from doing normalization and stemming. I have gone through the >> comments in

Re: [sqlite] [FTS3] Understanding the Flow of data through the tokenizer

2011-07-24 Thread Dan Kennedy

On 07/24/2011 08:16 PM, Abhinav Upadhyay wrote: > Hi, > > I am trying to write my own custom tokenizer to filter stopwords apart > from doing normalization and stemming. I have gone through the > comments in fts3_tokenizer.h and also read the implementation of the > simple tokenizer. While overall

[sqlite] [FTS3] Understanding the Flow of data through the tokenizer

2011-07-24 Thread Abhinav Upadhyay

Hi, I am trying to write my own custom tokenizer to filter stopwords apart from doing normalization and stemming. I have gone through the comments in fts3_tokenizer.h and also read the implementation of the simple tokenizer. While overall I am able to understand what I need to do to implement