Re: [sqlite] How can custom tokenizer tell it's parsing a search string?

Thanks for the reply, Dan! > On Dec 12, 2018, at 7:08 AM, Dan Kennedy wrote: > > Leaving stop words in while parsing queries won't quite work anyway. If your > tokenizer returns "the" when parsing a query, FTS3/4 will search for "the" in > the index. And it won't be there if the tokenizer

Re: [sqlite] How can custom tokenizer tell it's parsing a search string?

On 12/12/2018 03:37 AM, Jens Alfke wrote: Is there any way for a custom FTS4 tokenizer to know when it’s tokenizing a search string (the argument of a MATCH expression), as opposed to text to be indexed? Here’s my problem: I’ve implemented a custom tokenizer that skips “stop words” (noise

[sqlite] How can custom tokenizer tell it's parsing a search string?

Is there any way for a custom FTS4 tokenizer to know when it’s tokenizing a search string (the argument of a MATCH expression), as opposed to text to be indexed? Here’s my problem: I’ve implemented a custom tokenizer that skips “stop words” (noise words, like “the” and “a” in English.) It