On Dec 17, 2009, at 3:08 AM, Nick Hodapp wrote: > > Ah, Thanks. That makes sense. > > My tokenizer extracts text-node values from XHTML content and > delegates the > tokenizing of each node value to the Porter tokenizer. Since the > RHS of the > MATCH operator will never be XHTML, I should be able to just defer the > tokenizing of that input directly to the Porter tokenizer? And I > can do > that by registering a custom tokenizer-module in my read-only > application > that specifies the Porter functions, but has the name of the custom > tokenizer I used to index the XHTML data. > > Does that seem reasonable?
Probably. If you really can't use the same tokenizer in both for some reason. The tokens returned by parsing the RHS of the match operator will be compared using memcmp() to the tokens returned when parsing the inserted documents. Dan. > > Nick Hodapp > > > > > D. Richard Hipp wrote: >> >> >> On Dec 16, 2009, at 11:15 AM, D. Richard Hipp wrote: >> >>> >>> On Dec 16, 2009, at 11:12 AM, Nick Hodapp wrote: >>> >>>> I wrote a custom tokenizer for fts3, and I intend it to be used >>>> when I >>>> populate my database. >>>> >>>> I'd rather not ship the tokenizer in my embedded application which >>>> accesses >>>> the database in read-only mode. But it seems that fts3 checks for >>>> the >>>> existence of the tokenizer even when accessing a fts3 table for a >>>> read-only >>>> query. >>>> >>>> Is it possible to change this behavior (or, is the tokenizer >>>> actually >>>> required for a read-only query?) >>>> >>>> If I register a dummy tokenizer with the same name in my read-only >>>> application, would that work? >>> >>> >>> The tokenizer is needed for both reading and writing. It is needing >>> to parse the LHS of the MATCH operator when reading. >> >> Correction: ...the RHS of the MATCH operator.... >> >>> >>> You *must* register exactly the same tokenizer when reading as you >>> used for writing or your queries will not work. >>> >>> D. Richard Hipp >>> [email protected] >>> >>> >>> >>> _______________________________________________ >>> sqlite-users mailing list >>> [email protected] >>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> >> D. Richard Hipp >> [email protected] >> >> >> >> _______________________________________________ >> sqlite-users mailing list >> [email protected] >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> >> > > -- > View this message in context: > http://old.nabble.com/custom-fts3-tokenizer%2C-used-in-read-only--tp26813756p26817624.html > Sent from the SQLite mailing list archive at Nabble.com. > > _______________________________________________ > sqlite-users mailing list > [email protected] > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list [email protected] http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
