On Sun, Jul 24, 2011 at 1:40 AM, Abhinav Upadhyay
<er.abhinav.upadh...@gmail.com> wrote:
> On Sat, Jul 23, 2011 at 11:00 PM, Richard Hipp <d...@sqlite.org> wrote:
>> On Sat, Jul 23, 2011 at 1:01 PM, Abhinav Upadhyay <
>> er.abhinav.upadh...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am using the Sqlite3 amalgamation. I am trying to register a custom
>>> tokenizer with sqlite for my FTS application. The custom tokenizer is
>>> in it's separate source file. I have included sqlite3.h header with
>>> the tokenizer source but sqlite3.h does not contain the declaration of
>>> the various structs like sqlite3_tokenizer_module etc. So what is the
>>> usual way to resolve this ? These declarations are also not there in
>>> sqlite3ext.h . Although I see them there in sqlite3.c but I am not
>>> sure I want to include it ? What is the usual way to resolve this ?
>>> May be import fts3_tokenizer.h from the sqlite3 source ?
>>>
>>
>> Yes. Use fts3_tokenizer.h.
>
> Thanks, That worked. Now, I am able to compile everything.
>
> Next, I am having problem with using this tokenizer. I followed the
> exampple from the FTS3 documentation page on the website and
> registered the tokenizer using code like this:
>
> const sqlite3_tokenizer_module *stopword_tokenizer_module;
>
> sqlstr = "select fts3_tokenizer(:tokenizer_name, :tokenizer_ptr)";
> rc = sqlite3_prepare_v2(db, sqlstr, -1, &stmt, NULL);
> if (rc != SQLITE_OK) {
> sqlite3_close(db);
> sqlite3_shutdown();
> return -1;
> }
>
> idx = sqlite3_bind_parameter_index(stmt, ":tokenizer_name");
> rc = sqlite3_bind_text(stmt, idx, "my_tokenizer", -1, NULL);
> if (rc != SQLITE_OK) {
> fprintf(stderr, "%s\n", sqlite3_errmsg(db));
> sqlite3_finalize(stmt);
> return -1;
> }
>
> sqlite3Fts3MyTokenizerModule((const sqlite3_tokenizer_module
> **)&my_tokenizer_module);
> idx = sqlite3_bind_parameter_index(stmt, ":tokenizer_ptr");
> rc = sqlite3_bind_blob(stmt, idx, &my_tokenizer_module,
> sizeof(my_tokenizer_module), SQLITE_STATIC);
> if (rc != SQLITE_OK) {
> fprintf(stderr, "%s\n", sqlite3_errmsg(db));
> sqlite3_finalize(stmt);
> return -1;
> }
> rc = sqlite3_step(stmt);
> if (rc != SQLITE_ROW) {
> fprintf(stderr, "%s\n", sqlite3_errmsg(db));
> sqlite3_finalize(stmt);
> return -1;
> }
> sqlite3_finalize(stmt);
>
> It is working fine till above. After executing the above statements, I
> try to create an FTS table using this custom tokenizer, which also
> seem to be getting created. The problem is coming up when I try to
> insert data in the table. A simple statement like "insert into
> my_table values(...)" is giving out errors:
>
> unknown tokenizer: my_tokenizer
>
> I am sure I am missing something here, but don't know what ?
>
> Thanks
>
Nevermind. Seems like I need to register the tokenizer everytime I try
to query the database or store the values.
Thanks
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
