Re: [sqlite] [FTS3] Header to include for a custom tokenizer

[Abhinav Upadhyay]

On Sun, Jul 24, 2011 at 1:40 AM, Abhinav Upadhyay
 wrote:
> On Sat, Jul 23, 2011 at 11:00 PM, Richard Hipp  wrote:
>> On Sat, Jul 23, 2011 at 1:01 PM, Abhinav Upadhyay <
>> er.abhinav.upadh...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>>  I am using the Sqlite3 amalgamation. I am trying to register a custom
>>> tokenizer with sqlite for my FTS application. The custom tokenizer is
>>> in it's separate source file. I have included sqlite3.h header with
>>> the tokenizer source but sqlite3.h does not contain the declaration of
>>> the various structs like sqlite3_tokenizer_module etc. So what is the
>>> usual way to resolve this ? These declarations are also not there in
>>> sqlite3ext.h . Although I see them there in sqlite3.c but I am not
>>> sure I want to include it ? What is the usual way to resolve this ?
>>> May be import fts3_tokenizer.h from the sqlite3 source ?
>>>
>>
>> Yes.  Use fts3_tokenizer.h.
>
> Thanks, That worked. Now, I am able to compile everything.
>
> Next, I am having problem with using this tokenizer. I followed the
> exampple from the FTS3 documentation page on the website and
> registered the tokenizer using code like this:
>
> const sqlite3_tokenizer_module *stopword_tokenizer_module;
>
> sqlstr = "select fts3_tokenizer(:tokenizer_name, :tokenizer_ptr)";
>        rc = sqlite3_prepare_v2(db, sqlstr, -1, , NULL);
>        if (rc != SQLITE_OK) {
>                sqlite3_close(db);
>                sqlite3_shutdown();
>                return -1;
>        }
>
>        idx = sqlite3_bind_parameter_index(stmt, ":tokenizer_name");
>        rc = sqlite3_bind_text(stmt, idx, "my_tokenizer", -1, NULL);
>        if (rc != SQLITE_OK) {
>                fprintf(stderr, "%s\n", sqlite3_errmsg(db));
>                sqlite3_finalize(stmt);
>                return -1;
>        }
>
>        sqlite3Fts3MyTokenizerModule((const sqlite3_tokenizer_module
> **)_tokenizer_module);
>        idx = sqlite3_bind_parameter_index(stmt, ":tokenizer_ptr");
>        rc = sqlite3_bind_blob(stmt, idx, _tokenizer_module,
> sizeof(my_tokenizer_module), SQLITE_STATIC);
>        if (rc != SQLITE_OK) {
>                fprintf(stderr, "%s\n", sqlite3_errmsg(db));
>                sqlite3_finalize(stmt);
>                return -1;
>        }
>        rc = sqlite3_step(stmt);
>        if (rc != SQLITE_ROW) {
>                fprintf(stderr, "%s\n", sqlite3_errmsg(db));
>                sqlite3_finalize(stmt);
>                return -1;
>        }
>        sqlite3_finalize(stmt);
>
> It is working fine till above. After executing the above statements, I
> try to create an FTS table using this custom tokenizer, which also
> seem to be getting created. The problem is coming up when I try to
> insert data in the table. A simple statement like "insert into
> my_table values(...)" is giving out errors:
>
> unknown tokenizer: my_tokenizer
>
> I am sure I am missing something here, but don't know what ?
>
> Thanks
>

Nevermind. Seems like I need to register the tokenizer everytime I try
to query the database or store the values.

Thanks
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] [FTS3] Header to include for a custom tokenizer

[Abhinav Upadhyay]

On Sat, Jul 23, 2011 at 11:00 PM, Richard Hipp  wrote:
> On Sat, Jul 23, 2011 at 1:01 PM, Abhinav Upadhyay <
> er.abhinav.upadh...@gmail.com> wrote:
>
>> Hi,
>>
>>  I am using the Sqlite3 amalgamation. I am trying to register a custom
>> tokenizer with sqlite for my FTS application. The custom tokenizer is
>> in it's separate source file. I have included sqlite3.h header with
>> the tokenizer source but sqlite3.h does not contain the declaration of
>> the various structs like sqlite3_tokenizer_module etc. So what is the
>> usual way to resolve this ? These declarations are also not there in
>> sqlite3ext.h . Although I see them there in sqlite3.c but I am not
>> sure I want to include it ? What is the usual way to resolve this ?
>> May be import fts3_tokenizer.h from the sqlite3 source ?
>>
>
> Yes.  Use fts3_tokenizer.h.

Thanks, That worked. Now, I am able to compile everything.

Next, I am having problem with using this tokenizer. I followed the
exampple from the FTS3 documentation page on the website and
registered the tokenizer using code like this:

const sqlite3_tokenizer_module *stopword_tokenizer_module;

sqlstr = "select fts3_tokenizer(:tokenizer_name, :tokenizer_ptr)";
rc = sqlite3_prepare_v2(db, sqlstr, -1, , NULL);
if (rc != SQLITE_OK) {
sqlite3_close(db);
sqlite3_shutdown();
return -1;
}

idx = sqlite3_bind_parameter_index(stmt, ":tokenizer_name");
rc = sqlite3_bind_text(stmt, idx, "my_tokenizer", -1, NULL);
if (rc != SQLITE_OK) {
fprintf(stderr, "%s\n", sqlite3_errmsg(db));
sqlite3_finalize(stmt);
return -1;
}

sqlite3Fts3MyTokenizerModule((const sqlite3_tokenizer_module
**)_tokenizer_module);
idx = sqlite3_bind_parameter_index(stmt, ":tokenizer_ptr");
rc = sqlite3_bind_blob(stmt, idx, _tokenizer_module,
sizeof(my_tokenizer_module), SQLITE_STATIC);
if (rc != SQLITE_OK) {
fprintf(stderr, "%s\n", sqlite3_errmsg(db));
sqlite3_finalize(stmt);
return -1;
}
rc = sqlite3_step(stmt);
if (rc != SQLITE_ROW) {
fprintf(stderr, "%s\n", sqlite3_errmsg(db));
sqlite3_finalize(stmt);
return -1;
}
sqlite3_finalize(stmt);

It is working fine till above. After executing the above statements, I
try to create an FTS table using this custom tokenizer, which also
seem to be getting created. The problem is coming up when I try to
insert data in the table. A simple statement like "insert into
my_table values(...)" is giving out errors:

unknown tokenizer: my_tokenizer

I am sure I am missing something here, but don't know what ?

Thanks
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] [FTS3] Header to include for a custom tokenizer

[Richard Hipp]

On Sat, Jul 23, 2011 at 1:01 PM, Abhinav Upadhyay <
er.abhinav.upadh...@gmail.com> wrote:

> Hi,
>
>  I am using the Sqlite3 amalgamation. I am trying to register a custom
> tokenizer with sqlite for my FTS application. The custom tokenizer is
> in it's separate source file. I have included sqlite3.h header with
> the tokenizer source but sqlite3.h does not contain the declaration of
> the various structs like sqlite3_tokenizer_module etc. So what is the
> usual way to resolve this ? These declarations are also not there in
> sqlite3ext.h . Although I see them there in sqlite3.c but I am not
> sure I want to include it ? What is the usual way to resolve this ?
> May be import fts3_tokenizer.h from the sqlite3 source ?
>

Yes.  Use fts3_tokenizer.h.


>
> Thanks
> Abhinav
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users