Re: [sqlite] FTS3: synonyms dictionary and tokens length

2011-07-15 Thread Alexey Pechnikov 
2011/7/15 Dan Kennedy :
>> But queries to original text will not work:
>> select text from fts where fts match 'sqlite educate';
>
> I think it will. Query strings - like 'sqlite educate' - are
> also parsed using the tokenizer. So the query will be transformed
> to 'dbms educate' before it is run.

I wanted to say - we can't rank results differently. The query 'sqlite educate'
may return result 'dbms education' with low rank and 'sqlite education'
with high rank.

-- 
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] FTS3: synonyms dictionary and tokens length

2011-07-15 Thread Dan Kennedy 
On 07/15/2011 01:10 PM, Alexey Pechnikov wrote:
> 2011/7/15 Dan Kennedy:
>> I think you could just have the tokenizer return "dbms" whenever
>> it sees "sqlite" in the input.
>
> But queries to original text will not work:
> select text from fts where fts match 'sqlite educate';

I think it will. Query strings - like 'sqlite educate' - are
also parsed using the tokenizer. So the query will be transformed
to 'dbms educate' before it is run.



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] FTS3: synonyms dictionary and tokens length

2011-07-15 Thread Alexey Pechnikov 
2011/7/15 Dan Kennedy :
> I think you could just have the tokenizer return "dbms" whenever
> it sees "sqlite" in the input.

But queries to original text will not work:
select text from fts where fts match 'sqlite educate';

You can see synonyms dictionary in PostgreSQL full-text search
as example. It's very common task for many applications.

-- 
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] FTS3: synonyms dictionary and tokens length

2011-07-14 Thread Dan Kennedy 
On 07/15/2011 03:28 AM, Alexey Pechnikov wrote:
> I want to add the table CREATE TABLE ext_fts_synonyms(word text not
> null unique, synonym text not null); insert into
> ext_fts_synonyms('sqlite','sqlite dbms');
>
> And replace in tokenizer the term 'sqlite' to 2 terms 'sqlite' and
> 'dbms' for search by queries like to
>> select text from fts where fts match 'dbms educate';
> 'SQLite may be useful for education'
>
> But how to return from tokenizer 2 terms or more instead of single
> term?..

I think you could just have the tokenizer return "dbms" whenever
it sees "sqlite" in the input.

Both documents and tokens that appear in queries are transformed
by tokenizers.

Dan.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] FTS3: synonyms dictionary and tokens length

2011-07-14 Thread Alexey Pechnikov 
I want to add the table
CREATE TABLE ext_fts_synonyms(word text not null unique, synonym text not null);
insert into ext_fts_synonyms('sqlite','sqlite dbms');

And replace in tokenizer the term 'sqlite' to 2 terms 'sqlite' and 'dbms'
for search by queries like to
> select text from fts where fts match 'dbms educate';
'SQLite may be useful for education'

But how to return from tokenizer 2 terms or more instead of single term?..

Note: Snowball stemmer and stopwords table I did add and can do:
> select text from fts where fts match 'sqlite educate'
SQLite may be useful for education


-- 
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] FTS3: synonyms dictionary and tokens length

2011-07-14 Thread Dan Kennedy 
On 07/14/2011 07:42 PM, Alexey Pechnikov wrote:
>> No. Don't return anything for a stop word. Just advance to the next
>> non stop-word token and return it.
>
> Thanks, I did and it's work.
>
> And another question... Is there any way to use multi-word synonyms? Like to:
> sqlite ->  Open Source SQLite DBMS
>
> I think the single token "Open Source SQLite DBMS" will not useful.

I don't quite follow. Can you rephrase the question?


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] FTS3: synonyms dictionary and tokens length

2011-07-14 Thread Alexey Pechnikov 
> No. Don't return anything for a stop word. Just advance to the next
> non stop-word token and return it.

Thanks, I did and it's work.

And another question... Is there any way to use multi-word synonyms? Like to:
sqlite -> Open Source SQLite DBMS

I think the single token "Open Source SQLite DBMS" will not useful.

-- 
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] FTS3: synonyms dictionary and tokens length

2011-07-14 Thread Dan Kennedy 
On 07/14/2011 05:29 PM, Alexey Pechnikov wrote:
> With 0-length token in icuNext there is the error:
> Error: SQL logic error or missing database
>
> May xNext returns 0 length when the token is stopword?

No. Don't return anything for a stop word. Just advance to the next
non stop-word token and return it.


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] FTS3: synonyms dictionary and tokens length

2011-07-14 Thread Alexey Pechnikov 
With 0-length token in icuNext there is the error:
Error: SQL logic error or missing database

May xNext returns 0 length when the token is stopword?

-- 
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] FTS3: synonyms dictionary and tokens length

2011-07-13 Thread Dan Kennedy 
On 07/13/2011 05:05 PM, Alexey Pechnikov wrote:
> With synonyms dictionary the result token length can be more then
> original token length.
> Is it problem for current realization of FTS?

I don't think so. If it is, it's a bug.

Dan.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] FTS3: synonyms dictionary and tokens length

2011-07-13 Thread Alexey Pechnikov 
With synonyms dictionary the result token length can be more then
original token length.
Is it problem for current realization of FTS?

-- 
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users