Re: [sqlite] SQLite and Qt

2010-06-08 Thread Sam Carleton
On Tue, Jun 8, 2010 at 10:25 PM, Bill King  wrote:

>
> >
> >1. How do I control the version of SQLite used with Qt?
> >
> ./configure -system-sqlite will use the sqlite compiled for your system.
>

No, not how do I compile SQLite into Qt, I have done that.  I want to
control the VERSION.  I don't know if the distro of Qt is using the latest
version of SQLite or not.  Also, if I opt to NOT upgrade the Apache server,
I would like Qt to use the same version of SQLite that the Apache server is
using ;)

Sam
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] SQLite and Qt

2010-06-08 Thread Bill King
On 06/09/2010 11:52 AM, ext Sam Carleton wrote:
> I have been using SQLite for a few years now in an Apache server module and
> it works great for me.  The system is expanding, I am working on a Qt based
> GUI program that needs to access the SQLite database.  (The Qt program is
> only going to run on the same machine as the Apache Server.)
>
> If I am not mistaken, the version of SQLite used by the Apache server is
> 3.5.???.  I would like to start taking advantage of the foreign key feature
> of v3.6, which is my question:
>
>1. How do I control the version of SQLite used with Qt?
>   
./configure -system-sqlite will use the sqlite compiled for your system.
>2. How do I get the version of SQLite from both the native API and from
>Qt?
>3. How do I set the foreign key pragma in Qt?
>   
Just using QSqlQuery's exec function should work fine to exec "PRAGMA
foreign_keys = ON"
> Sam
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>   


-- 
Bill King, Software Engineer
Qt Development Frameworks, Nokia Pty Ltd
Brisbane Office

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] 3.6.23 segmentation fault with trigger and DEFAULT VALUES

2010-06-08 Thread 陶渊俊
> Actually it's found in version 3.6.23 and fixed in the trunk of SQLite

> repository but SQLite didn't have a release since then. So this bug

> will be fixed in the next version.

 

*  Pavel

 

I got it. Thank you very much indeed.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] SQLite and Qt

2010-06-08 Thread Sam Carleton
I have been using SQLite for a few years now in an Apache server module and
it works great for me.  The system is expanding, I am working on a Qt based
GUI program that needs to access the SQLite database.  (The Qt program is
only going to run on the same machine as the Apache Server.)

If I am not mistaken, the version of SQLite used by the Apache server is
3.5.???.  I would like to start taking advantage of the foreign key feature
of v3.6, which is my question:

   1. How do I control the version of SQLite used with Qt?
   2. How do I get the version of SQLite from both the native API and from
   Qt?
   3. How do I set the foreign key pragma in Qt?

Sam
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] copy data from one db to another

2010-06-08 Thread Simon Slavin

On 8 Jun 2010, at 10:47pm, Scott Frankel wrote:

> On Jun 8, 2010, at 2:28 PM, Simon Slavin wrote:
> 
>> Either write a program to read record-by-record and write record-by- 
>> record, or use PostgreSQL functions to write to SQL commands then  
>> execute those commands to create a new SQLite database.
> 
> Before heading down the path of issuing record-by-record statements, I  
> thought there might be some hope with generic SQL commands or bulk  
> importing ... a newbie's hope ;)

SQLite includes a command-line tool which can be used to dump a database to a 
text file of SQL commands, or read the text-file and use it to create a new 
database.  If PostgreSQL has an equivalent tool, you can do it all without 
writing a line of code.  Better still, you'll have the text file which will act 
as an excellent backup of your data in case something goes wrong.  (You may 
have to use a text editor to make minor changes to the format of the SQL 
commands.)

When I backup any SQL database for long-term archive purposes I always back it 
up as SQL commands, never in its native file format.  This means I don't have 
to worry about trying to find some software to read an obsolete file format.  
(Assuming that there will be software which can read a .zip file for the 
foreseeable future.)

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] copy data from one db to another

2010-06-08 Thread Scott Frankel

On Jun 8, 2010, at 2:28 PM, Simon Slavin wrote:

>
> On 8 Jun 2010, at 9:02pm, Scott Frankel wrote:
>
>> On Jun 8, 2010, at 12:51 PM, Jean-Christophe Deschamps wrote:
>>
 What's the best way to copy data from one db to another?

 Given 2 databases with identical schemas, one full of data and the
 other empty, the brute force way would be to perform selects on the
 source db, then for each row, perform an insert into the  
 destination
 db.  Is there a more efficient way?
>>>
>>> The easiest is either to simply copy the file as Igor suggested or  
>>> use
>>> the backup API (very easy too).
>>
>> I forgot to mention, the source is a PostgreSQL db, not SQLite, so
>> there's no source file to copy.
>
> Then you have to use the functions of one library to read your data  
> and the functions of another to write your data.  There's no way to  
> do it inside one SQL command since the SQLite library can't read a  
> PostgreSQL database and the PostgreSQL library can't write a SQLite  
> database.

Right.


> Either write a program to read record-by-record and write record-by- 
> record, or use PostgreSQL functions to write to SQL commands then  
> execute those commands to create a new SQLite database.

Before heading down the path of issuing record-by-record statements, I  
thought there might be some hope with generic SQL commands or bulk  
importing ... a newbie's hope ;)

Thanks for the info!
Scott



>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] copy data from one db to another

2010-06-08 Thread Simon Slavin

On 8 Jun 2010, at 9:02pm, Scott Frankel wrote:

> On Jun 8, 2010, at 12:51 PM, Jean-Christophe Deschamps wrote:
> 
>>> What's the best way to copy data from one db to another?
>>> 
>>> Given 2 databases with identical schemas, one full of data and the
>>> other empty, the brute force way would be to perform selects on the
>>> source db, then for each row, perform an insert into the destination
>>> db.  Is there a more efficient way?
>> 
>> The easiest is either to simply copy the file as Igor suggested or use
>> the backup API (very easy too).
> 
> I forgot to mention, the source is a PostgreSQL db, not SQLite, so  
> there's no source file to copy.

Then you have to use the functions of one library to read your data and the 
functions of another to write your data.  There's no way to do it inside one 
SQL command since the SQLite library can't read a PostgreSQL database and the 
PostgreSQL library can't write a SQLite database.

Either write a program to read record-by-record and write record-by-record, or 
use PostgreSQL functions to write to SQL commands then execute those commands 
to create a new SQLite database.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] copy data from one db to another

2010-06-08 Thread Adam DeVita
The db that you open your initial connection to is called main by default. I
haven't had the occasion to use a temp or memory db so I can't comment.

The attach statement works as normal SQL.
attach 'path to your db' as 'some_alias_name'

like
attach 'c:\temp dir\db2.db' as 'db2'

Suppose both files have a table named 'some_table'.

select * from db2.some_table ; /*refers to the attached db*/
select * from main.some_table ; /*refers to the db you first made a
connection with.*/

I find the command line tool wonderful for testing out syntax.

Happy Computing

Adam


On Tue, Jun 8, 2010 at 4:09 PM, Scott Frankel  wrote:

>
> On Jun 8, 2010, at 12:07 PM, Adam DeVita wrote:
>
> > start by doing an open db1 (as main)
> > then attach path to db2 as 'db2'
> >
> > insert into main.table_one_name select * from db2.table_one_name ;
> >
> > This selects all records from db2 and puts them into db1 in one
> > statement.
>
> I've been reading about the ATTACH DATABASE cmd, but was confused by
> the documentation's warnings about main and temp dbs, namings, and
> transactions using :memory:.  I'll take a closer look.
>
> Thanks!
> Scott
>
>
>
>
> >
> > Adam
> > On Tue, Jun 8, 2010 at 3:02 PM, Scott Frankel
> >  wrote:
> >
> >>
> >> Hi all,
> >>
> >> What's the best way to copy data from one db to another?
> >>
> >> Given 2 databases with identical schemas, one full of data and the
> >> other empty, the brute force way would be to perform selects on the
> >> source db, then for each row, perform an insert into the destination
> >> db.  Is there a more efficient way?
> >>
> >> Thanks in advance!
> >> Scott
> >>
> >>
> >> ___
> >> sqlite-users mailing list
> >> sqlite-users@sqlite.org
> >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >>
> >
> >
> >
> > --
> > VerifEye Technologies Inc.
> > 905-948-0015x245
> > 7100 Warden Ave, Unit 3
> > Markham ON, L3R 8B5
> > Canada
> > ___
> > sqlite-users mailing list
> > sqlite-users@sqlite.org
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
VerifEye Technologies Inc.
905-948-0015x245
7100 Warden Ave, Unit 3
Markham ON, L3R 8B5
Canada
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] copy data from one db to another

2010-06-08 Thread Scott Frankel

On Jun 8, 2010, at 12:07 PM, Adam DeVita wrote:

> start by doing an open db1 (as main)
> then attach path to db2 as 'db2'
>
> insert into main.table_one_name select * from db2.table_one_name ;
>
> This selects all records from db2 and puts them into db1 in one  
> statement.

I've been reading about the ATTACH DATABASE cmd, but was confused by  
the documentation's warnings about main and temp dbs, namings, and  
transactions using :memory:.  I'll take a closer look.

Thanks!
Scott




>
> Adam
> On Tue, Jun 8, 2010 at 3:02 PM, Scott Frankel  
>  wrote:
>
>>
>> Hi all,
>>
>> What's the best way to copy data from one db to another?
>>
>> Given 2 databases with identical schemas, one full of data and the
>> other empty, the brute force way would be to perform selects on the
>> source db, then for each row, perform an insert into the destination
>> db.  Is there a more efficient way?
>>
>> Thanks in advance!
>> Scott
>>
>>
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>
>
>
> -- 
> VerifEye Technologies Inc.
> 905-948-0015x245
> 7100 Warden Ave, Unit 3
> Markham ON, L3R 8B5
> Canada
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] copy data from one db to another

2010-06-08 Thread Scott Frankel

On Jun 8, 2010, at 12:51 PM, Jean-Christophe Deschamps wrote:

>
>> What's the best way to copy data from one db to another?
>>
>> Given 2 databases with identical schemas, one full of data and the
>> other empty, the brute force way would be to perform selects on the
>> source db, then for each row, perform an insert into the destination
>> db.  Is there a more efficient way?
>
> The easiest is either to simply copy the file as Igor suggested or use
> the backup API (very easy too).

I forgot to mention, the source is a PostgreSQL db, not SQLite, so  
there's no source file to copy.  Though a backup might be  
interesting ...



>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] copy data from one db to another

2010-06-08 Thread Jean-Christophe Deschamps

>What's the best way to copy data from one db to another?
>
>Given 2 databases with identical schemas, one full of data and the
>other empty, the brute force way would be to perform selects on the
>source db, then for each row, perform an insert into the destination
>db.  Is there a more efficient way?

The easiest is either to simply copy the file as Igor suggested or use 
the backup API (very easy too).


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] copy data from one db to another

2010-06-08 Thread Igor Tandetnik
Scott Frankel  wrote:
> What's the best way to copy data from one db to another?
> 
> Given 2 databases with identical schemas, one full of data and the
> other empty

Why not just copy the whole file over?
-- 
Igor Tandetnik


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] copy data from one db to another

2010-06-08 Thread Adam DeVita
start by doing an open db1 (as main)
then attach path to db2 as 'db2'

insert into main.table_one_name select * from db2.table_one_name ;

This selects all records from db2 and puts them into db1 in one statement.

Adam
On Tue, Jun 8, 2010 at 3:02 PM, Scott Frankel  wrote:

>
> Hi all,
>
> What's the best way to copy data from one db to another?
>
> Given 2 databases with identical schemas, one full of data and the
> other empty, the brute force way would be to perform selects on the
> source db, then for each row, perform an insert into the destination
> db.  Is there a more efficient way?
>
> Thanks in advance!
> Scott
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
VerifEye Technologies Inc.
905-948-0015x245
7100 Warden Ave, Unit 3
Markham ON, L3R 8B5
Canada
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] database corruption problem

2010-06-08 Thread Dan Kennedy

On Jun 9, 2010, at 12:51 AM, Dave Segleau wrote:

>
> On 6/8/2010 9:25 AM, Dan Kennedy wrote:
>>
>> Those pragmas should not cause a problem. Simon's referring
>> to "PRAGMA synchronous". The docs for which explain the
>> risks assumed by changing the default setting.
>>
>>http://www.sqlite.org/pragma.html#pragma_synchronous
>>
>>
> As Dan said, "PRAGMA synchronous" can cause database corruption under
> certain conditions of application or system failure.

System failure. Not application failure.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] copy data from one db to another

2010-06-08 Thread Scott Frankel

Hi all,

What's the best way to copy data from one db to another?

Given 2 databases with identical schemas, one full of data and the  
other empty, the brute force way would be to perform selects on the  
source db, then for each row, perform an insert into the destination  
db.  Is there a more efficient way?

Thanks in advance!
Scott


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] database corruption problem

2010-06-08 Thread Dave Segleau

On 6/8/2010 9:25 AM, Dan Kennedy wrote:
>
> Those pragmas should not cause a problem. Simon's referring
> to "PRAGMA synchronous". The docs for which explain the
> risks assumed by changing the default setting.
>
> http://www.sqlite.org/pragma.html#pragma_synchronous
>   
>
As Dan said, "PRAGMA synchronous" can cause database corruption under 
certain conditions of application or system failure.

Just as an FYI, "PRAGMA synchronous" will not cause database corruption 
in the latest Oracle Berkeley DB 11gR2 release that supports the SQLite 
API. The Berkeley DB write-ahead logging and recovery will ensure that 
the database remains consistent even in the event of application or 
system failure.

Regards,

Dave
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] database corruption problem

2010-06-08 Thread Simon Slavin

On 8 Jun 2010, at 5:25pm, Dan Kennedy wrote:

> Those pragmas should not cause a problem. Simon's referring
> to "PRAGMA synchronous". The docs for which explain the
> risks assumed by changing the default setting.
> 
>   http://www.sqlite.org/pragma.html#pragma_synchronous

You're both right.  I didn't bother to look up Dan's precise combination of 
PRAGMAs to see if the documentation contained any warnings.  But I did remember 
that some PRAGMAs can cause corruption under some circumstances.  However, Dan 
has already said he has read the 'how to corrupt your database' text which 
discusses everything he should be worrying about.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] database corruption problem

2010-06-08 Thread Dan Kennedy

On Jun 8, 2010, at 10:54 PM, Daniel Stutzbach wrote:

> On Tue, Jun 8, 2010 at 10:02 AM, Simon Slavin   
> wrote:
>
>> I'm not sure whether a power-cut at a particularly bad time could  
>> cause
>> something like this.  The journaling mechanism built into SQLite  
>> should be
>> avoiding it, but your combination of PRAGMAs might be defeating the  
>> normal
>> defence mechanism.
>>
>
> If my combination of pragmas defeat the defense mechanisms, that  
> would be an
> SQLite bug, wouldn't it?  The documentation for them certainly does  
> not
> suggest that there would be any loss in reliability.  Can anyone  
> confirm or
> deny that they might be causing a problem?
>
> I started using PRAGMA locking_mode = EXCLUSIVE and PRAGMA  
> journal_mode =
> TRUNCATE because certain aggressive virus-scanners/system-software  
> lock
> files when they're scanning them.  With those pragmas, I can lock the
> database and journal once at the beginning and not worry about getting
> "unable to open database file" errors later.

Those pragmas should not cause a problem. Simon's referring
to "PRAGMA synchronous". The docs for which explain the
risks assumed by changing the default setting.

   http://www.sqlite.org/pragma.html#pragma_synchronous


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] database corruption problem

2010-06-08 Thread Daniel Stutzbach
On Tue, Jun 8, 2010 at 10:02 AM, Simon Slavin  wrote:

> I'm not sure whether a power-cut at a particularly bad time could cause
> something like this.  The journaling mechanism built into SQLite should be
> avoiding it, but your combination of PRAGMAs might be defeating the normal
> defence mechanism.
>

If my combination of pragmas defeat the defense mechanisms, that would be an
SQLite bug, wouldn't it?  The documentation for them certainly does not
suggest that there would be any loss in reliability.  Can anyone confirm or
deny that they might be causing a problem?

I started using PRAGMA locking_mode = EXCLUSIVE and PRAGMA journal_mode =
TRUNCATE because certain aggressive virus-scanners/system-software lock
files when they're scanning them.  With those pragmas, I can lock the
database and journal once at the beginning and not worry about getting
"unable to open database file" errors later.
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] database corruption problem

2010-06-08 Thread Pavel Ivanov
> Yes.  It's quite possible to have corruption which shows only when you hit a 
> particular row while using a particular index.  The corruption can hide in 
> the file while you carry on adding rows or searching using other indices and 
> come to light only under some weird circumstance in the future.

It's even possible to have a corruption which wouldn't come to you as
an error but just as a successful selection of malformed data. It
happens when for some reason number of rows in the index is different
from number of rows in the table. So in this case you will notice
corruption only when you try to work with that malformed data.


Pavel

On Tue, Jun 8, 2010 at 11:02 AM, Simon Slavin  wrote:
>
> On 8 Jun 2010, at 3:39pm, Daniel Stutzbach wrote:
>
>> After talking with my user a bit more, he reported that he had a power
>> outage around a week prior to the error.  Is it possible that some part of
>> the database became corrupt in a way that didn't immediately cause any
>> problems?
>
> Yes.  It's quite possible to have corruption which shows only when you hit a 
> particular row while using a particular index.  The corruption can hide in 
> the file while you carry on adding rows or searching using other indices and 
> come to light only under some weird circumstance in the future.
>
> I'm not sure whether a power-cut at a particularly bad time could cause 
> something like this.  The journaling mechanism built into SQLite should be 
> avoiding it, but your combination of PRAGMAs might be defeating the normal 
> defence mechanism.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] database corruption problem

2010-06-08 Thread Daniel Stutzbach
On Tue, Jun 8, 2010 at 6:32 AM, Nick Shaw  wrote:

> 1) File/Disk corruption by the OS. On Windows, this can happen if the
> system suffers a power loss in the middle of a write, most noticeable
> when you're writing a large amount of data to a file (I realise it
> shouldn't happen, but I've seen it a number of times, especially on
> embedded XP devices);
>

After talking with my user a bit more, he reported that he had a power
outage around a week prior to the error.  Is it possible that some part of
the database became corrupt in a way that didn't immediately cause any
problems?  (e.g., a block that was in use became marked as a free or
somesuch?)

My application processes data in the background when the user isn't actively
using the computer, and this particular user throws quite a lot of data at
my application.  There's better-than-average odds that my application was
writing to the database when the power went out.


> 3) Something in your app is corrupting the stack or memory used by
> SqLite. That can cause no end of problems, and is usually very hard to
> pin down unless you can reproduce it.
>

Although I can't rule memory corruption out entirely, I think it's pretty
unlikely.  If my application had a general memory-corruption bug, I would
expect to occasionally get crash reports.  (My application automatically
creates a minidump and launches a Crash Reporter application in the event of
an invalid memory access)

If my application had a memory-corruption bug that was specifically misusing
a pointer used to interface with SQLite, that might not ever result in an
invalid memory access crash.  However, the bug would have to be in Python's
sqlite3 module.  Since that's pretty widely used, that also seems unlikely.


> If this is the first report you've ever had on this, and you have a fair
> few systems installed out in the field, my bet would be on #1.
>

It's the first report I've had.  I have a decent number of installations,
although this user probably puts my application under more stress than any
other user at the moment.

Definitely sounds like #1 is the most likely culprit.  Given that I
apparently cannot trust the OS/hardware to do the right thing during a power
outage, are there any best-practices for backing up the database?

I could do an integrity check on the database then copy the database file.
 That way, I only erase the old backup when I know the primary is good.
 However, my application will basically be unresponsive during this time.

I see that there's an online backup API, but a power outage during the
backup might leave me with two corrupt databases instead of just one (if I
understand correctly).

Thanks everyone for your input,
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] database corruption problem

2010-06-08 Thread Simon Slavin

On 8 Jun 2010, at 3:39pm, Daniel Stutzbach wrote:

> After talking with my user a bit more, he reported that he had a power
> outage around a week prior to the error.  Is it possible that some part of
> the database became corrupt in a way that didn't immediately cause any
> problems?

Yes.  It's quite possible to have corruption which shows only when you hit a 
particular row while using a particular index.  The corruption can hide in the 
file while you carry on adding rows or searching using other indices and come 
to light only under some weird circumstance in the future.

I'm not sure whether a power-cut at a particularly bad time could cause 
something like this.  The journaling mechanism built into SQLite should be 
avoiding it, but your combination of PRAGMAs might be defeating the normal 
defence mechanism.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] 3.6.23 segmentation fault with trigger and DEFAULT VALUES

2010-06-08 Thread Pavel Ivanov
> Even though it had been fixed in 3.6.23, why I could reproduce it?

Actually it's found in version 3.6.23 and fixed in the trunk of SQLite
repository but SQLite didn't have a release since then. So this bug
will be fixed in the next version.


Pavel

2010/6/8 陶渊俊 :
> This is my first time to reply email of gmane-news.
> so if there is anything wrong, please forgive me.
>
>
>
>>Which operating system are you running ?
>
>
>
>>Are you running your code in the command-line program or with your own
> program ?
>
>
>
>>If you are running your own program, are you accessing SQLite with the C
> functions, or are you using a library
>
>>someone else made ?
>
>
>
>>Simon.
>
>
>
> My operating system is fedora10.
>
> Uname -a :
>
> Linux localhost.localdomain 2.6.22.18-co-0.7.5 #1 PREEMPT Mon Sep 14
> 22:21:15 UTC 2009 i686 i686 i386 GNU/Linux
>
>
>
> Yesterday, I only tested it in the command line.
> Today, I tested example program. The result was as same as command line.
> The program I used was a example:
>
> http://www.sqlite.org/quickstart.html
>
>
>
>>Information on this page:  http://www.sqlite.org/lang_createtrigger.html
>
>>Reads:
>
>>   The "INSERT INTO *table* DEFAULT VALUES" form of the
>>INSERTstatement is not
>>supported.
>
>>I do not think it is a problem/defect/bug. What say?
>
>>Harsha
>
> I had read this page, what the page said is that sqlite doesn`t support a
> "DEFAULT VALUES" form of the INSERT statement with a trigger.
>
>
>
>
>
>
> But I think even though sqlite doesn`t support it, the core dump is
> a fatal error, it will crash the totall program which is using sqlite.
>
>
> Whatever user do, at least there should be dump an error to the control
> or err-log like "DEFAULT VALUES with trigger is not supported by sqlite!"
>
>
> and let the main program continue rather than a core dump.
> So I think it is a bug.
>
>>  Known bug.  Fixed:
>
>>  http://www.sqlite.org/src/info/f3162063fd
>
>>-j
>
>>--
>>Jay A. Kreibich < J A Y  @  K R E I B I.C H >
>
> Yeah, my investigation is from the email:
>
>
> http://thread.gmane.org/gmane.comp.db.sqlite.general/56571
>
>
>
>
>
> So I had read the page what you provided.
>
>
> Even though it had been fixed in 3.6.23, why I could reproduce it?
>
>
> My sqlite version is 3.6.23.1, Need I some lately update?or patch?
>
> -bash-3.2# sqlite3 -version
> 3.6.23
> -bash-3.2#
>
> Thanks for all reply.
>
>
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] returning a larger snippet in FTS3 match

2010-06-08 Thread P Kishor
On Tue, Jun 8, 2010 at 9:14 AM, Jay A. Kreibich  wrote:
> On Tue, Jun 08, 2010 at 08:49:22AM -0500, P Kishor scratched on the wall:
>> On Tue, Jun 8, 2010 at 8:29 AM, Jay A. Kreibich  wrote:
>> > On Tue, Jun 08, 2010 at 06:56:33AM -0500, P Kishor scratched on the wall:
>> >> Hi all, re-asking this in case it missed some of the keener eyes -- I
>> >> am using the Snippet() function to return a snippet of text from my
>> >> FTS3 table showing the MATCH context. I would like to make the
>> >> returned snippet longer. Is that possible?
>> >
>> > ?Yes. ? ? http://sqlite.org/fts3.html#section_4_2
>
>> First, what the heck does '-15' mean?
>
>  Read the two paragraphs that start "Assuming N is a positive
>  value..." and "If N is a negative value...".
>
>> Second, specifying
>> anything more than 4 arguments gives me a SQLite error. For example,
>
>  Are you using the latest build?  It appears to have been updated from
>  an older version that only accepted four arguments.
>


ahhh... that could be it. I am using DBD::SQLite 1.29, which uses sqlite 3.6.22.

It would be nice if each documentation page included the version
number of sqlite for which that specific documentation was valid.
Otherwise, there is no knowing what works and what doesn't work.


-- 
Puneet Kishor
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] returning a larger snippet in FTS3 match

2010-06-08 Thread Jay A. Kreibich
On Tue, Jun 08, 2010 at 08:49:22AM -0500, P Kishor scratched on the wall:
> On Tue, Jun 8, 2010 at 8:29 AM, Jay A. Kreibich  wrote:
> > On Tue, Jun 08, 2010 at 06:56:33AM -0500, P Kishor scratched on the wall:
> >> Hi all, re-asking this in case it missed some of the keener eyes -- I
> >> am using the Snippet() function to return a snippet of text from my
> >> FTS3 table showing the MATCH context. I would like to make the
> >> returned snippet longer. Is that possible?
> >
> > ?Yes. ? ? http://sqlite.org/fts3.html#section_4_2

> First, what the heck does '-15' mean?

  Read the two paragraphs that start "Assuming N is a positive
  value..." and "If N is a negative value...".

> Second, specifying
> anything more than 4 arguments gives me a SQLite error. For example,

  Are you using the latest build?  It appears to have been updated from
  an older version that only accepted four arguments.

   -j

-- 
Jay A. Kreibich < J A Y  @  K R E I B I.C H >

"Intelligence is like underwear: it is important that you have it,
 but showing it to the wrong people has the tendency to make them
 feel uncomfortable." -- Angela Johnson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] returning a larger snippet in FTS3 match

2010-06-08 Thread P Kishor
On Tue, Jun 8, 2010 at 8:29 AM, Jay A. Kreibich  wrote:
> On Tue, Jun 08, 2010 at 06:56:33AM -0500, P Kishor scratched on the wall:
>> Hi all, re-asking this in case it missed some of the keener eyes -- I
>> am using the Snippet() function to return a snippet of text from my
>> FTS3 table showing the MATCH context. I would like to make the
>> returned snippet longer. Is that possible?
>
>  Yes.     http://sqlite.org/fts3.html#section_4_2
>
>


Yup, I read that, and I just can't get it. Per the docs

"The snippet function is used to create formatted fragments of
document text for display as part of a full-text query results report.
The snippet function may be passed between one and four arguments, as
follows:

ArgumentDefault Value   Description
0   N/A  The first argument to the snippet function must always be the
special hidden column of the FTS3 table that takes the same name as
the table itself.
1   ""The "start match" text.
2   ""   The "end match" text.
3   "..." The "ellipses" text.
4   -1   The FTS3 table column number to extract the returned fragments
of text from. Columns are numbered from left to right starting with
zero. A negative value indicates that the text may be extracted from
any column.
5   -15  The absolute value of this integer argument is used as the
(approximate) number of tokens to include in the returned text value.
The maximum allowable absolute value is 64. The value of this argument
is refered to as N in the discussion below."

So, first of all, the docs say that Snippet function takes between one
and four arguments, and then it lists 6 possible arguments. From what
I read above, it is the sixth argument, that is, argument 5, where I
can specify the "approximate" number of tokens in the returned text
value. First, what the heck does '-15' mean? Second, specifying
anything more than 4 arguments gives me a SQLite error. For example,

Snippet(fts_pages, '', '', '', -1, 64) AS context

gives the following error

wrong number of arguments to function snippet() at
/Users/punkish/Sites/punkish/_perl/Punkish.pm line 535,  line 36.,
referer: http://punkish.local/
[Tue Jun 08 08:44:54 2010] [error] [client 127.0.0.1] perl(3344)
malloc: *** error for object 0x100b7dc00: pointer being freed was not
allocated, referer: http://punkish.local/
[Tue Jun 08 08:44:54 2010] [error] [client 127.0.0.1] *** set a
breakpoint in malloc_error_break to debug,


What am I missing?



-- 
Puneet Kishor http://www.punkish.org
Carbon Model http://carbonmodel.org
Charter Member, Open Source Geospatial Foundation http://www.osgeo.org
Science Commons Fellow, http://sciencecommons.org/about/whoweare/kishor
Nelson Institute, UW-Madison http://www.nelson.wisc.edu
---
Assertions are politics; backing up assertions with evidence is science
===
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] database corruption problem

2010-06-08 Thread Daniel Stutzbach
On Tue, Jun 8, 2010 at 6:57 AM, Max Vlasov  wrote:

> You didn't mention your language/development tools. It would help also.
>

The application is written primarily in Python 2.6, using the sqlalchemy
package to talk to the database.

Parts of the program are written in C and loaded as DLLs using Python's
ctypes module.  Some of the DLLs are build using MINGW gcc and others are
built with Microsoft Visual C.

The user who experienced the problem is running Windows XP.


> I also suggest to implement artificial tests like constantly emulating user
> actions in your program. Once it helped me to find the cause of the same
> corruption error. It would be some dev-only menu item that activates a loop
> constantly reading randomly and writing randomly until specially
> interrupted.


The GUI part of my application actually lives in a completely separate
process.  It communicates with the backend over a socket, so automated
testing is straightforward and part of my development process.  Up until
now, I have not been looking for database corruption specifically however.


> While the program reads and writes you check PRAGMA
> integrity_check; from time to time and stop if result shows corruption.
> If you can't reproduce the problem on your machine you can send specially
> prepared version to the user and ask him to perform the similar actions


Thanks.  I will add the PRAGMA integrity_check to my test process.
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] database corruption problem

2010-06-08 Thread Jay A. Kreibich
On Tue, Jun 08, 2010 at 12:32:22PM +0100, Nick Shaw scratched on the wall:
> Daniel Stutzbach wrote:
> > Are there any other known ways that the database might become corrupt?
> 
> My list of possible causes would be:
> 1) File/Disk corruption by the OS. On Windows, this can happen if the
> system suffers a power loss in the middle of a write, most noticeable
> when you're writing a large amount of data to a file (I realise it
> shouldn't happen, but I've seen it a number of times, especially on
> embedded XP devices);

  In theory, the journal file should protect against this, but that
  assumes the system flushes and syncs its I/O correctly.

  I'm not sure I'd trust an embedded XP device to do that, especially
  on flash storage.

   -j

-- 
Jay A. Kreibich < J A Y  @  K R E I B I.C H >

"Intelligence is like underwear: it is important that you have it,
 but showing it to the wrong people has the tendency to make them
 feel uncomfortable." -- Angela Johnson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] returning a larger snippet in FTS3 match

2010-06-08 Thread Jay A. Kreibich
On Tue, Jun 08, 2010 at 06:56:33AM -0500, P Kishor scratched on the wall:
> Hi all, re-asking this in case it missed some of the keener eyes -- I
> am using the Snippet() function to return a snippet of text from my
> FTS3 table showing the MATCH context. I would like to make the
> returned snippet longer. Is that possible?

  Yes. http://sqlite.org/fts3.html#section_4_2


   -j

-- 
Jay A. Kreibich < J A Y  @  K R E I B I.C H >

"Intelligence is like underwear: it is important that you have it,
 but showing it to the wrong people has the tendency to make them
 feel uncomfortable." -- Angela Johnson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] database corruption problem

2010-06-08 Thread Max Vlasov
On Mon, Jun 7, 2010 at 7:31 PM, Daniel Stutzbach <
dan...@stutzbachenterprises.com> wrote:

> I'm the author of a small Windows application that embeds SQLite.  I'm
> currently using version 3.5.9 with PRAGMA locking_mode = EXCLUSIVE
> and PRAGMA journal_mode = TRUNCATE.  I just received a crash report from a
> user with the dreaded error: "database disk image is malformed".
>
> My application is multi-threaded, although only the main thread calls
> SQLite
> functions (the other threads do non-DB stuff).
>
>
You didn't mention your language/development tools. It would help also.

I also suggest to implement artificial tests like constantly emulating user
actions in your program. Once it helped me to find the cause of the same
corruption error. It would be some dev-only menu item that activates a loop
constantly reading randomly and writing randomly until specially
interrupted. While the program reads and writes you check PRAGMA
integrity_check; from time to time and stop if result shows corruption.
If you can't reproduce the problem on your machine you can send specially
prepared version to the user and ask him to perform the similar actions

Max Vlasov,
maxerist.net
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] returning a larger snippet in FTS3 match

2010-06-08 Thread P Kishor
Hi all, re-asking this in case it missed some of the keener eyes -- I
am using the Snippet() function to return a snippet of text from my
FTS3 table showing the MATCH context. I would like to make the
returned snippet longer. Is that possible?

On Sun, Jun 6, 2010 at 10:36 AM, P Kishor  wrote:
> I would like to return a much larger snippet in the MATCH results than
> what comes back as default. Is it possible to specify a snippet
> length?
>
> --
> Puneet Kishor
>



-- 
Puneet Kishor http://www.punkish.org
Carbon Model http://carbonmodel.org
Charter Member, Open Source Geospatial Foundation http://www.osgeo.org
Science Commons Fellow, http://sciencecommons.org/about/whoweare/kishor
Nelson Institute, UW-Madison http://www.nelson.wisc.edu
---
Assertions are politics; backing up assertions with evidence is science
===
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] database corruption problem

2010-06-08 Thread Nick Shaw
Daniel Stutzbach wrote:
> Are there any other known ways that the database might become corrupt?

My list of possible causes would be:
1) File/Disk corruption by the OS. On Windows, this can happen if the
system suffers a power loss in the middle of a write, most noticeable
when you're writing a large amount of data to a file (I realise it
shouldn't happen, but I've seen it a number of times, especially on
embedded XP devices);
2) File corruption by a separate app doing something naughty to the file
(e.g. a virus).
3) Something in your app is corrupting the stack or memory used by
SqLite. That can cause no end of problems, and is usually very hard to
pin down unless you can reproduce it.

If this is the first report you've ever had on this, and you have a fair
few systems installed out in the field, my bet would be on #1.

Thanks,
Nick.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] sql api for ufs

2010-06-08 Thread bch
On Mon, Jun 07, 2010 at 12:08:40PM -0700, George Georgalis wrote:
> On Mon 07 Jun 2010 at 05:07:43 PM +0200, Kees Nuyt wrote:
> >
> >That could be implemented as an appication with a set of
> >virtual tables, backed by the readdir() and stat() system
> >calls.
> >I haven't heard of any implementation, although fossil
> >http://www.fossil-scm.org/index.html/doc/tip/www/index.wiki
> >has a few routines which might be interesting.
> 
> Interesting approach. I've never heard of virtual tables.
> Unfortunately, I don't know anything about the backend of
> generating indices or optimizing db queries either. But I
> see what you are getting at, maybe I can work it out.
> 
> Fossil, looks interesting too. I'm adding it to my library of
> resources on this project.

NetBSD users interested in fossil may be interested to see the following links 
pertaining to fossil's relationship w/ NetBSD:

http://methodlogic.net/BSDFossil.html
http://methodlogic.net/FossilDiscussionInNetBSDIRCChannel.html
and fossil in pkgsrc @ pkgsrc/devel/fossil

> If it's not clear, I want to do development on top of such an
> application. The key goal here is to address data sprawl. Duplicate
> files on multiple (offline) disks, S3 like resources and no index
> for 5 year old (or 1 month) data. I'm looking to index, merge and
> purge duplicate data; and be able to opportunistically build indices
> and run transactions while media is transiently online.
> 
> So, something that barely works---builds indexes from fs data,
> supports rudimentary queries and generates filesystem manipulation
> commands is my first goal. I'll use it to discover duplicates and
> organize files. I'm sure this project will evolve considerably.
> 
> >I know Microsoft does (or tries to do) something similar,
> >perhaps MSDN is of any help?
> 
> The only microsoft system I have is a multimedia player. Not sure
> how I could roll their products into my developments
> 
> Thanks,
> Cheers,
> George

-- 
Brad Harder
Method Logic Digital Consulting
http://methodlogic.net
http://twitter.com/bcharder

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Oracle connection

2010-06-08 Thread Gary_Gabriel
Simon Hax wrote:
> for clearification:
>
> the roots of the question:
> business needs; our client has a big Oracle infrastructure.
> Now they want, in relation to their infrastructure a litte App based on 
> SQlite.
>
> as I mentioned earlier:
> to copy data from an Oracle DB to another Oracle DB there is just one single 
> line necessary:
>
> import into mytable select a,b,c from remotetable remoteOracleDB
>
> somewhere in my local OracleDB I have to define a bit earlier a database link 
> with IP-adress, Port, SID UserID and passwort concerning the remote OracleDB.
> That's it.
>
> Are am I to lazy to implement by my self ? :  Yes and No
>
> I have implemented it within C# and ADO.net.
> Then it's not only one line of code, which is easy to read an easy to 
> maintain.
>   
Moving outside of the Oracle environment increases complexity (work).
> It's a lot more:
> The basic concept is a loop, suprise suprise !
> But you have to take into account that you have to convert every single 
> datatype,
> (e.g. DATE, there is a different representation Sqlite versus Oracle)
> you have to look for character-set and so on and so on.
>
> My wish:
> If someone else already has done this work 
> Thatfore I think an ODBC driver has to be implemented (there I am not an 
> expert) ...
>
> and a database Link to Oracle should be defineable by ATTACH (IP-Adress:Port, 
> OracleSID, UserId, Password)
>
> that would be heaven on earth.
>   
Hi Simon,

"I have implemented it within C# and ADO.net.
Then it's not only one line of code, which is easy to read an easy to maintain."

It is not clear does this refer to the Oracle installation or to SQLite? 
- If it refers to SQLite why is it not sufficient?

There's an SQLite ODBC driver that is proven and Googling has shown that 
it has been applied in Oracle installations in the past:

Christian Werner: SQLite ODBC Driver 
. http://www.ch-werner.de/sqliteodbc/

Using the ODBC driver
- Query the SQLite app table for the data and put it into a temporary 
table. Use a temporary table or memory db to make the management and 
querying easier. Install the ODBC driver and you can query the SQLite 
database over the ODBC connection. Christian Werner's instructions are 
sufficient. Past Googling informs of satisfactory Oracle ODBC support. 
Query the SQLite db using ODBC as it is then an Oracle resource.
- You might want to start formating the data in this table during this 
session.
- Use the ODBC connection with the local Oracle(local) db and pass the 
data in the temp table to Oracle(local)
- The IP link between the local and remote Oracle dbs should be 
transparent and enable passing the (SQLite) data which has now become 
native. Between the Oracle(remote) and Oracle(local); it is an Oracle task.

Working a bit more conceptually use the SQLite C .csv extension.
- After compiling the extension; load it.
- Query the SQLite app table for the data and put it into a table 
generated by the extension. This extension reads and writes from a .csv 
formatted file.
- Writing the data to this (extension) table after creating the table 
puts the table data into the .csv file. Oracle(local) reads the .csv 
table as it would a spreadsheet.

To do it programmatically requires bindings for SQLite and Oracle, as 
Pavel pointed out.

Alternatively write the data in the temp table to a file and import the 
file data into Oracle(local)

- Gary Gabriel
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] database corruption problem

2010-06-08 Thread Daniel Stutzbach
I'm the author of a small Windows application that embeds SQLite.  I'm
currently using version 3.5.9 with PRAGMA locking_mode = EXCLUSIVE
and PRAGMA journal_mode = TRUNCATE.  I just received a crash report from a
user with the dreaded error: "database disk image is malformed".

My application is multi-threaded, although only the main thread calls SQLite
functions (the other threads do non-DB stuff).

I have read http://www.sqlite.org/lockingv3.html#how_to_corrupt, so I know
the database can become corrupt if someone deletes the journal, if hardware
is faulty, or the OS is faulty.

Are there any other known ways that the database might become corrupt?

Any suggestions for how I might pinpoint the cause of the corruption?  Is
there anything I can do to make my usage of the database more robust?
 (aside from making backups of the database, which I will now likely
implement)
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] last_insert_rowid & INTEGER PRIMARY KEY columns

2010-06-08 Thread Andy Gibbs
>> [...] does
>>
>> db last_insert_rowid
>>
>> reliably return the_key of the most recently inserted database row,
>> so that the returned value may safely (across vacuums etc) be used
>> as a foreign reference to t's the_key column?
>
> Yes.

Actually there *is* a caveat, which is that if the insert does not actually 
complete -- for example, you have a trigger on insert that could raise an 
'ignore' -- then the last_insert_rowid will still return the rowid of the 
last row that was actually inserted (as you would expect), but then if you 
blindly use this in a foreign key constraint without checking whether the 
row you wanted *was* inserted, you may find you reference the wrong row...

SQLite version 3.6.23.1
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> create table tab(i);
sqlite> create trigger trig before insert on tab when new.i=5 begin
   ...>   select raise(ignore);
   ...> end;
sqlite> insert into tab values(10);
sqlite> select last_insert_rowid();
1
sqlite> insert into tab values(20);
sqlite> select last_insert_rowid();
2
sqlite> insert into tab values(5);
sqlite> select last_insert_rowid();
2
sqlite>


Just a word of caution, although it may not apply to your situation...  ;o)

However, there are work-arounds to the above 'caveat'.

Rgds
Andy

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Is it possible to generate tf-idf matrix from the FTS3 table?

2010-06-08 Thread Scott Hess
On Sat, Jun 5, 2010 at 10:29 AM, Han-Teng Liao  wrote:
>    I intend to use my existing datasets stored in sqlite3 database for some
> linguistic analysis for Chinese language. After I have successfully
> installed and run the FTS3 Extension and ICU Extension, I am curious whether
> it is theoretically possible to generate the tf-idf matrix from the FTS3
> table?  If so, please do not hesitate to point me to the rough direction
> that I should take.

For each term the index encodes the documents containing the term and
the hits within the document, so it is certainly possible.  You might
look at adapting the testing function dumpDoclistFunc() in fts3.c.
The performance would be about the same as doing a search on that
term.  You wouldn't want to do this in an attempt to optimize queries,
because it already would touch all of the term's data.  But it might
make sense as a ranking input.  If you're working with a static data
set, use optimize() to consolidate the indices once you've loaded the
data.

-scott
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users