Re: [sqlite] DBD::SQLite

2006-03-16 Thread Cyril Scetbon
just a $dbh->do("insert into mytable(varint) values ('01234')"; It's not working correctly with SQLite but no problem with Oracle. Chris Werner a écrit : Can you give a code example? I have just tried, and can load string values with a leading 0 and m/^\d+$/ I suspect the problem is in your

Re: [sqlite] sqlite performance with sizeable tables

2006-03-16 Thread spaminos-sqlite
Sorry it took me some time to get back to this thread. - Original Message From: Christian Smith <[EMAIL PROTECTED]> > When your database does not fit in memory, yes, you're right, the OS may > well get caching wrong, and in the worst way possible. Two things though: > - SQLite does have

Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

2006-03-16 Thread John Stanton
Doug Nebeker wrote: It sounds to me that he isn't saying it is leaking--it simply isn't releasing memory after a SELECT statement finishes (is finalized). This might be by design. I would expect the data to be released if the database connection is closed, but not necessarily after each

Re: [sqlite] Re: concers about database size

2006-03-16 Thread John Stanton
Jay Sprenkle wrote: On 3/16/06, Daniel Franke <[EMAIL PROTECTED]> wrote: The original idea was to get rid of thousands of files to store their data in one single container. Those (ASCII) files add up to approx 5GB ... If so, are you trying to use a blender to stir the ocean? You might

[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

2006-03-16 Thread [EMAIL PROTECTED]
Hi, Thank you very much for your help, i have modify the line, it compiles, it doesn´t fire any exception, but memory is not freed at all (works the same way) Anyone has another idea?, i tought that SQLite was working since a long time ago on Windows CE devices, it seems quite strange that no

Re: [sqlite] Re: concers about database size

2006-03-16 Thread Carl Jacobs
Daniel > The most common scenarios: > - get a single marker from subset of individuals > - get a subset of markers from a single individual > - get a subset of markers from a subset of individuals Sounds like this might define your database. Each individual has 500,000 markers, but maybe they

[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

2006-03-16 Thread [EMAIL PROTECTED]
Hi Robert, I was talking about 3 selects satements using the same connections. Anyway, thank you very much for your advice of using PRAGMA cache size=8, that solved all the problems related to sqlite3_exec memory problems with a select statement, baut the memory problems are not solved at all,

[sqlite] DBD::SQLite

2006-03-16 Thread Cyril Scetbon
Hi, I use DBD::SQLite for accessing a SQLite database, but there's an issue when I tyr to insert a number starting with a 0. In fact, DBD::SQLite seems to trim the starting 0 of the number. So, when I insert 0234 in a table I find 234 instead. Anyone has encoutered and resolved this bug ?

Re: [sqlite] Re: concers about database size

2006-03-16 Thread Jay Sprenkle
On 3/16/06, Daniel Franke <[EMAIL PROTECTED]> wrote: > > > That would be an excellent question to add to the FAQ: > > "How do I estimate the resource requirements for a database?" > > I spent some time to create 3GB of sample data (just zeros, about half the > size of the actual data set I have to

Re: [sqlite] Re: concers about database size

2006-03-16 Thread Daniel Franke
> That would be an excellent question to add to the FAQ: > "How do I estimate the resource requirements for a database?" I spent some time to create 3GB of sample data (just zeros, about half the size of the actual data set I have to deal with). I'm currently importing it into the database. As

Re: [sqlite] Re: concers about database size

2006-03-16 Thread Jay Sprenkle
> Why would it require a lot of RAM? I ask not because I doubt you, but > because my intuition says that a BTree based database should scale > pretty well. While certainly it would run faster if you can fit the > whole thing in RAM, if the index can be made to fit in RAM it seems > like the data

Re: [sqlite] Re: concers about database size

2006-03-16 Thread Nathan Kurz
On Thu, Mar 16, 2006 at 11:34:31AM -0600, Jay Sprenkle wrote: > > > You may legitimately need one really large table but most applications > > > don't. > > Too bad. My guess is that you're doing the right thing trying to consolidate. > It's going to take expensive hardware no matter what you end

RE: [sqlite] Re: concers about database size

2006-03-16 Thread Fred Williams
> -Original Message- > From: Jay Sprenkle [mailto:[EMAIL PROTECTED] > Sent: Thursday, March 16, 2006 10:15 AM > To: sqlite-users@sqlite.org > Subject: Re: [sqlite] Re: concers about database size > > > > Does sound like an awful lot of data. I think the question might be > > reworded to

Re: [sqlite] Re: concers about database size

2006-03-16 Thread Jay Sprenkle
> > You may legitimately need one really large table but most applications > > don't. > > The most common scenarios: > - get a single marker from subset of individuals > - get a subset of markers from a single individual > - get a subset of markers from a subset of individuals > > There is no

Re: [sqlite] concers about database size

2006-03-16 Thread Kervin L. Pierre
Hello Daniel, Daniel Franke wrote: Was sqlite designed for those numbers? The docs state that sqlite supports "databases up to 2 terabytes in size". OTOH, "supports" is not the same as "works-well-with"?! Any suggestions whether my descision to use sqlite was appropiate for this table design?

RE: [sqlite] File locking additions

2006-03-16 Thread CARTER-HITCHIN, David, FM
Hi, > I definitely like all the things you've proposed. We currently use > SQLite over NFS (to a NetApp) and while it mostly works, we do > occasionally get locking issues that I'd love to be able to > get rid of. Do you run Solaris or Linux? Cheers, David Carter-Hitchin. -- Royal Bank of

Re: [sqlite] SQLite_Busy from DBD-SQLite

2006-03-16 Thread Sripathi Raj
I'm using Perl 5.8.7 on Windows. When the database is locked, the statement execution returns a FALSE value like it does for any other error. On 3/16/06, Jay Sprenkle <[EMAIL PROTECTED]> wrote: > > On 3/16/06, Sripathi Raj <[EMAIL PROTECTED]> wrote: > > Hi, > > > > How do I know that a statement

Re: [sqlite] Re: concers about database size

2006-03-16 Thread Daniel Franke
> How do you use your data? Do you really need to compare all the information > about any and all individuals? > [...] > You may legitimately need one really large table but most applications > don't. The most common scenarios: - get a single marker from subset of individuals - get a subset of

Re: [sqlite] File locking additions

2006-03-16 Thread Matt Sergeant
On 7-Mar-06, at 7:06 PM, Adam Swift wrote: In order to provide locking support for database files mounted from remote file systems (NFS, SMB, AFP, etc) as well as provide compatible locking support between database clients both local and remote, I would like to propose some additions to

Re: [sqlite] Re: concers about database size

2006-03-16 Thread Jay Sprenkle
> The table in question: > CREATE TABLE genotypes (markerid integer NOT NULL REFERENCES marker(id), > individualid integer NOT NULL REFERENCES > individuals(id), > genA integer, > genB integer); > > I don't see how to segment

[sqlite] Re: concers about database size

2006-03-16 Thread Daniel Franke
Note: this is a combined reply to answers sent by Fred and Jay. Fred wrote: > > > If so, are you trying to use a blender to stir the ocean? > > > You might reevaluate if you're using the right tool for the job. > > > > That's my question: IS sqlite the right tool here? =) > > And I believe he is

[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE

2006-03-16 Thread [EMAIL PROTECTED]
Hi Again Robert, I have run your program on the CE emulator (Pocket PC 2003) and i got the same memory leak. I have inserted 2 buttons on a MFC dialog application. The first button executes your code and the second button closes the application. If you examine the memory you will discover that

Re: [sqlite] Re: concers about database size

2006-03-16 Thread Jay Sprenkle
On 3/16/06, Daniel Franke <[EMAIL PROTECTED]> wrote: > Each single file contains detailed genotypic information of many > individuals at a given genomic region. We have to implement _loads_ of > quality control measures to ensure the maximum possible data > correctness. Earlier, we did this

Re: [sqlite] Re: concers about database size

2006-03-16 Thread Jay Sprenkle
> Does sound like an awful lot of data. I think the question might be > reworded to ask is there any manageable logical groups of data that > might lend themselves to simple segmentation into separate > tables/databases? I thought about asking that as well, but wondered if there wasn't a simpler

[sqlite] Re: concers about database size

2006-03-16 Thread Daniel Franke
> > > You might reevaluate if you're using the right tool for the job. > > That's my question: IS sqlite the right tool here? =) > Then I guess the right question is what are your goals? To make > maintenance easier? > Why were the thousands of files a problem? Short answer: I want to improve -

RE: [sqlite] Re: concers about database size

2006-03-16 Thread Fred Williams
> -Original Message- > From: Daniel Franke [mailto:[EMAIL PROTECTED] > Sent: Thursday, March 16, 2006 9:32 AM > To: sqlite-users@sqlite.org > Subject: [sqlite] Re: concers about database size > > > > > But now, there's another thing.I figured out how large my database > > > will become

RE: [sqlite] DBD::SQLite SQLite Ver 3.2.7

2006-03-16 Thread Chris Werner
Title: RE: [sqlite] DBD::SQLite SQLite Ver 3.2.7 Steve, Jarl, Thanks for the replies. Eric thanks for the info. So this needs to be a patch in the DBD-SQLite-1.11 module? Something like the attached patch I assume? Matt Seargeant, How do we get this into the CPAN archives? Thank you,

Re: [sqlite] Re: concers about database size

2006-03-16 Thread Jay Sprenkle
On 3/16/06, Daniel Franke <[EMAIL PROTECTED]> wrote: > The original idea was to get rid of thousands of files to store their data > in one single container. Those (ASCII) files add up to approx 5GB ... > > > If so, are you trying to use a blender to stir the ocean? > > You might reevaluate if

[sqlite] Re: concers about database size

2006-03-16 Thread Daniel Franke
> > But now, there's another thing.I figured out how large my database > > will become and I'm scared of its size: up to 20GB and more! A single > > table, 4 columns, each holding an integer (32 bit) will have > > approximately 750 million rows. This mounts up to ~11GB. Adding an > > unique

Re: [sqlite] concers about database size

2006-03-16 Thread Jay Sprenkle
On 3/16/06, Daniel Franke <[EMAIL PROTECTED]> wrote: > Hi all ... > a couple of days back, I had a question about the performance of large > tables. You really got me going there, thanks again! > > But now, there's another thing.I figured out how large my database > will become and I'm scared of

[sqlite] concers about database size

2006-03-16 Thread Daniel Franke
Hi all ... a couple of days back, I had a question about the performance of large tables. You really got me going there, thanks again! But now, there's another thing.I figured out how large my database will become and I'm scared of its size: up to 20GB and more! A single table, 4 columns, each

Re: [sqlite] SQLITE_CORRUPT problem in Mac OS X

2006-03-16 Thread drh
"Srinivasan Thandapani" <[EMAIL PROTECTED]> wrote: > Hi, > > In my Mac OS X, I am linking sqlite(3.2.8) library statically to my C++ > program. > > I have defined macros > -DNO_TCL -DHAVE_USLEEP -DTEMP_STORE=3 > -DSQLITE_MAX_PAGE_SIZE=32768 -DTHREADSAFE=1 > in Makefile which creates the sqlite

Re: [sqlite] SQLite_Busy from DBD-SQLite

2006-03-16 Thread Jay Sprenkle
On 3/16/06, Sripathi Raj <[EMAIL PROTECTED]> wrote: > Hi, > > How do I know that a statement failed to execute because the database was > locked? > > Raj > Raj, what language are you using?

[sqlite] SQLITE_CORRUPT problem in Mac OS X

2006-03-16 Thread Srinivasan Thandapani
Hi, In my Mac OS X, I am linking sqlite(3.2.8) library statically to my C++ program. I have defined macros -DNO_TCL -DHAVE_USLEEP -DTEMP_STORE=3 -DSQLITE_MAX_PAGE_SIZE=32768 -DTHREADSAFE=1 in Makefile which creates the sqlite library. But often I am getting SQLITE_CORRUT error. My program

RE: [sqlite] Help needed to diagnose "NOT IN" query

2006-03-16 Thread CARTER-HITCHIN, David, FM
Hello Ulrich, > I don't know wether it's faster, but try > > select a,b,c from d where c in (select c from d except select > c from e) Sorry to say but this made no difference :-( I'll try my method next... Thanks for your help, David Carter-Hitchin. -- Royal Bank of Scotland Interest Rate

[sqlite] SQLite_Busy from DBD-SQLite

2006-03-16 Thread Sripathi Raj
Hi, How do I know that a statement failed to execute because the database was locked? Raj