[sqlite] memory issue with large amount of inserts
Hi, I am using SQLite 3.5.4, my application has encountered a memory management issue which I think is caused by sqlite. I have searched the forum and found this thread describing most similar situation as mine http://www.nabble.com/Memory-Usage-to13798003.html#a13850915 My application is accessing sqlite through JAVA on local machine, windows xp sp2, there are 4 threads accessing 4 *different* databases at the same time, each doing 2 million inserts to *different* databases. I have noticed that after about 3 hours, the system slowed down dramatically, such that processing every 1000 records(then 1000 inserts) took now 20 mins - 1 hr instead of sever seconds at the very beginning. I then checked the resource monitor and found all of my 2 gb memory has been used. I searched this forum and found talks about similar issues but solutions seem to be re-compiling sqlite in C using new parameters, but I am using JAVA and would like to know whether there is alternative/equivalent solution? And also the above thread I quote mentioned that closing/reopen connection can free up resources that sqlite occupy, i would take this as a simple but dirty workaround? Please may I have you suggestions , many thanks! -- View this message in context: http://www.nabble.com/memory-issue-with-large-amount-of-inserts-tp15225948p15225948.html Sent from the SQLite mailing list archive at Nabble.com. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Solaris bus error
Dan
I thought the test was from async3.test... But I'm probably wrong.
I got the info from ddd and asked it to tell me the contents of ppLock which
was 0.
Here is the output directly from gdb as you requested.
print *ppLock
$1 = (struct lockInfo *) 0x0
gdb) info threads
4 process 262671 0xfef9fa58 in _lwp_sema_wait () from /usr/lib/libc.so.1
3 process 197135 0xfef9cf4c in door_restart () from /usr/lib/libc.so.1
2 process 131599 0xfef9f3a8 in _signotifywait () from /usr/lib/libc.so.1
* 1 process 66063 0x00049750 in findLockInfo (fd=4, ppLock=0x1345ad,
ppOpen=0x1345a9) at ../sqliteSrc/sqlite-3.5.5/src/os_unix.c:670
Let me know if you need anything else.
Dan <[EMAIL PROTECTED]> wrote:
On Feb 1, 2008, at 6:37 AM, Ken wrote:
> Recompiled without the -O2 flags:
>
> #0 0x00049750 in findLockInfo (fd=4, ppLock=0x1345ad,
> ppOpen=0x1345a9)
> at ../sqliteSrc/sqlite-3.5.5/src/os_unix.c:670
> 670 *ppLock = pLock;
>
> *ppLock is Null
Hi Ken,
This doesn't look quite right to me. In the debugger you can
do "print *ppLock" and it doesn't complain?
The test case you were running (async.test) uses more than one
thread, so maybe the one we're looking at is not the one that
caused the problem.
Can you try running [info threads] in the debugger, and then
see if you can get a stack trace for another thread? Thanks.
Dan.
>
> Ken wrote: Core/backtrace info:
>
>
> (gdb) core-file core
> Core was generated by `/home/ixion/LIB/sqlite3test/.libs/
> testfixture ../sqliteSrc/sqlite-3.5.5/test/qu'.
> Program terminated with signal 10, Bus error.
> Reading symbols from /home/ixion/LIB/sqlite3test/.libs/
> libsqlite3.so.0...done.
> Loaded symbols for /home/ixion/LIB/sqlite3test/.libs/libsqlite3.so.0
> Reading symbols from /usr/lib/libpthread.so.1...done.
> Loaded symbols for /usr/lib/libpthread.so.1
> Reading symbols from /usr/local/lib/libtcl8.5.so...done.
> Loaded symbols for /usr/local/lib/libtcl8.5.so
> Reading symbols from /usr/lib/libdl.so.1...done.
> Loaded symbols for /usr/lib/libdl.so.1
> Reading symbols from /usr/lib/libsocket.so.1...done.
> Loaded symbols for /usr/lib/libsocket.so.1
> Reading symbols from /usr/lib/libnsl.so.1...done.
> Loaded symbols for /usr/lib/libnsl.so.1
> Reading symbols from /usr/lib/libm.so.1...done.
> Loaded symbols for /usr/lib/libm.so.1
> Reading symbols from /usr/lib/librt.so.1...done.
> Loaded symbols for /usr/lib/librt.so.1
> Reading symbols from /usr/lib/libc.so.1...done.
> Loaded symbols for /usr/lib/libc.so.1
> Reading symbols from /usr/local/lib/libgcc_s.so.1...done.
> Loaded symbols for /usr/local/lib/libgcc_s.so.1
> Reading symbols from /usr/lib/libmp.so.2...done.
> Loaded symbols for /usr/lib/libmp.so.2
> Reading symbols from /usr/lib/libaio.so.1...done.
> Loaded symbols for /usr/lib/libaio.so.1
> Reading symbols from /usr/platform/SUNW,Ultra-5_10/lib/libc_psr.so.
> 1...done.
> Loaded symbols for /usr/platform/SUNW,Ultra-5_10/lib/libc_psr.so.1
> Reading symbols from /usr/lib/libthread.so.1...done.
> Loaded symbols for /usr/lib/libthread.so.1
> #0 0x000345d4 in findLockInfo (fd=567296, ppLock=0xd1995,
> ppOpen=0xd1991)
> at ../sqliteSrc/sqlite-3.5.5/src/os_unix.c:671
> 671 if( ppOpen!=0 ){
> (gdb) backtrace
> #0 0x000345d4 in findLockInfo (fd=567296, ppLock=0xd1995,
> ppOpen=0xd1991)
> at ../sqliteSrc/sqlite-3.5.5/src/os_unix.c:671
> #1 0x000355b4 in unixOpen (pVfs=0x8a96c,
> zPath=0xc0380 "/home/ixion/LIB/sqlite3test/chocolate/banana/
> vanilla/file.db", pFile=0xd196d, flags=262, pOutFlags=0x)
> at ../sqliteSrc/sqlite-3.5.5/src/os_unix.c:2268
> #2 0x000340bc in sqlite3OsOpen (pVfs=0x2102,
> zPath=0xc0380 "/home/ixion/LIB/sqlite3test/chocolate/banana/
> vanilla/file.db", pFile=0xd196d, flags=262, pFlagsOut=0x)
> at ../sqliteSrc/sqlite-3.5.5/src/os.c:112
>
> ppOpen os_unxi.c/671 is not !=0 but *ppOpen is 0...
>
>
> James Dennett wrote: > -Original Message-
>> From: Ken [mailto:[EMAIL PROTECTED]
>> Sent: Thursday, January 31, 2008 2:55 PM
>> To: sqlite
>> Subject: [sqlite] Solaris bus error
>>
>>
>> After addressing the -lrt sched yield on solaris.
>>
>> make test resulted in:
>>
>> async3-1.0... Ok
>> async3-1.1...make: *** [test] Bus Error (core dumped)
>>
>> Any ideas?
>
> Getting a stack trace out of that core file with a debugger would seem
> to be the next step, and seeing the full text output if there is any
> more.
>
> -- James
>
>
>
>
> James Dennett wrote: > -Original Message-
>> From: Ken [mailto:[EMAIL PROTECTED]
>> Sent: Thursday, January 31, 2008 2:55 PM
>> To: sqlite
>> Subject: [sqlite] Solaris bus error
>>
>>
>> After addressing the -lrt sched yield on solaris.
>>
>> make test resulted in:
>>
>> async3-1.0... Ok
>> async3-1.1...make: *** [test] Bus Error (core dumped)
>>
>> Any ideas?
>
> Getting a stack trace out of that core file with a debugger would seem
> to be the next step, and seeing the full text output if there is any
>
Re: [sqlite] memory issue with large amount of inserts
I guess it's not as simple as that you're application is hogging the Java VM so much that the garbage collector is never running? You can always force it to run explicitly if so... /Jonas On Fri, Feb 1, 2008 at 1:46 PM, zqzuk <[EMAIL PROTECTED]> wrote: > > Hi, I am using SQLite 3.5.4, my application has encountered a memory > management issue which I think is caused by sqlite. I have searched the > forum and found this thread describing most similar situation as mine > > http://www.nabble.com/Memory-Usage-to13798003.html#a13850915 > > My application is accessing sqlite through JAVA on local machine, windows xp > sp2, there are 4 threads accessing 4 *different* databases at the same time, > each doing 2 million inserts to *different* databases. > > I have noticed that after about 3 hours, the system slowed down > dramatically, such that processing every 1000 records(then 1000 inserts) > took now 20 mins - 1 hr instead of sever seconds at the very beginning. I > then checked the resource monitor and found all of my 2 gb memory has been > used. > > I searched this forum and found talks about similar issues but solutions > seem to be re-compiling sqlite in C using new parameters, but I am using > JAVA and would like to know whether there is alternative/equivalent > solution? And also the above thread I quote mentioned that closing/reopen > connection can free up resources that sqlite occupy, i would take this as a > simple but dirty workaround? > > > Please may I have you suggestions , many thanks! > > -- > View this message in context: > http://www.nabble.com/memory-issue-with-large-amount-of-inserts-tp15225948p15225948.html > Sent from the SQLite mailing list archive at Nabble.com. > > > - > To unsubscribe, send email to [EMAIL PROTECTED] > - > > - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] memory issue with large amount of inserts
just to add that I am using transaction, and the batch size is 2000 zqzuk wrote: > > Hi, I am using SQLite 3.5.4, my application has encountered a memory > management issue which I think is caused by sqlite. I have searched the > forum and found this thread describing most similar situation as mine > > http://www.nabble.com/Memory-Usage-to13798003.html#a13850915 > > My application is accessing sqlite through JAVA on local machine, windows > xp sp2, there are 4 threads accessing 4 *different* databases at the same > time, each doing 2 million inserts to *different* databases. > > I have noticed that after about 3 hours, the system slowed down > dramatically, such that processing every 1000 records(then 1000 inserts) > took now 20 mins - 1 hr instead of sever seconds at the very beginning. I > then checked the resource monitor and found all of my 2 gb memory has been > used. > > I searched this forum and found talks about similar issues but solutions > seem to be re-compiling sqlite in C using new parameters, but I am using > JAVA and would like to know whether there is alternative/equivalent > solution? And also the above thread I quote mentioned that closing/reopen > connection can free up resources that sqlite occupy, i would take this as > a simple but dirty workaround? > > > Please may I have you suggestions , many thanks! > > -- View this message in context: http://www.nabble.com/memory-issue-with-large-amount-of-inserts-tp15225948p15226252.html Sent from the SQLite mailing list archive at Nabble.com. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] SQLite 3.5.5 on OpenBSD running on HPPA (HP9000) make test
Rob Sciuk <[EMAIL PROTECTED]> wrote: > Dear DRH, > > I post the previous (FreeBSD/Ubuntu) and this *ONLY* to exercise the new > 3.5.5 bits given the wholsale changes. I hope you find them useful, but > AFAICT, 3.5.5 looks pretty good from a perf/stability standpoint. > > In the obscure machine/OS category, I lit this up on an HP9000 K200 > running not HP-UX, but rather OpenBSD 4.2 (heh heh). Build was > uneventful, and testfixture ran to completion with the following output. > We do not understand the async3 problem and cannot reproduce it. All the other issues have been fixed in CVS HEAD. -- D. Richard Hipp <[EMAIL PROTECTED]> - To unsubscribe, send email to [EMAIL PROTECTED] -
[sqlite] SQLite 3.5.5 on OpenBSD running on HPPA (HP9000) make test
Dear DRH,
I post the previous (FreeBSD/Ubuntu) and this *ONLY* to exercise the new
3.5.5 bits given the wholsale changes. I hope you find them useful, but
AFAICT, 3.5.5 looks pretty good from a perf/stability standpoint.
In the obscure machine/OS category, I lit this up on an HP9000 K200
running not HP-UX, but rather OpenBSD 4.2 (heh heh). Build was
uneventful, and testfixture ran to completion with the following output.
$ uname -a
OpenBSD hppa.controlq.com 4.2 xyzzy#0 hppa
$ make test | grep -v Ok | tee test.output.HPPA
./testfixture ../test/quick.test
async3-1.0...
Error: error deleting "chocolate/banana/vanilla/file.db": permission denied
async3-1.1...
Error: table abc already exists
async3-1.2.2...
Expected: [1 {database is locked}]
Got: [1 {attempt to write a readonly database}]
async3-1.3.2...
Expected: [1 {database is locked}]
Got: [1 {attempt to write a readonly database}]
async3-1.4.2...
Expected: [1 {database is locked}]
Got: [1 {attempt to write a readonly database}]
async3-1.5.2...
Expected: [1 {database is locked}]
Got: [1 {attempt to write a readonly database}]
async3-1.6.2...
Expected: [1 {database is locked}]
Got: [1 {attempt to write a readonly database}]
async3-1.7.2...
Expected: [1 {database is locked}]
Got: [1 {attempt to write a readonly database}]
bind-4.4...
Error: floating point value is Not a Number
bind-4.5...
Expected: [null real real]
Got: []
btree-2.1.1...CURSOR 485C3688 rooted at1(rw) currently at 1.0 eof
PAGE 1: flags=0x09 frag=0 parent=0
cell 0: i=913..1023 chld=0nk=2020 nd=1792 payload=*** This is a very
PAGE 2: flags=0x08 frag=0 parent=0
PAGE 1 addr=42205894 nRef=2
Skipping crash5 tests: not compiled with -DSQLITE_MEMDEBUG...
lock4-1.3...
Error: database is locked
Skipping malloc5 tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping malloc6 tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping malloc7 tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping malloc8 tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping malloc9 tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocA tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocB tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocC tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocD tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocE tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocF tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocG tests: not compiled with -DSQLITE_MEMDEBUG...
printf-1.7.6...
Expected: [Three integers: (100) ( f4240) (3641100)]
Got: [Three integers: ( 100) ( f4240) (3641100)]
printf-1.8.6...
Expected: [Three integers: (9) (3b9ac9ff) (7346544777)]
Got: [Three integers: ( 9) (3b9ac9ff) (7346544777)]
printf-1.9.7...
Expected: [Three integers: ( 0) ( 0x0) ( 0)]
Got: [Three integers: ( 0) ( 0) ( 0)]
Skipping safety tests since SQLITE_DEBUG is off
time with cache: 38383126 microseconds per iteration
time without cache: 145730273 microseconds per iteration
result=one 10 three 5 two 14
speed3-1.incrvacuum.. 4612032 uS216.82417 row/s
speed3-1.normal... 9631095 uS103.83035 row/s
Incrvacuum: Read 1098, wrote 94
Normal: Read 10479, wrote 43
speed3-2.incrvacuum.. 1803619 uS554.44082 row/s
speed3-2.normal... 3585545 uS278.89763 row/s
Incrvacuum: Read 2843, wrote 1
Normal: Read 10011, wrote 0
tcl-1.6...
Expected: [1 {syntax error in expression "x*"}]
Got: [1 {invalid bareword "x"
in expression "x*";
should be "$x" or "{x}" or "x(...)" or ...}]
vtab6-2.2...
Expected: [1 2 3 {} 2 3 4 1 3 4 5 2]
Got: [1 2 3 {} 2 3 4 {} 3 4 5 {}]
vtab6-2.4...
Expected: [1 2 3 {} {} {} 2 3 4 {} {} {} 3 4 5 1 2 3]
Got: [1 2 3 {} {} {} 2 3 4 {} {} {} 3 4 5 {} {} {}]
vtab6-2.5...
Expected: [2 3 4 {} {} {} 3 4 5 1 2 3]
Got: [2 3 4 {} {} {} 3 4 5 {} {} {}]
vtab6-2.6...
Expected: [1 2 3 {} {} {} 2 3 4 {} {} {}]
Got: [1 2 3 {} {} {} 2 3 4 {} {} {} 3 4 5 {} {} {}]
vtab6-7.1...
Expected: [1 999 999 2 131 130 999]
Got: [1 999 999 999 999 999 999]
vtab6-9.1.1...
Expected: []
Got: [2 22 {}]
vtab6-9.2...
Expected: []
Got: [2 22 {}]
22 errors out of 38217 tests
Failures on these tests: async3-1.0 async3-1.1 async3-1.2.2 async3-1.3.2
async3-1.4.2 async3-1.5.2 async3-1.6.2 async3-1.7.2 bind-4.4 bind-4.5 lock4-1.3
printf-1.7.6 printf-1.8.6 printf-1.9.7 tcl-1.6 vtab6-2.2 vtab6-2.4 vtab6-2.5
vtab6-2.6 vtab6-7.1 vtab6-9.1.1 vtab6-9.2
All memory allocations freed - no leaks
Maximum memory usage: 14162418 bytes
Current memory usage: 0 bytes
*** Error code 1
Stop in /u0/sw/SQLite/sqlite-3.5.5/OpenBSD (line 568 of Makefile).
-
To unsubscribe, send email to [EMAIL PROTECTED]
-
RE: [sqlite] Using LIKE to check the first digits?
If you need LIKE 'abc%' and for it to not be case insensitive, then LIKE is preferred. Also, LIKE is standard SQL so if you're writing SQL that has to run across different vendors, you'd use LIKE. Sam --- We're Hiring! Seeking a passionate developer to join our team building Flex based products. Position is in the Washington D.C. metro area. If interested contact [EMAIL PROTECTED] -Original Message- From: P Kishor [mailto:[EMAIL PROTECTED] Sent: Thursday, January 31, 2008 9:28 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Using LIKE to check the first digits? if GLOB is so good, I am curious what are the instances in which one would prefer col LIKE '123%' over col GLOB '123*' - To unsubscribe, send email to [EMAIL PROTECTED] - - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] SQLite 3.5.5 on OpenBSD running on HPPA (HP9000) make test
On Fri, 1 Feb 2008, [EMAIL PROTECTED] wrote: We do not understand the async3 problem and cannot reproduce it. All the other issues have been fixed in CVS HEAD. Actually, upon inspection, that may be an artifact of an earlier attempt to run the tests as root, the "chocolate" directory was, in fact owned by root. I'm re-running the tests, having removed that directory, and I'll repost the results upon completion ... Cheers, Rob. - To unsubscribe, send email to [EMAIL PROTECTED] -
[sqlite] 3.5.5. Bus error on HPUX
I think this is probably a very simiilar issue as found on the solaris.
The error occurred during asyn3.test.
Here is the gdb output
analyze-99.1... Ok
async3-1.0... Ok
async3-1.1...make: *** [test] Bus error (core dumped)
hermes:herm10:/home/ixion/LIB/sqliteSrc/tst>> ls core
core
hermes:herm10:/home/ixion/LIB/sqliteSrc/tst>> gdb .libs/testfixture
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "hppa2.0w-hp-hpux11.11"...
(gdb) core-file core
Core was generated by `testfixture'.
Program terminated with signal 10, Bus error.
warning: The shared libraries were not privately mapped; setting a
breakpoint in a shared library will not work until you rerun the program.
Reading symbols from /usr/lib/dld.sl...done.
Reading symbols from /home/ixion/LIB/sqliteSrc/tst/.libs/libsqlite3.sl.8...done.
Reading symbols from /usr/local/lib/libtcl8.5.sl...done.
Reading symbols from /usr/lib/libpthread.1...done.
Reading symbols from /usr/lib/libxnet.2...done.
Reading symbols from /usr/lib/libxti.2...done.
Reading symbols from /usr/lib/libm.2...done.
Reading symbols from /usr/lib/libc.2...done.
Reading symbols from /usr/lib/libdld.2...done.
#0 0x00031984 in sqlite3MinimumFileFormat ()
at ../sqlite-3.5.5/src/os_unix.c:2400
2400 memset(pFile, 0, sizeof(unixFile));
(gdb) info threads
* 3 system thread 29226 0x00031984 in sqlite3MinimumFileFormat ()
at ../sqlite-3.5.5/src/os_unix.c:2400
2 system thread 29237 0x in ?? ()
1 system thread 29234 0xc0055560 in __pthread_start ()
from /usr/lib/libpthread.1
(gdb) print pFile
$1 = (sqlite3_file *) 0x4005787f
(gdb) print *pFile
$2 = {pMethods = 0x0}
(gdb)
Re: [sqlite] memory issue with large amount of inserts
I read pieces of information from other threads but I got more confused... I think I d better ask this question first: how exactly does sqlite manage memory usage, when does it attemp to release memory? Does page size/ number of pages affect insert operations? Does transactional-insert lock up memory and never release them? Because from my application it seems that as more inserts have been done in batches, memory consuption is increasing... I have tried close and reopen the connection but with no help. Any suggestions please, many thanks! zqzuk wrote: > > Hi, I am using SQLite 3.5.4, my application has encountered a memory > management issue which I think is caused by sqlite. I have searched the > forum and found this thread describing most similar situation as mine > > http://www.nabble.com/Memory-Usage-to13798003.html#a13850915 > > My application is accessing sqlite through JAVA on local machine, windows > xp sp2, there are 4 threads accessing 4 *different* databases at the same > time, each doing 2 million inserts to *different* databases. > > I have noticed that after about 3 hours, the system slowed down > dramatically, such that processing every 1000 records(then 1000 inserts) > took now 20 mins - 1 hr instead of sever seconds at the very beginning. I > then checked the resource monitor and found all of my 2 gb memory has been > used. > > I searched this forum and found talks about similar issues but solutions > seem to be re-compiling sqlite in C using new parameters, but I am using > JAVA and would like to know whether there is alternative/equivalent > solution? And also the above thread I quote mentioned that closing/reopen > connection can free up resources that sqlite occupy, i would take this as > a simple but dirty workaround? > > > Please may I have you suggestions , many thanks! > > -- View this message in context: http://www.nabble.com/memory-issue-with-large-amount-of-inserts-tp15225948p15231834.html Sent from the SQLite mailing list archive at Nabble.com. - To unsubscribe, send email to [EMAIL PROTECTED] -
RE: [sqlite] Using LIKE to check the first digits?
Regarding: >>Indices won't help with LIKE unless the column as a NOCASE collation. Use GLOB instead: >> ... WHERE number GLOB '1234*'; >>Note that "*" is the wildcard character with GLOB, not "%" >>as in LIKE. The above will use an index on the number column if it is available. === Thus, Dr. Hipp answers that age-old question, "What's not to like?" This email and any attachments have been scanned for known viruses using multiple scanners. We believe that this email and any attachments are virus free, however the recipient must take full responsibility for virus checking. This email message is intended for the named recipient only. It may be privileged and/or confidential. If you are not the named recipient of this email please notify us immediately and do not copy it or use it for any purpose, nor disclose its contents to any other person. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] db4o 500x faster than sqlite?
gerpux wrote: > > I've heard that the guys at db4o said that, under certain > circunstances, db4o is 500x faster than sqlite: > Is this because of the jdbc driver? > What would be a more realistic measure? (db4o is an object database, > not a relational one) > They are using the poleposition benchmark > (http://polepos.sourceforge.net). > Anybody knows if this benchmark is accurate for sqlite? Hi. "db4o OVER 500 TIMES FASTER THAN SQLITE" http://www.db4o.com/about/news/newsletter/2007_11.aspx Hm, interesting...Let's take a closer look. What they are comparing is called Barcelona Circuit: "writes, reads, queries and deletes objects with a 5 level inheritance structure" http://polepos.sourceforge.net/results/html/barcelona_write.html 500x times? On 3 objects and 100 iterations db4o took 1.3 seconds and SQLite 710 seconds. Do you believe in correct test? So they have B0-B4 classes inherited from each other where B0 is a base one for all. Each has 1 integer field. B2 has an index on the field. WRITE: Creating N objects of class B4 and setting all their fields to values from 0 to N. Commiting all N objects together. READ: Select all B4 objects at once. QUERY: Get count of all objects. Find all B4 objects where B2.i = x. x is rising from 0 to N. Everything in 1 transaction. DELETE: Delete all B4 objects in 1 transaction. The evil is in details as we know... First of all - settings: DB4O no options provided SQLITE: sqlite.autocommit=true sqlite.executebatch=false Now take a look at the code (some run options seems like missing): 1. db4o - sync probably off (at least it's definetly not flushing every transaction). 2. sqlite - sync 100% on. 3. db4o - wrapped with transactions (don't forget #1 - that means it's not even flushing transactions) 4. sqlite - no transactions at all - each insert/update/delete is 100% flushed separately 5. db4o - uses denormalized table - 1 table with 1 binary field with object data + 1 index 6. sqlite - 5 normalized tables with 1 integer field each (+5 own indexes on parentid and no indexes on item id??) + 1 index and separate insert/delete + select makes 5 joins always 7. db4o uses internal cache of queries 8. sqlite prepared statements thrown after each cycle 9. db4o - uses 1 command for every step 10.sqlite statements executed one by one 11.db4o - uses fast cached count 12.sqlite - counts objects on each cycle 13.db4o - caches everything in memory and not using pages (reading chunks) 14.sqlite - small page size and small cache. And don't forget, JDBC driver of SQLite makes additional overhead in contrast. So I will do own tests to see what's the real difference. I'll try .NET provider of SQLite and db4o.NET. And also I'll do the test on the DB size more than available RAM - 100K objects not serious... And will post back later with full code here and DB4O forum. My prediction? I'll do it A LOT faster on SQLite than db4o. -- View this message in context: http://www.nabble.com/db4o-500x-faster-than-sqlite--tp13484012p15231844.html Sent from the SQLite mailing list archive at Nabble.com. - To unsubscribe, send email to [EMAIL PROTECTED] -
[sqlite] SQLite 3.5.5 make test on OpenBSD 4.2 on HPPA
rerun, with chocolate directory removed prior to test run.
./testfixture ../test/quick.test
bind-4.4...
Error: floating point value is Not a Number
bind-4.5...
Expected: [null real real]
Got: []
btree-2.1.1...CURSOR 4041F588 rooted at1(rw) currently at 1.0 eof
PAGE 1: flags=0x09 frag=0 parent=0
cell 0: i=913..1023 chld=0nk=2020 nd=1792 payload=*** This is a very
PAGE 2: flags=0x08 frag=0 parent=0
PAGE 1 addr=4370A094 nRef=2
Skipping crash5 tests: not compiled with -DSQLITE_MEMDEBUG...
lock4-1.3...
Error: database is locked
Skipping malloc5 tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping malloc6 tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping malloc7 tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping malloc8 tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping malloc9 tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocA tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocB tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocC tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocD tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocE tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocF tests: not compiled with -DSQLITE_MEMDEBUG...
Skipping mallocG tests: not compiled with -DSQLITE_MEMDEBUG...
printf-1.7.6...
Expected: [Three integers: (100) ( f4240) (3641100)]
Got: [Three integers: ( 100) ( f4240) (3641100)]
printf-1.8.6...
Expected: [Three integers: (9) (3b9ac9ff) (7346544777)]
Got: [Three integers: ( 9) (3b9ac9ff) (7346544777)]
printf-1.9.7...
Expected: [Three integers: ( 0) ( 0x0) ( 0)]
Got: [Three integers: ( 0) ( 0) ( 0)]
Skipping safety tests since SQLITE_DEBUG is off
time with cache: 39849317 microseconds per iteration
time without cache: 144188993 microseconds per iteration
result=one 10 three 5 two 14
speed3-1.incrvacuum.. 4612508 uS216.80179 row/s
speed3-1.normal... 9956014 uS100.44180 row/s
Incrvacuum: Read 1098, wrote 94
Normal: Read 10479, wrote 43
speed3-2.incrvacuum.. 1801945 uS554.95589 row/s
speed3-2.normal... 3599141 uS277.84407 row/s
Incrvacuum: Read 2843, wrote 1
Normal: Read 10011, wrote 0
tcl-1.6...
Expected: [1 {syntax error in expression "x*"}]
Got: [1 {invalid bareword "x"
in expression "x*";
should be "$x" or "{x}" or "x(...)" or ...}]
vtab6-2.2...
Expected: [1 2 3 {} 2 3 4 1 3 4 5 2]
Got: [1 2 3 {} 2 3 4 {} 3 4 5 {}]
vtab6-2.4...
Expected: [1 2 3 {} {} {} 2 3 4 {} {} {} 3 4 5 1 2 3]
Got: [1 2 3 {} {} {} 2 3 4 {} {} {} 3 4 5 {} {} {}]
vtab6-2.5...
Expected: [2 3 4 {} {} {} 3 4 5 1 2 3]
Got: [2 3 4 {} {} {} 3 4 5 {} {} {}]
vtab6-2.6...
Expected: [1 2 3 {} {} {} 2 3 4 {} {} {}]
Got: [1 2 3 {} {} {} 2 3 4 {} {} {} 3 4 5 {} {} {}]
vtab6-7.1...
Expected: [1 999 999 2 131 130 999]
Got: [1 999 999 999 999 999 999]
vtab6-9.1.1...
Expected: []
Got: [2 22 {}]
vtab6-9.2...
Expected: []
Got: [2 22 {}]
14 errors out of 38178 tests
Failures on these tests: bind-4.4 bind-4.5 lock4-1.3 printf-1.7.6 printf-1.8.6
printf-1.9.7 tcl-1.6 vtab6-2.2 vtab6-2.4 vtab6-2.5 vtab6-2.6 vtab6-7.1
vtab6-9.1.1 vtab6-9.2
All memory allocations freed - no leaks
Maximum memory usage: 14162418 bytes
Current memory usage: 0 bytes
*** Error code 1
Stop in /u0/sw/SQLite/sqlite-3.5.5/OpenBSD (line 568 of Makefile).
-
To unsubscribe, send email to [EMAIL PROTECTED]
-
Re: [sqlite] Precompiled binaries for Windows Not Version 3.5.5
I see also SQLite 5.5.4 version compiled, is there SQLite 3.5.5 precompiled version for M$ Windows or if there is not drh. please compile it :) On Feb 1, 2008 12:15 AM, Robert Wishlaw <[EMAIL PROTECTED]> wrote: > Under the Precompiled binaries for Windows section of the downloads > page, the sqlite-3.5.5.zip, sqlitedll-3.5.5.zip and > tclsqlite-3.5.5.zip files contain 3.5.4 binaries from December 12, > 2007. > > Robert Wishlaw > > > - > To unsubscribe, send email to [EMAIL PROTECTED] > > - > >
Re: [sqlite] Precompiled binaries for Windows Not Version 3.5.5
"Zarko Popovski" <[EMAIL PROTECTED]> wrote: > I see also SQLite 5.5.4 version compiled, is there SQLite 3.5.5 precompiled > version for M$ Windows or if there is not drh. please compile it :) > I fixed that problem this morning. When did you last look? -- D. Richard Hipp <[EMAIL PROTECTED]> - To unsubscribe, send email to [EMAIL PROTECTED] -
