[sqlite] SQLite crashes

[Zsbán Ambrus]

On the SQLITE_OMIT_LOOKASIDE macro.

On Fri, Nov 27, 2015 at 2:45 PM, Richard Hipp  wrote:
> The lookaside memory allocator is a fast memory pool used by
> individual database connections for short-lived memory allocations.
> It makes SQLite run faster, but by bypassing the system
> malloc()/free() it can mask some types of heap corruption.  Best to
> turn it off when dealing with suspected heap problems.

This is useful to know.  Can you put it to the documentation under
http://sqlite.org/compile.html#omit_lookaside please?

-- Ambrus

[sqlite] SQLite crashes

[Harmen de Jong - CoachR Group B.V.]

> (1) Recompile with the following compile-time options:  -DSQLITE_DEBUG 
> -DSQLITE_OMIT_LOOKASIDE
> 
> (2) Enable whatever AddressSanitizer tools
> (https://en.wikipedia.org/wiki/AddressSanitizer) you have available on your 
> platform, in addition to (1) above.
>
> (3) If you do not have an easily accessible address sanitizer utility, 
> consider recompiling SQLite using -DSQLITE_MEMDEBUG (in > > addition to the 
> compile-time options shown in (1) above).
>
> --
> D. Richard Hipp

@D. Richard Hipp:
We have recompiled SQLITE with the following options and have it up and running 
now:
Recompiled with: -DSQLITE_DEBUG,  -DSQLITE_OMIT_LOOKASIDE and -DSQLITE_MEMDEBUG.

It is clear to us why we have to compile with -DSQLITE_OMIT_LOOKASIDE, but what 
about the other two options?
1. What is the effect of -DSQLITE_DEBUG?
2. What is the effect of -DSQLITE_MEMDEBUG?
Could we expect any extra output or extra info while analyzing crash dumps by 
using these compile-time options?


> Can you run
>
> PRAGMA integrity_check
>
> on the database ?  If you don't want to tie up your live copy, copy it 
> somewhere else and run the PRAGMA on the copy.
>
> Simon.

@Simon: The result of this pragma was "Ok".

[sqlite] SQLite crashes

[Richard Hipp]

On 11/27/15, Harmen de Jong - CoachR Group B.V.  wrote:
>> (1) Recompile with the following compile-time options:  -DSQLITE_DEBUG
>> -DSQLITE_OMIT_LOOKASIDE
>>
>> (2) Enable whatever AddressSanitizer tools
>> (https://en.wikipedia.org/wiki/AddressSanitizer) you have available on
>> your platform, in addition to (1) above.
>>
>> (3) If you do not have an easily accessible address sanitizer utility,
>> consider recompiling SQLite using -DSQLITE_MEMDEBUG (in > > addition to
>> the compile-time options shown in (1) above).
>>
>> --
>> D. Richard Hipp
>
> @D. Richard Hipp:
> We have recompiled SQLITE with the following options and have it up and
> running now:
> Recompiled with: -DSQLITE_DEBUG,  -DSQLITE_OMIT_LOOKASIDE and
> -DSQLITE_MEMDEBUG.
>
> It is clear to us why we have to compile with -DSQLITE_OMIT_LOOKASIDE, but
> what about the other two options?

The lookaside memory allocator is a fast memory pool used by
individual database connections for short-lived memory allocations.
It makes SQLite run faster, but by bypassing the system
malloc()/free() it can mask some types of heap corruption.  Best to
turn it off when dealing with suspected heap problems.

> 1. What is the effect of -DSQLITE_DEBUG?

This option enables assert() statements.  If there are any internal
inconsistencies within SQLite, they are likely to be caught by the
numerous assert() statements in the code.

> 2. What is the effect of -DSQLITE_MEMDEBUG?

This adds a wrapper around system malloc()/free() that works to test
the integrity of the heap.  It is not a great wrapper.  (Things like
AddressSanitizer are better.)  But it is better than nothing.  Errors
are printed if heap problems are detected.

> Could we expect any extra output or extra info while analyzing crash dumps
> by using these compile-time options?
>
>
>> Can you run
>>
>> PRAGMA integrity_check
>>
>> on the database ?  If you don't want to tie up your live copy, copy it
>> somewhere else and run the PRAGMA on the copy.
>>
>> Simon.
>
> @Simon: The result of this pragma was "Ok".
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>


-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] SQLite crashes

[Simon Slavin]

On 20 Nov 2015, at 3:07pm, Harmen de Jong - CoachR Group B.V.  wrote:

> SQLITE crashes at least once a day on a heavily loaded database

Can you run

PRAGMA integrity_check

on the database ?  If you don't want to tie up your live copy, copy it 
somewhere else and run the PRAGMA on the copy.

Simon.

[sqlite] SQLite crashes

[Harmen de Jong - CoachR Group B.V.]

SQLITE crashes at least once a day on a heavily loaded database (database size 
is approximately: 17 GB). On the same server we are running several other 
instances of SQLite that have databases that are much bigger in size with 
approximately the same load, however these are not crashing. The biggest 
difference between the crashing instance and the other databases is that the 
crashing instance uses a table with 11,600,000 rows, where the tables in the 
not-crashing instances don't even come close to this number of rows.

Below you will find our investigations so far. Any ideas on how we could 
further investigate these crashes?

OS: Windows Server 2012 R2 Standard
SQLite version: 3.9.2
Bug report: SQLite crashes at least one time a day. It crashes in the function 
sqlite3DbMallocRaw. The callstack is listed below. The query being prepared is 
always different. A few screenshots of the investigation of the crash dump:
http://www.coachrdevelopment.com/share/SQLiteCrash1.png
http://www.coachrdevelopment.com/share/SQLiteCrash2.png

> SQLService 1.0.exe!sqlite3DbMallocRaw(sqlite3 * 
> db=0x007d1d071c80, unsigned __int64 n=1) Line 21963C
   SQLService 1.0.exe!sqlite3VMPrintf(sqlite3 * 
db=0x007d1d071c80, const char * zFormat, char * ap) Line 23021C
   SQLService 1.0.exe!sqlite3MPrintf(sqlite3 * db, const char * 
zFormat=0x7ff6a1ac0660, ...) Line 23039C
   SQLService 1.0.exe!valueFromExpr(sqlite3 * 
db=0x007d1d071c80, Expr * pExpr=0x007d7df9b9b0, unsigned char 
enc='\x1', unsigned char affinity='B', Mem * * ppVal=0x007d2456e7a8, 
ValueNewStat4Ctx * pCtx=0x) Line 66286 C
   SQLService 1.0.exe!sqlite3ColumnDefault(Vdbe * 
v=0x007d70f14250, Table * pTab=0x007d3e67e0b0, int i, int iReg=16) Line 
115652 C
   SQLService 1.0.exe!sqlite3ExprCodeGetColumnOfTable(Vdbe * 
v=0x007d70f14250, Table * pTab=0x007d3e67e0b0, int iTabCur, int 
iCol=12, int regOut=16) Line 87462   C
   SQLService 1.0.exe!sqlite3Update(Parse * 
pParse=0x007d79e65df0, SrcList * pTabList=0x007d12116540, ExprList * 
pChanges=0x007d12116940, Expr * pWhere=0x007d121179c0, int onError=10) 
Line 116106  C
   SQLService 1.0.exe!yy_reduce(yyParser * 
yypParser=0x007d51fc24c0, int yyruleno=173) Line 128717C
   SQLService 1.0.exe!sqlite3Parser(void * yyp=0x007d51fc24c0, 
int yymajor=1, Token yyminor, Parse * pParse) Line 129602   C
   SQLService 1.0.exe!sqlite3RunParser(Parse * 
pParse=0x007d79e65df0, const char * zSql=0x007d121159c0, char * * 
pzErrMsg=0x007d2456eba8) Line 130352  C
   SQLService 1.0.exe!sqlite3Prepare(sqlite3 * 
db=0x007d1d071c80, const char * zSql=0x007d599c7c60, int nBytes, int 
saveSqlFlag=1, Vdbe * pReprepare=0x, sqlite3_stmt * * 
ppStmt=0x007d2456f718, const char * * pzTail=0x) Line 
108364   C
   SQLService 1.0.exe!sqlite3LockAndPrepare(sqlite3 * 
db=0x007d1d071c80, const char * zSql=0x007d599c7c60, int nBytes=49, int 
saveSqlFlag=1, Vdbe * pOld=0x, sqlite3_stmt * * 
ppStmt=0x007d2456f718, const char * * pzTail=0x) Line 
108465   C
   SQLService 1.0.exe!sqlite3_prepare_v2(sqlite3 * db, const char * 
zSql, int nBytes, sqlite3_stmt * * ppStmt, const char * * 
pzTail=0x) Line 108544  C

Best regards,
Harmen

[sqlite] SQLite crashes

[Richard Hipp]

On 11/20/15, Harmen de Jong - CoachR Group B.V.  wrote:
>
> Any ideas on how we could further investigate these crashes?
>

(1) Recompile with the following compile-time options:  -DSQLITE_DEBUG
-DSQLITE_OMIT_LOOKASIDE

(2) Enable whatever AddressSanitizer tools
(https://en.wikipedia.org/wiki/AddressSanitizer) you have available on
your platform, in addition to (1) above.

(3) If you do not have an easily accessible address sanitizer utility,
consider recompiling SQLite using -DSQLITE_MEMDEBUG (in addition to
the compile-time options shown in (1) above).

-- 
D. Richard Hipp
drh at sqlite.org

Re: [sqlite] Sqlite crashes on FTS query

[Richard Hipp]

On Sat, Mar 23, 2013 at 6:12 PM, Tamás Kovács  wrote:

> Here's how to reproduce it:
> - Get the latest sqlite shell for windows:
> http://www.sqlite.org/2013/sqlite-shell-win32-x86-3071600.zip
> - Download and extract this database:
> https://dl.dropbox.com/u/242199/db.zip
> - Run this query: SELECT rowid FROM ad_text WHERE title MATCH 'text:t60
> NEAR/6 lenovo';
>
> The problem is probably not in the sqlite shell, the same crash occurs if I
> try the query in Sqlite Manager (the firefox plugin) which uses sqlite
> 3.7.7.1.
> I'm running windows vista sp1 64-bit.
>

Thanks for the report.  Ticket entered here:
http://www.sqlite.org/src/info/38b1ae018f

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Sqlite crashes on FTS query

[Jean-Christophe Deschamps]

At 23:12 23/03/2013, you wrote:
´¯¯¯

Here's how to reproduce it:
- Get the latest sqlite shell for windows:
http://www.sqlite.org/2013/sqlite-shell-win32-x86-3071600.zip
- Download and extract this database: 
https://dl.dropbox.com/u/242199/db.zip

- Run this query: SELECT rowid FROM ad_text WHERE title MATCH 'text:t60
NEAR/6 lenovo';

The problem is probably not in the sqlite shell, the same crash occurs 
if I

try the query in Sqlite Manager (the firefox plugin) which uses sqlite
3.7.7.1.
I'm running windows vista sp1 64-bit.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

`---

I did what you suggest and SQLite Expert running v3.7.16 returned rowid 
4782 in 40ms.


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Sqlite crashes on FTS query

[Tamás Kovács]

Here's how to reproduce it:
- Get the latest sqlite shell for windows:
http://www.sqlite.org/2013/sqlite-shell-win32-x86-3071600.zip
- Download and extract this database: https://dl.dropbox.com/u/242199/db.zip
- Run this query: SELECT rowid FROM ad_text WHERE title MATCH 'text:t60
NEAR/6 lenovo';

The problem is probably not in the sqlite shell, the same crash occurs if I
try the query in Sqlite Manager (the firefox plugin) which uses sqlite
3.7.7.1.
I'm running windows vista sp1 64-bit.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite crashes due to invalid pointer

[Richard Hipp]

On Sun, Dec 12, 2010 at 3:28 AM, Wanadoo Hartwig <
hartwig.wiesm...@wanadoo.nl> wrote:

> Hi,
>
> sorry, but I thought that there would have been an automatic conversion to
> a float. The comparison is not 1 but a floating point literal like 1.0.
>

I'm thinking you are making other assumptions as well.   Please send

(1) Your complete schema
(2) The exact text (byte-for-byte) of your query
(3) The implementation of your function1() function



>
> Am 12.12.2010 um 03:49 schrieb Richard Hipp:
>
> > On Sat, Dec 11, 2010 at 7:07 PM, Wanadoo Hartwig <
> > hartwig.wiesm...@wanadoo.nl> wrote:
> >
> >> Hi,
> >>
> >> take the following SQL statement:
> >>
> >> UPDATE tableA SET column1=(SELECT column2 FROM tableB WHERE
> >> function1(column3) < 1 ORDER BY function1(column3) LIMIT 1);
> >>
> >> Actually, this statement does not make sense because the ORDER BY
> >> expression does not fulfill the requirements of an ORDER BY expression.
> I
> >> wrote it by mistake.
> >>
> >> Interestingly SQLite only crashes if function1 is a user supplied
> function
> >> (using sqlite3_create_function). I tried the same with the core abs()
> >> function but then SQLite works.
> >>
> >
> > The statement you supply above never generates an OP_Real instruction.
> > OP_Real is only generated if your statement contains a floating point
> > literal, which yours does not.  Are you user that the UPDATE statement
> you
> > are giving above is the statement that is crashing?
> >
> >
>
> This is the more precise statement:
>
> UPDATE tableA SET column1=(SELECT column2 FROM tableB WHERE
> function1(1.0,column3) < 1.0 ORDER BY function1(1.0,column3) LIMIT 1);
>
> Actually there is a WHERE expression in the update statement, too. But
> there is no difference if it exists or not (tested).
>
> >
> >
> >> I am using SQLite 3.7.2 and the crash occurs here:
> >>
> >> case OP_Real: {/* same as TK_FLOAT, out2-prerelease */
> >> pOut->flags = MEM_Real;
> >> assert( !sqlite3IsNaN(*pOp->p4.pReal) ); <-- crashes here because of
> >> invalid pointer to p4
> >> pOut->r = *pOp->p4.pReal;
> >> break;
> >> }
> >>
> >>
> >> Hartwig
> >>
> >> ___
> >> sqlite-users mailing list
> >> sqlite-users@sqlite.org
> >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >>
> >
> >
> >
> > --
> > D. Richard Hipp
> > d...@sqlite.org
> > ___
> > sqlite-users mailing list
> > sqlite-users@sqlite.org
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> Hartwig
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite crashes due to invalid pointer

[Wanadoo Hartwig]

Hi,

sorry, but I thought that there would have been an automatic conversion to a 
float. The comparison is not 1 but a floating point literal like 1.0.

Am 12.12.2010 um 03:49 schrieb Richard Hipp:

> On Sat, Dec 11, 2010 at 7:07 PM, Wanadoo Hartwig <
> hartwig.wiesm...@wanadoo.nl> wrote:
> 
>> Hi,
>> 
>> take the following SQL statement:
>> 
>> UPDATE tableA SET column1=(SELECT column2 FROM tableB WHERE
>> function1(column3) < 1 ORDER BY function1(column3) LIMIT 1);
>> 
>> Actually, this statement does not make sense because the ORDER BY
>> expression does not fulfill the requirements of an ORDER BY expression. I
>> wrote it by mistake.
>> 
>> Interestingly SQLite only crashes if function1 is a user supplied function
>> (using sqlite3_create_function). I tried the same with the core abs()
>> function but then SQLite works.
>> 
> 
> The statement you supply above never generates an OP_Real instruction.
> OP_Real is only generated if your statement contains a floating point
> literal, which yours does not.  Are you user that the UPDATE statement you
> are giving above is the statement that is crashing?
> 
> 

This is the more precise statement:

UPDATE tableA SET column1=(SELECT column2 FROM tableB WHERE 
function1(1.0,column3) < 1.0 ORDER BY function1(1.0,column3) LIMIT 1);

Actually there is a WHERE expression in the update statement, too. But there is 
no difference if it exists or not (tested).

> 
> 
>> I am using SQLite 3.7.2 and the crash occurs here:
>> 
>> case OP_Real: {/* same as TK_FLOAT, out2-prerelease */
>> pOut->flags = MEM_Real;
>> assert( !sqlite3IsNaN(*pOp->p4.pReal) ); <-- crashes here because of
>> invalid pointer to p4
>> pOut->r = *pOp->p4.pReal;
>> break;
>> }
>> 
>> 
>> Hartwig
>> 
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>> 
> 
> 
> 
> -- 
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Hartwig


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite crashes due to invalid pointer

[Richard Hipp]

On Sat, Dec 11, 2010 at 7:07 PM, Wanadoo Hartwig <
hartwig.wiesm...@wanadoo.nl> wrote:

> Hi,
>
> take the following SQL statement:
>
> UPDATE tableA SET column1=(SELECT column2 FROM tableB WHERE
> function1(column3) < 1 ORDER BY function1(column3) LIMIT 1);
>
> Actually, this statement does not make sense because the ORDER BY
> expression does not fulfill the requirements of an ORDER BY expression. I
> wrote it by mistake.
>
> Interestingly SQLite only crashes if function1 is a user supplied function
> (using sqlite3_create_function). I tried the same with the core abs()
> function but then SQLite works.
>

The statement you supply above never generates an OP_Real instruction.
OP_Real is only generated if your statement contains a floating point
literal, which yours does not.  Are you user that the UPDATE statement you
are giving above is the statement that is crashing?




> I am using SQLite 3.7.2 and the crash occurs here:
>
> case OP_Real: {/* same as TK_FLOAT, out2-prerelease */
>  pOut->flags = MEM_Real;
>  assert( !sqlite3IsNaN(*pOp->p4.pReal) ); <-- crashes here because of
> invalid pointer to p4
>  pOut->r = *pOp->p4.pReal;
>  break;
> }
>
>
> Hartwig
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] SQLite crashes due to invalid pointer

[Wanadoo Hartwig]

Hi,

take the following SQL statement:

UPDATE tableA SET column1=(SELECT column2 FROM tableB WHERE function1(column3) 
< 1 ORDER BY function1(column3) LIMIT 1);

Actually, this statement does not make sense because the ORDER BY expression 
does not fulfill the requirements of an ORDER BY expression. I wrote it by 
mistake.

Interestingly SQLite only crashes if function1 is a user supplied function 
(using sqlite3_create_function). I tried the same with the core abs() function 
but then SQLite works.
I am using SQLite 3.7.2 and the crash occurs here:

case OP_Real: {/* same as TK_FLOAT, out2-prerelease */
  pOut->flags = MEM_Real;
  assert( !sqlite3IsNaN(*pOp->p4.pReal) ); <-- crashes here because of invalid 
pointer to p4
  pOut->r = *pOp->p4.pReal;
  break;
}


Hartwig

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite crashes when trying to INSERT into table with wrong default value

[Dan Kennedy]

On Sep 30, 2009, at 1:33 PM, ste...@schallerl.com wrote:

> Hi all,
>
> i've got a table:
> CREATE TABLE [usr_user] ([id] INTEGER PRIMARY KEY AUTOINCREMENT,
> [name] TEXT, [email] TEXT, [password] TEXT, [createDate] DATETIME
> DEFAULT (NOW()))

Looks like the same problem as this one:

   http://www.sqlite.org/src/info/2d401a94287

This was fixed for 3.6.18.

Dan.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] SQLite crashes when trying to INSERT into table with wrong default value

[stefan]

Hi all,

i've got a table:
CREATE TABLE [usr_user] ([id] INTEGER PRIMARY KEY AUTOINCREMENT,  
[name] TEXT, [email] TEXT, [password] TEXT, [createDate] DATETIME  
DEFAULT (NOW()))

see the problem? i figured out myself that "NOW()" as a default value  
doesn't work (i'm new with sqlite, coming from mysql - so it didn't  
occur to me right away).

now, when i tried to insert a row into this table:
INSERT INTO usr_user ('asdf', 'a...@asdf.com', 'somemd5hash')

it crashed sqliteman, sqlitestudio and apache with php5/pdo. in php i  
was able to step through the code, crashing occured when preparing  
that statement.

for reference, the error reported in apaches error log file is:
[Wed Sep 30 08:24:51 2009] [notice] Parent: child process exited with  
status 3221225477 -- Restarting.
i remember the status code to be a different one when trying it from home.

i've tried to replicate it on 3 different systems, all with the same result:
* winXP 32b
* win7 64b
* Linux 2.6.27.7-grsec #25 SMP Mon Feb 16 16:37:38 CET 2009 i686

sqlite info:
PECL Module version: (bundled) 1.0.1 $Id: pdo_sqlite.c,v 1.10.2.6.2.4  
2008/12/31 11:17:42 sebastian Exp $
SQLite Library:  3.3.7

now, this error is easily fixed by just removing/replacing the  
offending default value, but shouldn't it be handelt differently than  
just crashing?

thanks, stefan


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite Crashes

[Shawn Wilsher]

>  I put a pointer to the mozilla bug report here:
I probably should have mentioned our bug report as well.  It's bug 408518 [1].

>  These stack traces don't make any sense to me either. The definition
>  of sqlite3_enable_shared_cache() in SQLite cvs is:
That, and as far as I can tell it's ever called by any other sqlite code.

>int sqlite3_enable_shared_cache(int enable){
>  sqlite3SharedCacheEnabled = enable;
>  return SQLITE_OK;
>}
>
>  sqlite3SharedCacheEnable is a file scoped int.
hmm, are there some threadsafty issues there with setting and reading
that value from (possibly) multiple threads?  I don't think it's
related to this (I don't think anything in core code in mozilla
actually toggles those - but add-ons can do it) however.

>  Stack overflow possibly? Will keep thinking this.
Someone mentioned in the mozilla bug that the new allocator we
switched to may have landed around the time we started seeing this.
I'll look into it further.

Cheers,

Shawn Wilsher
Mozilla Developer

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=408518
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite Crashes

[Dan]

On Mar 2, 2008, at 4:42 AM, Shawn Wilsher wrote:

> Hey all,
>
> Over at Mozilla we've been seeing a large amount of crashes in
> sqlite3_enable_shared_cache.  The stack frames don't make a whole lot
> of sense to me, so I thought I'd inform you and hope that you might
> have a better idea as to what is going on.  If you have any questions,
> feel free to ask.  If I don't know the answer, I'll get the people who
> should know involved.  We'd really like to try and resolve this issue,
> so insight on this matter would be greatly appreciated.
>
> http://tinyurl.com/2393qs
>
> We are presently using the latest version of sqlite.

Hi Shawn,

I put a pointer to the mozilla bug report here:

   http://www.sqlite.org/cvstrac/tktview?tn=2970

These stack traces don't make any sense to me either. The definition
of sqlite3_enable_shared_cache() in SQLite cvs is:

   int sqlite3_enable_shared_cache(int enable){
 sqlite3SharedCacheEnabled = enable;
 return SQLITE_OK;
   }

sqlite3SharedCacheEnable is a file scoped int.

Some of the stack traces have sqlite3_enable_shared_cache() being called
from Mozilla code, but some others show it being called from other
parts of SQLite that make no sense to me. This one is particularly
odd:

   http://crash-stats.mozilla.com/report/index/8b54f1c5-e8aa-11dc- 
b466-001a4bd43e5c

Stack overflow possibly? Will keep thinking this.

Regards,
Dan.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] SQLite Crashes

[Shawn Wilsher]

Hey all,

Over at Mozilla we've been seeing a large amount of crashes in
sqlite3_enable_shared_cache.  The stack frames don't make a whole lot
of sense to me, so I thought I'd inform you and hope that you might
have a better idea as to what is going on.  If you have any questions,
feel free to ask.  If I don't know the answer, I'll get the people who
should know involved.  We'd really like to try and resolve this issue,
so insight on this matter would be greatly appreciated.

http://tinyurl.com/2393qs

We are presently using the latest version of sqlite.

Cheers,

Shawn Wilsher
Mozilla Developer
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Sqlite crashes when i imort huge list

[Jay Sprenkle]

On 6/26/06, C.Peachment <[EMAIL PROTECTED]> wrote:


>>The use of sequential numbers as the trailing part of the 'word'
>>results in a continual rebalancing of the b-tree with each insertion.

>Is that right considering it looks like you have not created an index on
>the word column before inserting the data?


Oops, you got that right :-)

I was doing a lazy Sunday morning quick and dirty implementation
and forgot to define the index. If you assume there had been
a PRIMARY KEY phrase after the field definition then I think
my comment would be valid.


I wrote a command line importer. You're welcome to try it if your
problem wasn't already solved. It is much slower to insert into an indexed
table than to insert all the records then create the index. It doesn't use
time rebalancing the trees when it's done as a batch at the end.


--
SqliteImporter and SqliteReplicator: Command line utilities for Sqlite
http://www.reddawn.net/~jsprenkl/Sqlite

Cthulhu Bucks!
http://www.cthulhubucks.com

RE: [sqlite] Sqlite crashes when i imort huge list

[C.Peachment]

On Mon, 26 Jun 2006 11:20:59 +0100, Brandon, Nicholas (UK) wrote:



>>The use of sequential numbers as the trailing part of the 'word'
>>results in a continual rebalancing of the b-tree with each insertion.

>Is that right considering it looks like you have not created an index on
>the word column before inserting the data?


Oops, you got that right :-)

I was doing a lazy Sunday morning quick and dirty implementation
and forgot to define the index. If you assume there had been
a PRIMARY KEY phrase after the field definition then I think
my comment would be valid.

RE: [sqlite] Sqlite crashes when i imort huge list

[Brandon, Nicholas (UK)]

>The use of sequential numbers as the trailing part of the 'word'
>results in a continual rebalancing of the b-tree with each insertion.

Is that right considering it looks like you have not created an index on
the word column before inserting the data?

Nick



This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.

Re: [sqlite] Sqlite crashes when i imort huge list

[onemind]

Aha, sorry Richard, it seems you may have been right. I downloaded a
different gui call SQLite Administrator and it is importing now. The first
gui i used, used 100% of my cpu and crashes, this new one doesn't use much
but it is a sloow process.Have been running it for about 10 mins
and am only at 3%. Might go to bed now and hopefully it willl be done by the
morning.

Cheers
--
View this message in context: 
http://www.nabble.com/Sqlite-crashes-when-i-imort-huge-list-t1842991.html#a5034613
Sent from the SQLite forum at Nabble.com.

Re: [sqlite] Sqlite crashes when i imort huge list

[onemind]

Thanks guys,

Richard: The reason i didn't mention the software is because all guis just
create the text commands anyway so they all do the same thing. I doubt it is
a problem with the gui i used which was "SQLite database browser". 

Chris: Thanks for that but i dont know how that helps me. Are you suggesting
that i write a script that inserts one word at a time to avoid crashing or
was it just an example?

How would i write a command that would import from a text file without the
gui? Is it the .import command?

Any more help would be great.

Here is a small sample of the file that i am trying to insert:

AA
AAH
AAHED
AAHING
AAHS
AAL
AALII
AALIIS
AALS
AARDVARK
AARDVARKS
AARDWOLF
AARDWOLVES
AARGH
AARRGH
AARRGHH
AAS
AASVOGEL
AASVOGELS
AB
ABA
ABACA
ABACAS
ABACI
ABACK
ABACTERIAL
ABACUS
ABACUSES
ABAFT
ABAKA
ABAKAS
ABALONE
ABALONES
ABAMP
ABAMPERE
ABAMPERES
ABAMPS
ABANDON
ABANDONED
ABANDONER
ABANDONERS
ABANDONING
ABANDONMENT
ABANDONMENTS
ABANDONS
ABAPICAL
ABAS
ABASE
ABASED
ABASEDLY
ABASEMENT
ABASEMENTS
ABASER
ABASERS
ABASES
ABASH
ABASHED
ABASHEDLY
ABASHES
ABASHING
ABASHMENT
ABASHMENTS
ABASIA
ABASIAS
ABASING
ABATABLE
ABATE
ABATED
ABATEMENT
ABATEMENTS
ABATER
ABATERS
ABATES
ABATING
ABATIS
ABATISES
ABATOR
ABATORS
ABATTIS
ABATTISES
ABATTOIR
ABATTOIRS
ABAXIAL
ABAXILE
ABAYA
ABAYAS
ABBA
ABBACIES
ABBACY
ABBAS
ABBATIAL
ABBE
ABBES
ABBESS
ABBESSES
ABBEY
ABBEYS
ABBOT
ABBOTCIES
ABBOTCY
ABBOTS
ABBOTSHIP
ABBOTSHIPS
ABBREVIATE
ABBREVIATED
ABBREVIATES
ABBREVIATING
ABBREVIATION
ABBREVIATIONS
ABBREVIATOR
ABBREVIATORS
ABCOULOMB
ABCOULOMBS
ABDICABLE
ABDICATE
ABDICATED
ABDICATES
ABDICATING
ABDICATION
ABDICATIONS
ABDICATOR
ABDICATORS
ABDOMEN
ABDOMENS
ABDOMINA
ABDOMINAL
ABDOMINALLY
ABDOMINALS
ABDUCE
ABDUCED
ABDUCENS
ABDUCENT
ABDUCENTES
ABDUCES
ABDUCING
ABDUCT
ABDUCTED
ABDUCTEE
ABDUCTEES
ABDUCTING
ABDUCTION
ABDUCTIONS
ABDUCTOR
ABDUCTORES
ABDUCTORS
ABDUCTS
ABEAM
ABECEDARIAN

Thanks :)
--
View this message in context: 
http://www.nabble.com/Sqlite-crashes-when-i-imort-huge-list-t1842991.html#a5034321
Sent from the SQLite forum at Nabble.com.

Re: [sqlite] Sqlite crashes when i imort huge list

[C.Peachment]

There was a small error in my previous post.
The 'begin transaction;' line was missed when
I copied the text out of the script and editted
it to remove the code around the text.

On Sat, 24 Jun 2006 20:10:58 -0700 (PDT), onemind wrote:

>I am using the sqlite gui and click import table from csv. I select a txt
>file that contain over 200,000 words in a list. Sqlite works fine with a
>smaller list of 200-300 words but when i import my big list, it hangs for
>ages or completely crashes my computer.

>Does anyone know how i can import this list into a table successfully?

The following text file, "word.txt", was created by a small script:

create table wordlist (word text);
begin transaction;
select current_time;
insert into wordlist values ("test0");
insert into wordlist values ("test1");
...
insert into wordlist values ("test18");
insert into wordlist values ("test19");
commit transaction;
select current_time;
select count(*) from wordlist;

This text file was executed within the sqlite3 command line utility
by typing the command:

.read word.txt

The resulting screen output was

11:20:40
11:20:48
20

indicating the insertion of 200,000 words into a simple database
required 8 seconds of clock time on a 1.5 ghz AMD Sempron cpu
running MS-Windows XP.

The use of sequential numbers as the trailing part of the 'word'
results in a continual rebalancing of the b-tree with each insertion.
For performances reasons, this is probably the worst kind of data
to insert into a database. Random words inserted result in many
leaves that are partially filled and fewer rebalance acts.

Eight seconds to insert the lot is pretty good.

You could create a similar text file from the CSV file using your
editor.

Chris

Re: [sqlite] Sqlite crashes when i imort huge list

[C.Peachment]

On Sat, 24 Jun 2006 20:10:58 -0700 (PDT), onemind wrote:

>I am using the sqlite gui and click import table from csv. I select a txt
>file that contain over 200,000 words in a list. Sqlite works fine with a
>smaller list of 200-300 words but when i import my big list, it hangs for
>ages or completely crashes my computer.

>Does anyone know how i can import this list into a table successfully?

The following text file was created by a small script:

create table wordlist (word text);
select current_time;
begin transaction;
insert into wordlist values ("test0");
insert into wordlist values ("test1");
...
insert into wordlist values ("test18");
insert into wordlist values ("test19");
commit transaction;
select current_time;
select count(*) from wordlist;

This text file was executed within the sqlite3 command line utility
by typing the command:

.read word.txt

The resulting screen output was

11:20:40
11:20:48
20

indicating the insertion of 200,000 words into a simple database
required 8 seconds of clock time on a 1.5 ghz AMD Sempron cpu
running MS-Windows XP.

The use of sequential numbers as the trailing part of the 'word'
results in a continual rebalancing of the b-tree with each insertion.
For performances reasons, this is probably the worst kind of data
to insert into a database. Random words inserted result in many
leaves that are partially filled and fewer rebalance acts.

Eight seconds to insert the lot is pretty good.

You could create a similar text file from the CSV file using your
editor.

Chris

[sqlite] Sqlite crashes when i imort huge list

[onemind]

Hi,

I am using the sqlite gui and click import table from csv. I select a txt
file that contain over 200,000 words in a list. Sqlite works fine with a
smaller list of 200-300 words but when i import my big list, it hangs for
ages or completely crashes my computer.

Does anyone know how i can import this list into a table successfully?

Thanks for any help.
--
View this message in context: 
http://www.nabble.com/Sqlite-crashes-when-i-imort-huge-list-t1842991.html#a5030925
Sent from the SQLite forum at Nabble.com.

Re: [sqlite] SQLite crashes during rapid inserts!

[Darren Duncan]

At 2:07 PM +0100 6/24/04, Chris Ulliott wrote:
Hi all,
 I am using the SQLite DB with MS VC++ v 6. In the code below you can see
I am looping around a ODBC connection to an access database and I am
inserting those records into my new SQLite database. My problem is
SQLite.dll causes an access violation after inserting about 190 records.
Does anyone have any ideas what I am missing or if this is a bug?
 Code...
You can isolate whether this is a speed issue or a quantity / data 
type issue by slowing down the inserts.  For example, sleep a short 
time or a second between each row, and see if it still dies at 190 
records. -- Darren Duncan

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

[sqlite] SQLite crashes during rapid inserts!

[Chris Ulliott]

Hi all,
 
I am using the SQLite DB with MS VC++ v 6. In the code below you can see
I am looping around a ODBC connection to an access database and I am
inserting those records into my new SQLite database. My problem is
SQLite.dll causes an access violation after inserting about 190 records.
Does anyone have any ideas what I am missing or if this is a bug?
 
Code...
 
 
  nRecord=0;
  while(pRs->EndOfFile == FALSE)
  {
   nRecord++;
 
   // Create Insert Command

   strCmd = _T("INSERT INTO SCHEMATABLES VALUES(");
   for(int nField = 0; nField < pRs->Fields->Count; nField++)
   {
var.lVal = nField;
var.ChangeType(VT_I2, NULL);
strCmd += _T("\"");
strCmd += pRs->Fields->GetItem(var)->Value.bstrVal;
strCmd += _T("\", ");
   }
   strCmd = strCmd.Left(strCmd.GetLength()-2);
   strCmd += _T(");\n");
 
 
   // SQLite .. Insert record into SQLLite Database
   nReturn = procPrepare(pResult, (LPCTSTR)strCmd, -1, ,
);
   nReturn = procStep(stmt);
 
   // MDB
   pRs->MoveNext();
  }
 
  pRs->Close();
 
  nReturn = procClose(pResult);
  str = procErrMsg(pResult);
 
Thank you in advance,
 
Chris