Re: [sqlite] Read overflow in CREATE VIRTUAL TABLE query (formerly Corrupting pointers to the lookaside smallacator)

[Richard Hipp]

On Thu, Nov 27, 2014 at 7:33 AM, Dominique Devienne 
wrote:

> On Thu, Nov 27, 2014 at 12:57 PM, Dan Kennedy 
> wrote:
>
> > Thanks for tracking this down.Should be fixed here:
> >
> >   http://www.sqlite.org/src/info/f095cde579e7417306
>
>
> Dan, the test reads:
>
> ifcapable fts3 {  ... }
>
> yet both the comments and code use fts4. Should that be fts4, or it somehow
> doesn't matter? --DD
>

fts3 and fts4 are really the same code base.  fts4 merely enables some
extra options that are not backwards compatible, so it had to have a
different name.  The ifcapable command in the test harness only understands
"fts3".  So I believe Dan's test case is correct as written.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Read overflow in CREATE VIRTUAL TABLE query (formerly Corrupting pointers to the lookaside smallacator)

[Dominique Devienne]

On Thu, Nov 27, 2014 at 12:57 PM, Dan Kennedy  wrote:

> Thanks for tracking this down.Should be fixed here:
>
>   http://www.sqlite.org/src/info/f095cde579e7417306


Dan, the test reads:

ifcapable fts3 {  ... }

yet both the comments and code use fts4. Should that be fts4, or it somehow
doesn't matter? --DD
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Read overflow in CREATE VIRTUAL TABLE query (formerly Corrupting pointers to the lookaside smallacator)

[Paul]

Thank you very much, Dan!


> On 11/27/2014 05:56 PM, Paul wrote:
> > Currently we use various versions of SQLite:
> >
> > SQLite version 3.8.0.1 2013-08-29 17:35:01
> > SQLite version 3.8.2 2013-12-06 14:53:30
> > SQLite version 3.8.6 2014-08-15 11:46:33
> > SQLite version 3.8.7 2014-10-17 11:24:17
> >
> >
> > All of them are affected so I never considered it to be an sqlite bug.
> > But analyzing core file it seems like very much an sqlite bug :/
> >
> > Tell me if you need more info on this.
> 
> 
> Thanks for tracking this down.Should be fixed here:
> 
> http://www.sqlite.org/src/info/f095cde579e7417306
> 
> As far as I can see this is "just" a buffer overread - there is no 
> chance of an overwrite or database corruption. Easiest workaround is to 
> append "()" to your CREATE VIRTUAL TABLE statement. i.e.
> 
> CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343()
> 
> Dan.
> 
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Read overflow in CREATE VIRTUAL TABLE query (formerly Corrupting pointers to the lookaside smallacator)

[Dan Kennedy]

On 11/27/2014 05:56 PM, Paul wrote:

Currently we use various versions of SQLite:

SQLite version 3.8.0.1 2013-08-29 17:35:01
SQLite version 3.8.2 2013-12-06 14:53:30
SQLite version 3.8.6 2014-08-15 11:46:33
SQLite version 3.8.7 2014-10-17 11:24:17


All of them are affected so I never considered it to be an sqlite bug.
But analyzing core file it seems like very much an sqlite bug :/

Tell me if you need more info on this.



Thanks for tracking this down.Should be fixed here:

  http://www.sqlite.org/src/info/f095cde579e7417306

As far as I can see this is "just" a buffer overread - there is no 
chance of an overwrite or database corruption. Easiest workaround is to 
append "()" to your CREATE VIRTUAL TABLE statement. i.e.


  CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343()

Dan.






Thanks.


On 11/27/2014 03:20 PM, Paul wrote:

Here is how it looks with debug symbols are on:

#0 0x28c4113e in memcpy () from /lib/libc.so.7
#1 0x08854c20 in sqlite3StrAccumAppend (p=0xfffe8548, z=0x2c3fffda "vtb_enyqkyxs 
USING vtable_module_343", N=41) at sqlite3.c:21563
#2 0x087edf30 in sqlite3VXPrintf (pAccum=0xfffe8548, bFlags=1, fmt=0x892e543 "T", 
ap=0xfffe8610 "") at sqlite3.c:21439
#3 0x088077d5 in sqlite3VMPrintf (db=0x2c006788, zFormat=0x892e52d "CREATE VIRTUAL TABLE 
%T", ap=0xfffe860c "xx") at sqlite3.c:21638
#4 0x087f815e in sqlite3MPrintf (db=0x2c006788, zFormat=0x892e52d "CREATE VIRTUAL 
TABLE %T") at sqlite3.c:21654
#5 0x088759af in sqlite3VtabFinishParse (pParse=0x2c007688, pEnd=0x0) at 
sqlite3.c:112383
#6 0x0885bb21 in yy_reduce (yypParser=0x2c1d1408, yyruleno=309) at 
sqlite3.c:123403
#7 0x08856180 in sqlite3Parser (yyp=0x2c1d1408, yymajor=1, yyminor=..., 
pParse=0x2c007688) at sqlite3.c:123629
#8 0x087fc289 in sqlite3RunParser (pParse=0x2c007688, zSql=0x2c3fffc0 "CREATE 
VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", pzErrMsg=0xfffe8bc4) at 
sqlite3.c:124466
#9 0x088bbc82 in sqlite3Prepare (db=0x2c006788, zSql=0x2c3fffc0 "CREATE VIRTUAL 
TABLE temp.vtb_enyqkyxs USING vtable_module_343", nBytes=-1, saveSqlFlag=1, 
pReprepare=0x0, ppStmt=0xfffe8d0c, pzTail=0xfffe8d10) at sqlite3.c:103750
#10 0x087fb0ce in sqlite3LockAndPrepare (db=0x2c006788, zSql=0x2c3fffc0 "CREATE 
VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", nBytes=-1, saveSqlFlag=1, 
pOld=0x0, ppStmt=0xfffe8d0c, pzTail=0xfffe8d10) at sqlite3.c:103842
#11 0x087fa504 in sqlite3_prepare_v2 (db=0x2c006788, zSql=0x2c3fffc0 "CREATE VIRTUAL 
TABLE temp.vtb_enyqkyxs USING vtable_module_343", nBytes=-1, ppStmt=0xfffe8d0c, 
pzTail=0xfffe8d10) at sqlite3.c:103918
#12 0x087fa01d in sqlite3_exec (db=0x2c006788, zSql=0x2c3fffc0 "CREATE VIRTUAL TABLE 
temp.vtb_enyqkyxs USING vtable_module_343", xCallback=0x0, pArg=0x0, pzErrMsg=0x0) 
at sqlite3.c:99345
#13 0x086588e7 in sqlite::StorageBase::exec_query (this=0x2c3ff640, query=0x2c3fffc0 
"CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", quiet=false) 
at SqliteStorageBase.cpp:286


Interesting part is in frame #5

#5 0x088759af in sqlite3VtabFinishParse (pParse=0x2c007688, pEnd=0x0) at 
sqlite3.c:112383
112383 zStmt = sqlite3MPrintf(db, "CREATE VIRTUAL TABLE %T", 
>sNameToken);
(gdb) p pParse->sNameToken
$12 = {
z = 0x2c3fffda "vtb_enyqkyxs USING vtable_module_343",
n = 41
}
(gdb) p pParse->sNameToken.z + 40
$13 = 0x2c42 

As you can see, token size is invalid: 41. This causes memcpy() down at #0 to 
access invalid, unmapped address.
Hopefully it is only read overflow so the only consequence is just an 
occasional Segmentation fault when allocated
piece of string is at the specific place: near the end of the page in front of 
unmapped page.

Should I fill a bug report?

It's certainly very suspicious. Which SQLite version are you using?

Dan.






Is there a way to work this around?

Like append many spaces a the end of query?
Or maybe the problem is in absence of ';' at the end of query?
Meantime I'll try both of these cases.




We observe very similar problem.

#1 0x087ec9f7 in sqlite3VXPrintf ()
#2 0x087f816d in sqlite3MPrintf ()
#3 0x088781e5 in sqlite3VtabFinishParse ()
#4 0x0885190f in yy_reduce ()
#5 0x0884d4d8 in sqlite3Parser ()
#6 0x087fc0ce in sqlite3RunParser ()
#7 0x088aa396 in sqlite3Prepare ()
#8 0x087fae18 in sqlite3LockAndPrepare ()
#9 0x087f9a88 in sqlite3_exec ()
#10 0x086588a7 in sqlite::StorageBase::exec_query (this=0x2c2a47c0, query=0x2c7fffc0 
"CREATE VIRTUAL TABLE temp.vtb_wayxzmbo USING vtable_module_344", quiet=false) 
at SqliteStorageBase.cpp:286

It always crashes when "CREATE VIRTUAL TABLE ..." is being executed, always 
with the same backtrace.
I spent many days reviewing and testing my code to eliminate possible cause but 
so far I see nothing wrong with it.
Probability of this crash is so very low so that problem can be reproduced only 
on hi loaded production servers.
(Where such virtual tables are created and dropped millions of times during a 
day)

I am going to compile sqlite without optimizations and with debug 

Re: [sqlite] Read overflow in CREATE VIRTUAL TABLE query (formerly Corrupting pointers to the lookaside smallacator)

[Paul]

Currently we use various versions of SQLite: 

SQLite version 3.8.0.1 2013-08-29 17:35:01
SQLite version 3.8.2 2013-12-06 14:53:30
SQLite version 3.8.6 2014-08-15 11:46:33
SQLite version 3.8.7 2014-10-17 11:24:17


All of them are affected so I never considered it to be an sqlite bug.
But analyzing core file it seems like very much an sqlite bug :/

Tell me if you need more info on this.

Thanks.

> On 11/27/2014 03:20 PM, Paul wrote:
> > Here is how it looks with debug symbols are on:
> >
> > #0 0x28c4113e in memcpy () from /lib/libc.so.7
> > #1 0x08854c20 in sqlite3StrAccumAppend (p=0xfffe8548, z=0x2c3fffda 
> > "vtb_enyqkyxs USING vtable_module_343", N=41) at sqlite3.c:21563
> > #2 0x087edf30 in sqlite3VXPrintf (pAccum=0xfffe8548, bFlags=1, 
> > fmt=0x892e543 "T", ap=0xfffe8610 "") at sqlite3.c:21439
> > #3 0x088077d5 in sqlite3VMPrintf (db=0x2c006788, zFormat=0x892e52d "CREATE 
> > VIRTUAL TABLE %T", ap=0xfffe860c "xx") at sqlite3.c:21638
> > #4 0x087f815e in sqlite3MPrintf (db=0x2c006788, zFormat=0x892e52d "CREATE 
> > VIRTUAL TABLE %T") at sqlite3.c:21654
> > #5 0x088759af in sqlite3VtabFinishParse (pParse=0x2c007688, pEnd=0x0) at 
> > sqlite3.c:112383
> > #6 0x0885bb21 in yy_reduce (yypParser=0x2c1d1408, yyruleno=309) at 
> > sqlite3.c:123403
> > #7 0x08856180 in sqlite3Parser (yyp=0x2c1d1408, yymajor=1, yyminor=..., 
> > pParse=0x2c007688) at sqlite3.c:123629
> > #8 0x087fc289 in sqlite3RunParser (pParse=0x2c007688, zSql=0x2c3fffc0 
> > "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", 
> > pzErrMsg=0xfffe8bc4) at sqlite3.c:124466
> > #9 0x088bbc82 in sqlite3Prepare (db=0x2c006788, zSql=0x2c3fffc0 "CREATE 
> > VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", nBytes=-1, 
> > saveSqlFlag=1, pReprepare=0x0, ppStmt=0xfffe8d0c, pzTail=0xfffe8d10) at 
> > sqlite3.c:103750
> > #10 0x087fb0ce in sqlite3LockAndPrepare (db=0x2c006788, zSql=0x2c3fffc0 
> > "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", 
> > nBytes=-1, saveSqlFlag=1, pOld=0x0, ppStmt=0xfffe8d0c, pzTail=0xfffe8d10) 
> > at sqlite3.c:103842
> > #11 0x087fa504 in sqlite3_prepare_v2 (db=0x2c006788, zSql=0x2c3fffc0 
> > "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", 
> > nBytes=-1, ppStmt=0xfffe8d0c, pzTail=0xfffe8d10) at sqlite3.c:103918
> > #12 0x087fa01d in sqlite3_exec (db=0x2c006788, zSql=0x2c3fffc0 "CREATE 
> > VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", xCallback=0x0, 
> > pArg=0x0, pzErrMsg=0x0) at sqlite3.c:99345
> > #13 0x086588e7 in sqlite::StorageBase::exec_query (this=0x2c3ff640, 
> > query=0x2c3fffc0 "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING 
> > vtable_module_343", quiet=false) at SqliteStorageBase.cpp:286
> > 
> >
> > Interesting part is in frame #5
> >
> > #5 0x088759af in sqlite3VtabFinishParse (pParse=0x2c007688, pEnd=0x0) at 
> > sqlite3.c:112383
> > 112383 zStmt = sqlite3MPrintf(db, "CREATE VIRTUAL TABLE %T", 
> > >sNameToken);
> > (gdb) p pParse->sNameToken
> > $12 = {
> > z = 0x2c3fffda "vtb_enyqkyxs USING vtable_module_343",
> > n = 41
> > }
> > (gdb) p pParse->sNameToken.z + 40
> > $13 = 0x2c42 
> >
> > As you can see, token size is invalid: 41. This causes memcpy() down at #0 
> > to access invalid, unmapped address.
> > Hopefully it is only read overflow so the only consequence is just an 
> > occasional Segmentation fault when allocated
> > piece of string is at the specific place: near the end of the page in front 
> > of unmapped page.
> >
> > Should I fill a bug report?
> 
> It's certainly very suspicious. Which SQLite version are you using?
> 
> Dan.
> 
> 
> 
> 
> 
> >
> > Is there a way to work this around?
> >
> > Like append many spaces a the end of query?
> > Or maybe the problem is in absence of ';' at the end of query?
> > Meantime I'll try both of these cases.
> >
> >
> > 
> >> We observe very similar problem.
> >>
> >> #1 0x087ec9f7 in sqlite3VXPrintf ()
> >> #2 0x087f816d in sqlite3MPrintf ()
> >> #3 0x088781e5 in sqlite3VtabFinishParse ()
> >> #4 0x0885190f in yy_reduce ()
> >> #5 0x0884d4d8 in sqlite3Parser ()
> >> #6 0x087fc0ce in sqlite3RunParser ()
> >> #7 0x088aa396 in sqlite3Prepare ()
> >> #8 0x087fae18 in sqlite3LockAndPrepare ()
> >> #9 0x087f9a88 in sqlite3_exec ()
> >> #10 0x086588a7 in sqlite::StorageBase::exec_query (this=0x2c2a47c0, 
> >> query=0x2c7fffc0 "CREATE VIRTUAL TABLE temp.vtb_wayxzmbo USING 
> >> vtable_module_344", quiet=false) at SqliteStorageBase.cpp:286
> >>
> >> It always crashes when "CREATE VIRTUAL TABLE ..." is being executed, 
> >> always with the same backtrace.
> >> I spent many days reviewing and testing my code to eliminate possible 
> >> cause but so far I see nothing wrong with it.
> >> Probability of this crash is so very low so that problem can be reproduced 
> >> only on hi loaded production servers.
> >> (Where such virtual tables are created and dropped millions of times 
> >> during a day)
> >>
> >> I am going to compile sqlite without optimizations and with debug symbols 
> >>