Re: [sqlite] Read overflow in CREATE VIRTUAL TABLE query (formerly Corrupting pointers to the lookaside smallacator)
On Thu, Nov 27, 2014 at 7:33 AM, Dominique Deviennewrote: > On Thu, Nov 27, 2014 at 12:57 PM, Dan Kennedy > wrote: > > > Thanks for tracking this down.Should be fixed here: > > > > http://www.sqlite.org/src/info/f095cde579e7417306 > > > Dan, the test reads: > > ifcapable fts3 { ... } > > yet both the comments and code use fts4. Should that be fts4, or it somehow > doesn't matter? --DD > fts3 and fts4 are really the same code base. fts4 merely enables some extra options that are not backwards compatible, so it had to have a different name. The ifcapable command in the test harness only understands "fts3". So I believe Dan's test case is correct as written. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Read overflow in CREATE VIRTUAL TABLE query (formerly Corrupting pointers to the lookaside smallacator)
On Thu, Nov 27, 2014 at 12:57 PM, Dan Kennedywrote: > Thanks for tracking this down.Should be fixed here: > > http://www.sqlite.org/src/info/f095cde579e7417306 Dan, the test reads: ifcapable fts3 { ... } yet both the comments and code use fts4. Should that be fts4, or it somehow doesn't matter? --DD ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Read overflow in CREATE VIRTUAL TABLE query (formerly Corrupting pointers to the lookaside smallacator)
Thank you very much, Dan! > On 11/27/2014 05:56 PM, Paul wrote: > > Currently we use various versions of SQLite: > > > > SQLite version 3.8.0.1 2013-08-29 17:35:01 > > SQLite version 3.8.2 2013-12-06 14:53:30 > > SQLite version 3.8.6 2014-08-15 11:46:33 > > SQLite version 3.8.7 2014-10-17 11:24:17 > > > > > > All of them are affected so I never considered it to be an sqlite bug. > > But analyzing core file it seems like very much an sqlite bug :/ > > > > Tell me if you need more info on this. > > > Thanks for tracking this down.Should be fixed here: > > http://www.sqlite.org/src/info/f095cde579e7417306 > > As far as I can see this is "just" a buffer overread - there is no > chance of an overwrite or database corruption. Easiest workaround is to > append "()" to your CREATE VIRTUAL TABLE statement. i.e. > > CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343() > > Dan. > > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Read overflow in CREATE VIRTUAL TABLE query (formerly Corrupting pointers to the lookaside smallacator)
On 11/27/2014 05:56 PM, Paul wrote: Currently we use various versions of SQLite: SQLite version 3.8.0.1 2013-08-29 17:35:01 SQLite version 3.8.2 2013-12-06 14:53:30 SQLite version 3.8.6 2014-08-15 11:46:33 SQLite version 3.8.7 2014-10-17 11:24:17 All of them are affected so I never considered it to be an sqlite bug. But analyzing core file it seems like very much an sqlite bug :/ Tell me if you need more info on this. Thanks for tracking this down.Should be fixed here: http://www.sqlite.org/src/info/f095cde579e7417306 As far as I can see this is "just" a buffer overread - there is no chance of an overwrite or database corruption. Easiest workaround is to append "()" to your CREATE VIRTUAL TABLE statement. i.e. CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343() Dan. Thanks. On 11/27/2014 03:20 PM, Paul wrote: Here is how it looks with debug symbols are on: #0 0x28c4113e in memcpy () from /lib/libc.so.7 #1 0x08854c20 in sqlite3StrAccumAppend (p=0xfffe8548, z=0x2c3fffda "vtb_enyqkyxs USING vtable_module_343", N=41) at sqlite3.c:21563 #2 0x087edf30 in sqlite3VXPrintf (pAccum=0xfffe8548, bFlags=1, fmt=0x892e543 "T", ap=0xfffe8610 "") at sqlite3.c:21439 #3 0x088077d5 in sqlite3VMPrintf (db=0x2c006788, zFormat=0x892e52d "CREATE VIRTUAL TABLE %T", ap=0xfffe860c "xx") at sqlite3.c:21638 #4 0x087f815e in sqlite3MPrintf (db=0x2c006788, zFormat=0x892e52d "CREATE VIRTUAL TABLE %T") at sqlite3.c:21654 #5 0x088759af in sqlite3VtabFinishParse (pParse=0x2c007688, pEnd=0x0) at sqlite3.c:112383 #6 0x0885bb21 in yy_reduce (yypParser=0x2c1d1408, yyruleno=309) at sqlite3.c:123403 #7 0x08856180 in sqlite3Parser (yyp=0x2c1d1408, yymajor=1, yyminor=..., pParse=0x2c007688) at sqlite3.c:123629 #8 0x087fc289 in sqlite3RunParser (pParse=0x2c007688, zSql=0x2c3fffc0 "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", pzErrMsg=0xfffe8bc4) at sqlite3.c:124466 #9 0x088bbc82 in sqlite3Prepare (db=0x2c006788, zSql=0x2c3fffc0 "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", nBytes=-1, saveSqlFlag=1, pReprepare=0x0, ppStmt=0xfffe8d0c, pzTail=0xfffe8d10) at sqlite3.c:103750 #10 0x087fb0ce in sqlite3LockAndPrepare (db=0x2c006788, zSql=0x2c3fffc0 "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", nBytes=-1, saveSqlFlag=1, pOld=0x0, ppStmt=0xfffe8d0c, pzTail=0xfffe8d10) at sqlite3.c:103842 #11 0x087fa504 in sqlite3_prepare_v2 (db=0x2c006788, zSql=0x2c3fffc0 "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", nBytes=-1, ppStmt=0xfffe8d0c, pzTail=0xfffe8d10) at sqlite3.c:103918 #12 0x087fa01d in sqlite3_exec (db=0x2c006788, zSql=0x2c3fffc0 "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", xCallback=0x0, pArg=0x0, pzErrMsg=0x0) at sqlite3.c:99345 #13 0x086588e7 in sqlite::StorageBase::exec_query (this=0x2c3ff640, query=0x2c3fffc0 "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", quiet=false) at SqliteStorageBase.cpp:286 Interesting part is in frame #5 #5 0x088759af in sqlite3VtabFinishParse (pParse=0x2c007688, pEnd=0x0) at sqlite3.c:112383 112383 zStmt = sqlite3MPrintf(db, "CREATE VIRTUAL TABLE %T", >sNameToken); (gdb) p pParse->sNameToken $12 = { z = 0x2c3fffda "vtb_enyqkyxs USING vtable_module_343", n = 41 } (gdb) p pParse->sNameToken.z + 40 $13 = 0x2c42 As you can see, token size is invalid: 41. This causes memcpy() down at #0 to access invalid, unmapped address. Hopefully it is only read overflow so the only consequence is just an occasional Segmentation fault when allocated piece of string is at the specific place: near the end of the page in front of unmapped page. Should I fill a bug report? It's certainly very suspicious. Which SQLite version are you using? Dan. Is there a way to work this around? Like append many spaces a the end of query? Or maybe the problem is in absence of ';' at the end of query? Meantime I'll try both of these cases. We observe very similar problem. #1 0x087ec9f7 in sqlite3VXPrintf () #2 0x087f816d in sqlite3MPrintf () #3 0x088781e5 in sqlite3VtabFinishParse () #4 0x0885190f in yy_reduce () #5 0x0884d4d8 in sqlite3Parser () #6 0x087fc0ce in sqlite3RunParser () #7 0x088aa396 in sqlite3Prepare () #8 0x087fae18 in sqlite3LockAndPrepare () #9 0x087f9a88 in sqlite3_exec () #10 0x086588a7 in sqlite::StorageBase::exec_query (this=0x2c2a47c0, query=0x2c7fffc0 "CREATE VIRTUAL TABLE temp.vtb_wayxzmbo USING vtable_module_344", quiet=false) at SqliteStorageBase.cpp:286 It always crashes when "CREATE VIRTUAL TABLE ..." is being executed, always with the same backtrace. I spent many days reviewing and testing my code to eliminate possible cause but so far I see nothing wrong with it. Probability of this crash is so very low so that problem can be reproduced only on hi loaded production servers. (Where such virtual tables are created and dropped millions of times during a day) I am going to compile sqlite without optimizations and with debug
Re: [sqlite] Read overflow in CREATE VIRTUAL TABLE query (formerly Corrupting pointers to the lookaside smallacator)
Currently we use various versions of SQLite: SQLite version 3.8.0.1 2013-08-29 17:35:01 SQLite version 3.8.2 2013-12-06 14:53:30 SQLite version 3.8.6 2014-08-15 11:46:33 SQLite version 3.8.7 2014-10-17 11:24:17 All of them are affected so I never considered it to be an sqlite bug. But analyzing core file it seems like very much an sqlite bug :/ Tell me if you need more info on this. Thanks. > On 11/27/2014 03:20 PM, Paul wrote: > > Here is how it looks with debug symbols are on: > > > > #0 0x28c4113e in memcpy () from /lib/libc.so.7 > > #1 0x08854c20 in sqlite3StrAccumAppend (p=0xfffe8548, z=0x2c3fffda > > "vtb_enyqkyxs USING vtable_module_343", N=41) at sqlite3.c:21563 > > #2 0x087edf30 in sqlite3VXPrintf (pAccum=0xfffe8548, bFlags=1, > > fmt=0x892e543 "T", ap=0xfffe8610 "") at sqlite3.c:21439 > > #3 0x088077d5 in sqlite3VMPrintf (db=0x2c006788, zFormat=0x892e52d "CREATE > > VIRTUAL TABLE %T", ap=0xfffe860c "xx") at sqlite3.c:21638 > > #4 0x087f815e in sqlite3MPrintf (db=0x2c006788, zFormat=0x892e52d "CREATE > > VIRTUAL TABLE %T") at sqlite3.c:21654 > > #5 0x088759af in sqlite3VtabFinishParse (pParse=0x2c007688, pEnd=0x0) at > > sqlite3.c:112383 > > #6 0x0885bb21 in yy_reduce (yypParser=0x2c1d1408, yyruleno=309) at > > sqlite3.c:123403 > > #7 0x08856180 in sqlite3Parser (yyp=0x2c1d1408, yymajor=1, yyminor=..., > > pParse=0x2c007688) at sqlite3.c:123629 > > #8 0x087fc289 in sqlite3RunParser (pParse=0x2c007688, zSql=0x2c3fffc0 > > "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", > > pzErrMsg=0xfffe8bc4) at sqlite3.c:124466 > > #9 0x088bbc82 in sqlite3Prepare (db=0x2c006788, zSql=0x2c3fffc0 "CREATE > > VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", nBytes=-1, > > saveSqlFlag=1, pReprepare=0x0, ppStmt=0xfffe8d0c, pzTail=0xfffe8d10) at > > sqlite3.c:103750 > > #10 0x087fb0ce in sqlite3LockAndPrepare (db=0x2c006788, zSql=0x2c3fffc0 > > "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", > > nBytes=-1, saveSqlFlag=1, pOld=0x0, ppStmt=0xfffe8d0c, pzTail=0xfffe8d10) > > at sqlite3.c:103842 > > #11 0x087fa504 in sqlite3_prepare_v2 (db=0x2c006788, zSql=0x2c3fffc0 > > "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", > > nBytes=-1, ppStmt=0xfffe8d0c, pzTail=0xfffe8d10) at sqlite3.c:103918 > > #12 0x087fa01d in sqlite3_exec (db=0x2c006788, zSql=0x2c3fffc0 "CREATE > > VIRTUAL TABLE temp.vtb_enyqkyxs USING vtable_module_343", xCallback=0x0, > > pArg=0x0, pzErrMsg=0x0) at sqlite3.c:99345 > > #13 0x086588e7 in sqlite::StorageBase::exec_query (this=0x2c3ff640, > > query=0x2c3fffc0 "CREATE VIRTUAL TABLE temp.vtb_enyqkyxs USING > > vtable_module_343", quiet=false) at SqliteStorageBase.cpp:286 > > > > > > Interesting part is in frame #5 > > > > #5 0x088759af in sqlite3VtabFinishParse (pParse=0x2c007688, pEnd=0x0) at > > sqlite3.c:112383 > > 112383 zStmt = sqlite3MPrintf(db, "CREATE VIRTUAL TABLE %T", > > >sNameToken); > > (gdb) p pParse->sNameToken > > $12 = { > > z = 0x2c3fffda "vtb_enyqkyxs USING vtable_module_343", > > n = 41 > > } > > (gdb) p pParse->sNameToken.z + 40 > > $13 = 0x2c42 > > > > As you can see, token size is invalid: 41. This causes memcpy() down at #0 > > to access invalid, unmapped address. > > Hopefully it is only read overflow so the only consequence is just an > > occasional Segmentation fault when allocated > > piece of string is at the specific place: near the end of the page in front > > of unmapped page. > > > > Should I fill a bug report? > > It's certainly very suspicious. Which SQLite version are you using? > > Dan. > > > > > > > > > Is there a way to work this around? > > > > Like append many spaces a the end of query? > > Or maybe the problem is in absence of ';' at the end of query? > > Meantime I'll try both of these cases. > > > > > > > >> We observe very similar problem. > >> > >> #1 0x087ec9f7 in sqlite3VXPrintf () > >> #2 0x087f816d in sqlite3MPrintf () > >> #3 0x088781e5 in sqlite3VtabFinishParse () > >> #4 0x0885190f in yy_reduce () > >> #5 0x0884d4d8 in sqlite3Parser () > >> #6 0x087fc0ce in sqlite3RunParser () > >> #7 0x088aa396 in sqlite3Prepare () > >> #8 0x087fae18 in sqlite3LockAndPrepare () > >> #9 0x087f9a88 in sqlite3_exec () > >> #10 0x086588a7 in sqlite::StorageBase::exec_query (this=0x2c2a47c0, > >> query=0x2c7fffc0 "CREATE VIRTUAL TABLE temp.vtb_wayxzmbo USING > >> vtable_module_344", quiet=false) at SqliteStorageBase.cpp:286 > >> > >> It always crashes when "CREATE VIRTUAL TABLE ..." is being executed, > >> always with the same backtrace. > >> I spent many days reviewing and testing my code to eliminate possible > >> cause but so far I see nothing wrong with it. > >> Probability of this crash is so very low so that problem can be reproduced > >> only on hi loaded production servers. > >> (Where such virtual tables are created and dropped millions of times > >> during a day) > >> > >> I am going to compile sqlite without optimizations and with debug symbols > >>