Re: [sqlite] LSM bug

[Dan Kennedy]

On 04/29/2014 03:53 AM, sql...@charles.derkarl.org wrote:
I didn't know this list strips attachments, so the source file is here:
http://derkarl.org/~charles/runlsm.cpp

Thanks for this. It is a problem.

LSM accumulates data in a tree structure in shared-memory until there is 
"enough" (~1-2 MB) to flush through to the database file. But at the 
moment, it can only flush data to the db file between transactions. And 
the in-memory tree can only hold 2GB of data (including overhead). So 
things fail if a single transaction exceeds that limit. In the short 
term, it should be changed to return LSM_FULL for any transaction too 
large to handle. But the real fix should be to change things so that LSM 
can begin flushing data to the database file mid-transaction.

Dan.






On Monday, April 28, 2014 01:41:02 PM sql...@charles.derkarl.org wrote:
Hi,

I'm not exactly sure this is the right forum for my problem, as I know that
LSM is experimental, but here we go.

I tried loading a whole lot of data using LSM. The majority of the data
goes int a single huge transaction (begin/commit pair). My program
segfaults once we're 1.61GB into my data file.

I have attached the source code to my test program. You also will need my
data file, which is too big for email:

http://www.derkarl.org/~charles/lsmlog.bz2 (744 MiB)

Here is how you can run my test program:

bunzip2 < lsmlog.bz2 | pv | ./a.out lsmdbtocreate

(You can exclude "pv" from the pipeline if you don't have it installed)

Here is the backtrace:

treeShmalloc (pDb=pDb@entry=0x12b20a8, bAlign=bAlign@entry=1,
nByte=nByte@entry=12, pRc=pRc@entry=0x7fff2fd43f44)
     at src/lsm_tree.c:682
682             pNext->iNext = 0;
(gdb) bt
#0  treeShmalloc (pDb=pDb@entry=0x12b20a8, bAlign=bAlign@entry=1,
nByte=nByte@entry=12, pRc=pRc@entry=0x7fff2fd43f44)
     at src/lsm_tree.c:682
#1  0x000000000041122d in treeShmallocZero (pDb=pDb@entry=0x12b20a8,
nByte=nByte@entry=12, piPtr=piPtr@entry=0x7fff2fd43f4c,
     pRc=pRc@entry=0x7fff2fd43f44) at src/lsm_tree.c:711
#2  0x0000000000413114 in newTreeLeaf (pRc=0x7fff2fd43f44,
piPtr=0x7fff2fd43f4c, pDb=0x12b20a8) at src/lsm_tree.c:726
#3  treeInsertLeaf (iSlot=1, iTreeKey=2146172860, pCsr=0x7fff2fd43f50,
pDb=0x12b20a8) at src/lsm_tree.c:1039
#4  treeInsertEntry (pDb=pDb@entry=0x12b20a8, flags=8,
pKey=pKey@entry=0x12b2058, nKey=nKey@entry=17, pVal=<optimized out>,
     pVal@entry=0x12bb638, nVal=21) at src/lsm_tree.c:1552
#5  0x000000000041329f in lsmTreeInsert (pDb=pDb@entry=0x12b20a8,
pKey=pKey@entry=0x12b2058, nKey=nKey@entry=17,
     pVal=pVal@entry=0x12bb638, nVal=<optimized out>) at src/lsm_tree.c:1587
#6  0x0000000000404db0 in doWriteOp (pDb=0x12b20a8, bDeleteRange=<optimized
out>, pKey=0x12b2058, nKey=17, pVal=0x12bb638,
     nVal=<optimized out>) at src/lsm_main.c:696
#7  0x000000000040305d in main (argc=2, argv=0x7fff2fd44418) at
runlsm.cpp:41 (gdb) print pNext
$1 = (ShmChunk *) 0xffffffffffffffff



It's a pity that LSM isn't ready for production, because if the quality of
sqlite3 is indication, I'm going to really enjoy using it!

Charles
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users