On 04/29/2014 03:53 AM, sql...@charles.derkarl.org wrote:
I didn't know this list strips attachments, so the source file is here:
http://derkarl.org/~charles/runlsm.cpp
Thanks for this. It is a problem.
LSM accumulates data in a tree structure in shared-memory until there is
"enough" (~1-2 MB) to flush through to the database file. But at the
moment, it can only flush data to the db file between transactions. And
the in-memory tree can only hold 2GB of data (including overhead). So
things fail if a single transaction exceeds that limit. In the short
term, it should be changed to return LSM_FULL for any transaction too
large to handle. But the real fix should be to change things so that LSM
can begin flushing data to the database file mid-transaction.
Dan.
On Monday, April 28, 2014 01:41:02 PM sql...@charles.derkarl.org wrote:
Hi,
I'm not exactly sure this is the right forum for my problem, as I know that
LSM is experimental, but here we go.
I tried loading a whole lot of data using LSM. The majority of the data
goes int a single huge transaction (begin/commit pair). My program
segfaults once we're 1.61GB into my data file.
I have attached the source code to my test program. You also will need my
data file, which is too big for email:
http://www.derkarl.org/~charles/lsmlog.bz2 (744 MiB)
Here is how you can run my test program:
bunzip2 < lsmlog.bz2 | pv | ./a.out lsmdbtocreate
(You can exclude "pv" from the pipeline if you don't have it installed)
Here is the backtrace:
treeShmalloc (pDb=pDb@entry=0x12b20a8, bAlign=bAlign@entry=1,
nByte=nByte@entry=12, pRc=pRc@entry=0x7fff2fd43f44)
at src/lsm_tree.c:682
682 pNext->iNext = 0;
(gdb) bt
#0 treeShmalloc (pDb=pDb@entry=0x12b20a8, bAlign=bAlign@entry=1,
nByte=nByte@entry=12, pRc=pRc@entry=0x7fff2fd43f44)
at src/lsm_tree.c:682
#1 0x000000000041122d in treeShmallocZero (pDb=pDb@entry=0x12b20a8,
nByte=nByte@entry=12, piPtr=piPtr@entry=0x7fff2fd43f4c,
pRc=pRc@entry=0x7fff2fd43f44) at src/lsm_tree.c:711
#2 0x0000000000413114 in newTreeLeaf (pRc=0x7fff2fd43f44,
piPtr=0x7fff2fd43f4c, pDb=0x12b20a8) at src/lsm_tree.c:726
#3 treeInsertLeaf (iSlot=1, iTreeKey=2146172860, pCsr=0x7fff2fd43f50,
pDb=0x12b20a8) at src/lsm_tree.c:1039
#4 treeInsertEntry (pDb=pDb@entry=0x12b20a8, flags=8,
pKey=pKey@entry=0x12b2058, nKey=nKey@entry=17, pVal=<optimized out>,
pVal@entry=0x12bb638, nVal=21) at src/lsm_tree.c:1552
#5 0x000000000041329f in lsmTreeInsert (pDb=pDb@entry=0x12b20a8,
pKey=pKey@entry=0x12b2058, nKey=nKey@entry=17,
pVal=pVal@entry=0x12bb638, nVal=<optimized out>) at src/lsm_tree.c:1587
#6 0x0000000000404db0 in doWriteOp (pDb=0x12b20a8, bDeleteRange=<optimized
out>, pKey=0x12b2058, nKey=17, pVal=0x12bb638,
nVal=<optimized out>) at src/lsm_main.c:696
#7 0x000000000040305d in main (argc=2, argv=0x7fff2fd44418) at
runlsm.cpp:41 (gdb) print pNext
$1 = (ShmChunk *) 0xffffffffffffffff
It's a pity that LSM isn't ready for production, because if the quality of
sqlite3 is indication, I'm going to really enjoy using it!
Charles
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
