Re: [sqlite] LSM bug
On Tuesday, April 29, 2014 03:38:57 AM Dan Kennedy wrote: > On 04/29/2014 03:53 AM, sql...@charles.derkarl.org wrote: > > I didn't know this list strips attachments, so the source file is here: > > > > http://derkarl.org/~charles/runlsm.cpp > > Thanks for this. It is a problem. > > LSM accumulates data in a tree structure in shared-memory until there is > "enough" (~1-2 MB) to flush through to the database file. But at the > moment, it can only flush data to the db file between transactions. And > the in-memory tree can only hold 2GB of data (including overhead). So > things fail if a single transaction exceeds that limit. In the short > term, it should be changed to return LSM_FULL for any transaction too > large to handle. But the real fix should be to change things so that LSM > can begin flushing data to the database file mid-transaction. I'm also seeing a similar problem in which it silently discards entries, but I haven't been able to narrow down an example for you. Let me know if that would be helpful and I'll try harder. What could I do to improve LSM? Charles ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM bug
On 04/29/2014 03:53 AM, sql...@charles.derkarl.org wrote: I didn't know this list strips attachments, so the source file is here: http://derkarl.org/~charles/runlsm.cpp Thanks for this. It is a problem. LSM accumulates data in a tree structure in shared-memory until there is "enough" (~1-2 MB) to flush through to the database file. But at the moment, it can only flush data to the db file between transactions. And the in-memory tree can only hold 2GB of data (including overhead). So things fail if a single transaction exceeds that limit. In the short term, it should be changed to return LSM_FULL for any transaction too large to handle. But the real fix should be to change things so that LSM can begin flushing data to the database file mid-transaction. Dan. On Monday, April 28, 2014 01:41:02 PM sql...@charles.derkarl.org wrote: Hi, I'm not exactly sure this is the right forum for my problem, as I know that LSM is experimental, but here we go. I tried loading a whole lot of data using LSM. The majority of the data goes int a single huge transaction (begin/commit pair). My program segfaults once we're 1.61GB into my data file. I have attached the source code to my test program. You also will need my data file, which is too big for email: http://www.derkarl.org/~charles/lsmlog.bz2 (744 MiB) Here is how you can run my test program: bunzip2 < lsmlog.bz2 | pv | ./a.out lsmdbtocreate (You can exclude "pv" from the pipeline if you don't have it installed) Here is the backtrace: treeShmalloc (pDb=pDb@entry=0x12b20a8, bAlign=bAlign@entry=1, nByte=nByte@entry=12, pRc=pRc@entry=0x7fff2fd43f44) at src/lsm_tree.c:682 682 pNext->iNext = 0; (gdb) bt #0 treeShmalloc (pDb=pDb@entry=0x12b20a8, bAlign=bAlign@entry=1, nByte=nByte@entry=12, pRc=pRc@entry=0x7fff2fd43f44) at src/lsm_tree.c:682 #1 0x0041122d in treeShmallocZero (pDb=pDb@entry=0x12b20a8, nByte=nByte@entry=12, piPtr=piPtr@entry=0x7fff2fd43f4c, pRc=pRc@entry=0x7fff2fd43f44) at src/lsm_tree.c:711 #2 0x00413114 in newTreeLeaf (pRc=0x7fff2fd43f44, piPtr=0x7fff2fd43f4c, pDb=0x12b20a8) at src/lsm_tree.c:726 #3 treeInsertLeaf (iSlot=1, iTreeKey=2146172860, pCsr=0x7fff2fd43f50, pDb=0x12b20a8) at src/lsm_tree.c:1039 #4 treeInsertEntry (pDb=pDb@entry=0x12b20a8, flags=8, pKey=pKey@entry=0x12b2058, nKey=nKey@entry=17, pVal=, pVal@entry=0x12bb638, nVal=21) at src/lsm_tree.c:1552 #5 0x0041329f in lsmTreeInsert (pDb=pDb@entry=0x12b20a8, pKey=pKey@entry=0x12b2058, nKey=nKey@entry=17, pVal=pVal@entry=0x12bb638, nVal=) at src/lsm_tree.c:1587 #6 0x00404db0 in doWriteOp (pDb=0x12b20a8, bDeleteRange=, pKey=0x12b2058, nKey=17, pVal=0x12bb638, nVal=) at src/lsm_main.c:696 #7 0x0040305d in main (argc=2, argv=0x7fff2fd44418) at runlsm.cpp:41 (gdb) print pNext $1 = (ShmChunk *) 0x It's a pity that LSM isn't ready for production, because if the quality of sqlite3 is indication, I'm going to really enjoy using it! Charles ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM bug
I didn't know this list strips attachments, so the source file is here: http://derkarl.org/~charles/runlsm.cpp On Monday, April 28, 2014 01:41:02 PM sql...@charles.derkarl.org wrote: > Hi, > > I'm not exactly sure this is the right forum for my problem, as I know that > LSM is experimental, but here we go. > > I tried loading a whole lot of data using LSM. The majority of the data > goes int a single huge transaction (begin/commit pair). My program > segfaults once we're 1.61GB into my data file. > > I have attached the source code to my test program. You also will need my > data file, which is too big for email: > > http://www.derkarl.org/~charles/lsmlog.bz2 (744 MiB) > > Here is how you can run my test program: > > bunzip2 < lsmlog.bz2 | pv | ./a.out lsmdbtocreate > > (You can exclude "pv" from the pipeline if you don't have it installed) > > Here is the backtrace: > > treeShmalloc (pDb=pDb@entry=0x12b20a8, bAlign=bAlign@entry=1, > nByte=nByte@entry=12, pRc=pRc@entry=0x7fff2fd43f44) > at src/lsm_tree.c:682 > 682 pNext->iNext = 0; > (gdb) bt > #0 treeShmalloc (pDb=pDb@entry=0x12b20a8, bAlign=bAlign@entry=1, > nByte=nByte@entry=12, pRc=pRc@entry=0x7fff2fd43f44) > at src/lsm_tree.c:682 > #1 0x0041122d in treeShmallocZero (pDb=pDb@entry=0x12b20a8, > nByte=nByte@entry=12, piPtr=piPtr@entry=0x7fff2fd43f4c, > pRc=pRc@entry=0x7fff2fd43f44) at src/lsm_tree.c:711 > #2 0x00413114 in newTreeLeaf (pRc=0x7fff2fd43f44, > piPtr=0x7fff2fd43f4c, pDb=0x12b20a8) at src/lsm_tree.c:726 > #3 treeInsertLeaf (iSlot=1, iTreeKey=2146172860, pCsr=0x7fff2fd43f50, > pDb=0x12b20a8) at src/lsm_tree.c:1039 > #4 treeInsertEntry (pDb=pDb@entry=0x12b20a8, flags=8, > pKey=pKey@entry=0x12b2058, nKey=nKey@entry=17, pVal=, > pVal@entry=0x12bb638, nVal=21) at src/lsm_tree.c:1552 > #5 0x0041329f in lsmTreeInsert (pDb=pDb@entry=0x12b20a8, > pKey=pKey@entry=0x12b2058, nKey=nKey@entry=17, > pVal=pVal@entry=0x12bb638, nVal=) at src/lsm_tree.c:1587 > #6 0x00404db0 in doWriteOp (pDb=0x12b20a8, bDeleteRange= out>, pKey=0x12b2058, nKey=17, pVal=0x12bb638, > nVal=) at src/lsm_main.c:696 > #7 0x0040305d in main (argc=2, argv=0x7fff2fd44418) at > runlsm.cpp:41 (gdb) print pNext > $1 = (ShmChunk *) 0x > > > > It's a pity that LSM isn't ready for production, because if the quality of > sqlite3 is indication, I'm going to really enjoy using it! > > Charles ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
