Re: [sqlite] license to hack: pager in shmem
- Original Message - From: "D. Richard Hipp" <[EMAIL PROTECTED]> To: <sqlite-users@sqlite.org> Sent: Friday, June 17, 2005 9:42 PM Subject: Re: [sqlite] license to hack: pager in shmem > On Fri, 2005-06-17 at 21:21 -0400, Paul G wrote: > > i might put in some work on this, although it's a bit more work than i'd > > hoped (shm would have been faster afair). will this patch, assuming it looks > > decent and works (for a reasonable threshold of 'works'), be accepted into > > trunk? it's a bit too much to do just for myself, but assuming others would > > benefit as well, i might be up for it. > > > > Anything that increases the library footprint needs to be > surrounded with #ifndef SQLITE_OMIT_ already in place in my working copy. > See also http://www.sqlite.org/copyright.html no issue there. i'll take your mail as a yes. is there a separate -dev list i can moot my questions (if any) to or are they on-topic for this one? -p
Re: [sqlite] license to hack: pager in shmem
On Fri, 2005-06-17 at 21:21 -0400, Paul G wrote: > i might put in some work on this, although it's a bit more work than i'd > hoped (shm would have been faster afair). will this patch, assuming it looks > decent and works (for a reasonable threshold of 'works'), be accepted into > trunk? it's a bit too much to do just for myself, but assuming others would > benefit as well, i might be up for it. > Anything that increases the library footprint needs to be surrounded with #ifndef SQLITE_OMIT_ There are a lot of embedded users who care a great deal about footprint. See also http://www.sqlite.org/copyright.html -- D. Richard Hipp <[EMAIL PROTECTED]>
Re: [sqlite] license to hack: pager in shmem
richard, comments inline: - Original Message - From: "D. Richard Hipp" <[EMAIL PROTECTED]> To: <sqlite-users@sqlite.org> Sent: Friday, June 17, 2005 8:54 PM Subject: Re: [sqlite] license to hack: pager in shmem > On Fri, 2005-06-17 at 20:16 -0400, Paul G wrote: > > i'd like to hack sqlite3 to use shared memory for the page cache. > > > > A more useful hack, perhaps, would be to change the pager to use > memmap() (when available) instead of read() and write(). This i'm going to assume you meant mmap() ;] indeed, i've already looked at doing this (and have the beginnings of an implementation) but was put off by the intrusiveness of the change. > The downside: You'll need to change the pager API around some, which > means changes to the BTree layer. The current pager allocates some > extra memory immediately before and after the data page. And it tracks > each page using just a pointer to the data - since it can always find > the header and tailer using pointer arithmetic. That won't work with > memmap(). You'd have to relocate this extra stuff into a separate > allocation which you then track separately. right. when i looked at it, it just didn't seem like it would be all that time-efficient (wrt implementation work) to decouple the page header and EXTRA from the page itself. > Also, you'll have to take > out an exclusive lock on the database before making any changes to the > data buffers in memory, not just before calling write() as in the > current > solution. That will cut your currency some because it means taking an > exclusive lock much earlier in the write process. personally, i don't specifically care, since my usage pattern will be heavily trended towards reads (if i do any writes at all). however, this is indeed a valid concern. i'm sure something can be thought up here if there's user demand, but i don't think it makes sense to deal with this from the start (premature optimization and all that). > A new pager API that works for memmap() would also continue to work > for the old read()/write() method. So presumably you could create > a new pragma that lets the programmer decide at run-time which method > to use. Use memmap() for faster reads and less memory usage or > use read/write for better concurrency. i was actually going to do something like ":mmap:/path/to/db", in keeping with the :memory: feature, since this whole excercise started with me trying to determine whether i can tweak the :memory: implementation to do what i want. i might put in some work on this, although it's a bit more work than i'd hoped (shm would have been faster afair). will this patch, assuming it looks decent and works (for a reasonable threshold of 'works'), be accepted into trunk? it's a bit too much to do just for myself, but assuming others would benefit as well, i might be up for it. cheers, -p
Re: [sqlite] license to hack: pager in shmem
On Fri, 2005-06-17 at 20:16 -0400, Paul G wrote: > i'd like to hack sqlite3 to use shared memory for the page cache. > A more useful hack, perhaps, would be to change the pager to use memmap() (when available) instead of read() and write(). This allows multiple instances of SQLite to use the same cache, essentially. The various pages might be mapped to different places in each process address space, but they would all point to the same physical page of memory. This would likely also make SQLite much faster even for the general case of just a single process since it would avoid the constant copying of information from kernel space to user space and back again. And you would still be able to write. The downside: You'll need to change the pager API around some, which means changes to the BTree layer. The current pager allocates some extra memory immediately before and after the data page. And it tracks each page using just a pointer to the data - since it can always find the header and tailer using pointer arithmetic. That won't work with memmap(). You'd have to relocate this extra stuff into a separate allocation which you then track separately. Also, you'll have to take out an exclusive lock on the database before making any changes to the data buffers in memory, not just before calling write() as in the current solution. That will cut your currency some because it means taking an exclusive lock much earlier in the write process. A new pager API that works for memmap() would also continue to work for the old read()/write() method. So presumably you could create a new pragma that lets the programmer decide at run-time which method to use. Use memmap() for faster reads and less memory usage or use read/write for better concurrency. -- D. Richard Hipp <[EMAIL PROTECTED]>
