Re: [sqlite] Shared in-memory SQLite database in shared memory

[Howard Chu]

Jaco Breitenbach wrote:

Dear all,

My application consists of several indepent processes that must all access
(read) the same data table during processing.  In order to optimise memory
usage I was wondering if it is possible to load an in-memory SQLite
database into shared memory.  The database would be maintained by a
separate management process and all other processes would only require read
access.  This way only one copy of the database would have to be held in
system memory.

Is this possible with the current implementation of SQLite?  Any
suggestions would be welcome.


The OpenLDAP MDB (memory mapped database) library will do exactly what you 
want. Since it uses a shared memory map to access the DB, no matter how many 
processes access it concurrently there's only one copy of data present in RAM. 
It also performs reads faster than anything else, even pure in-memory 
databases. Nothing else is anywhere close to as efficient as MDB for reads.


Read more here http://highlandsun.com/hyc/mdb/

The port of SQLite using MDB as its backend is available on gitorious
https://gitorious.org/mdb/

--
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Shared in-memory SQLite database in shared memory

[Simon Slavin]

On 3 Nov 2012, at 3:26pm, Jaco Breitenbach  wrote:

> In my application I have multiple separate processes, not threads.

Sorry, I missed that.  You would seem to need some sort of sharing memory 
between your processes.  Don't know how to do that.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Shared in-memory SQLite database in shared memory

[Roger Andersson]

On 11/03/12 16:26, Jaco Breitenbach wrote:

I wanted to have the database in shared memory.

Maybe a ram drive?
What operating system are you running?

Cheers
Roger
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Shared in-memory SQLite database in shared memory

[Jaco Breitenbach]

Hi Simon,

In my application I have multiple separate processes, not threads.  I
believe the shared cache doesn't work across processes.  I essentially want
a single shared read-only database that can be accessed at high speed.
 Since the data set can become quite large and with multiple processes
requiring access to the identical data, I wanted to have the database in
shared memory.   The database will contain reference information with which
the data I'm processing must be enriched.  So once the database has been
loaded in memory, it will remain static and its contents will never change.

Best regards,
Jaco

On Saturday, November 3, 2012, Simon Slavin wrote:

>
> On 3 Nov 2012, at 12:26pm, Jaco Breitenbach 
> <jjbreitenb...@gmail.com<javascript:;>>
> wrote:
>
> > My application consists of several indepent processes that must all
> access
> > (read) the same data table during processing.  In order to optimise
> memory
> > usage I was wondering if it is possible to load an in-memory SQLite
> > database into shared memory.
>
> Not simply and not without a little strangeness.  However, SQLite has a
> Shared Cache mode which is easy to use, and you can open your database on
> disk, but declare a really big cache size for it.  This will remove the
> need to use an in-memory database since SQLite will gradually read the
> whole database in to the cache.  Read this:
>
> <http://www.sqlite.org/sharedcache.html>
>
> and this:
>
> <http://www.sqlite.org/pragma.html#pragma_cache_size>
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org <javascript:;>
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>


-- 
Sent from Gmail Mobile
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Shared in-memory SQLite database in shared memory

[Simon Slavin]

On 3 Nov 2012, at 12:26pm, Jaco Breitenbach <jjbreitenb...@gmail.com> wrote:

> My application consists of several indepent processes that must all access
> (read) the same data table during processing.  In order to optimise memory
> usage I was wondering if it is possible to load an in-memory SQLite
> database into shared memory.

Not simply and not without a little strangeness.  However, SQLite has a Shared 
Cache mode which is easy to use, and you can open your database on disk, but 
declare a really big cache size for it.  This will remove the need to use an 
in-memory database since SQLite will gradually read the whole database in to 
the cache.  Read this:

<http://www.sqlite.org/sharedcache.html>

and this:

<http://www.sqlite.org/pragma.html#pragma_cache_size>

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users