Re: [sqlite] multiple writers for in-memory datastore

2008-04-21 Thread James Gregurich
interesting. thanks for the tip. I"ll give it some consideration. -James On Apr 21, 2008, at 1:07 :50PM, Scott Hess wrote: > If you create a file on disk and set PRAGMA synchronous = OFF, you > should get pretty close to the performance of a shared in-memory > database on most modern desktop

Re: [sqlite] multiple writers for in-memory datastore

2008-04-21 Thread Scott Hess
If you create a file on disk and set PRAGMA synchronous = OFF, you should get pretty close to the performance of a shared in-memory database on most modern desktop operating systems - maybe close enough that you won't care to do anything beyond that. If you further look at the recent

Re: [sqlite] multiple writers for in-memory datastore

2008-04-20 Thread James Gregurich
yes. However, CoreData queues up modified managed objects in a managed object context and then commits them all in one shot making sure the serialization is done on the back side. So, it does basically what someone here recommended earlier. I just don't have to write the mechanism myself.

Re: [sqlite] multiple writers for in-memory datastore

2008-04-20 Thread Dennis Cote
James Gregurich wrote: > I think I will go with CoreData on MacOSX and figure out something > else to do on Windows later. > > > You do know that CoreData uses SQLite for its persistant storage. Dennis Cote ___ sqlite-users mailing list

Re: [sqlite] multiple writers for in-memory datastore

2008-04-20 Thread Dan
On Apr 20, 2008, at 12:29 AM, James Gregurich wrote: > > oh good! That isn't the version that ships with Leopard, but I can > live with deploying my own version as part of my app. > > Will l get the writer parallelism I'm after as long as each thread > writes exclusively into its own attached

Re: [sqlite] multiple writers for in-memory datastore

2008-04-20 Thread James Gregurich
for those who may be interested: I ran a test with SQLite version: 3.5.8 I tried the scheme described earlier with each thread sharing a connection but writing into its own attached in-memory db on that connection. Didn't work. all but the first writer thread failed with a SQLITE_ERROR

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread James Gregurich
new file in vfs, a new handler is created and > assigned to > that filename and registered in this map. > > > > > > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of James Gregurich > Sent: sábado, 19 de abril de 2008 17:02 > T

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread Virgilio Alexandre Fornazin
] On Behalf Of James Gregurich Sent: sábado, 19 de abril de 2008 17:02 To: General Discussion of SQLite Database Subject: Re: [sqlite] multiple writers for in-memory datastore I don't immediately see how that would solve the problem. The limitation of interest here (based on my perhaps limited

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread James Gregurich
I don't immediately see how that would solve the problem. The limitation of interest here (based on my perhaps limited understanding) is that locking has file-level granularity. I don't immediately see how a VST implementation would allow for changing the locking granularity of the overall

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread Virgilio Fornazin
what about creating a VFS for such task ? Can be accomplished in many ways, using heap memory, shared memory... not so easy to do, but not much complicated too... locking can be provided by multiple-readers single-writers locks strategies, etc... On Sat, Apr 19, 2008 at 2:29 PM, James Gregurich

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread James Gregurich
oh good! That isn't the version that ships with Leopard, but I can live with deploying my own version as part of my app. Will l get the writer parallelism I'm after as long as each thread writes exclusively into its own attached db? in other wordstwo bulk insert operations going on

Re: [sqlite] multiple writers for in-memory datastore

2008-04-19 Thread Dan
On Apr 19, 2008, at 6:06 AM, James Gregurich wrote: > > I'll ask this question. The answer is probably "no," but I'll ask it > for the sake of completeness. > > > Suppose I created an in-memory db. I use the attach command to > associate an additional in-memory db. Suppose I assign the main db

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread James Gregurich
I'll ask this question. The answer is probably "no," but I'll ask it for the sake of completeness. Suppose I created an in-memory db. I use the attach command to associate an additional in-memory db. Suppose I assign the main db to thread 1 and the associated db to thread 2. Can I share

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread Ken
How about this instead. Read your records, parse and format into some known format by your application. Write the data to disk in a file. Then put a single entry into a sqlite table. specifing the on disk file name. Sqlite may only have one write operation running concurrently. There are

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread James Gregurich
On Apr 18, 2008, at 2:33 :32PM, Dennis Cote wrote: > > To share an attached database the threads must be able to name it, and > this is only possible with a file database. you could change the open() function to be able to assign a name to an in-memory db and then keep a mapping of all the

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread Dennis Cote
James Gregurich wrote: >> You will have to open the memory database and attach the db file since >> SQLite can't attach to a memory database. > > is this information wrong? > http://www.blitzbasic.com/Community/posts.php?topic=60981 > No it's not. I was mistaken. SQLite can attach a memory

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread James Gregurich
On Apr 18, 2008, at 1:25 :36PM, Dennis Cote wrote: > James Gregurich wrote: >> >> suppose I create a temporary db file on disk. Each task ( a thread) >> opens a connection to the temp file and attaches an in-memory db to >> it. > > You will have to open the memory database and attach the db file

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread Dennis Cote
James Gregurich wrote: > > suppose I create a temporary db file on disk. Each task ( a thread) > opens a connection to the temp file and attaches an in-memory db to > it. You will have to open the memory database and attach the db file since SQLite can't attach to a memory database. I

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread James Gregurich
I'm working on a commercial, boxed, desktop product. I can't be creating new mounted disks on a customer's system every time he uses my application. How about this... suppose I create a temporary db file on disk. Each task ( a thread) opens a connection to the temp file and attaches

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread Dennis Cote
James Gregurich wrote: > If the sqlite statement had a temporary storage area so that I could > load up a bunch of rows and then commit them in one shot so that the > lock on the db was not held very long by a single transaction, that > would probably work. > Using a RAM disk you could

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread James Gregurich
If the sqlite statement had a temporary storage area so that I could load up a bunch of rows and then commit them in one shot so that the lock on the db was not held very long by a single transaction, that would probably work. However, my reading of the documentation leads me to believe

Re: [sqlite] multiple writers for in-memory datastore

2008-04-18 Thread Dennis Cote
James Gregurich wrote: > > I need to set up multiple writers to an in-memory datastore. I just > discovered that you can't have more than one connection to an in- > memory store. > > I can give each task its own independent datastore if there is a way I > can merge the contents of each

[sqlite] multiple writers for in-memory datastore

2008-04-18 Thread James Gregurich
hi! I need to set up multiple writers to an in-memory datastore. I just discovered that you can't have more than one connection to an in- memory store. I can give each task its own independent datastore if there is a way I can merge the contents of each store into a central store. Is there