Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-08 Thread John Stanton
A valid backup needs to be a snapshot, It is of no value if other processes were able to modify the file as it was being backed up. An exclusive (write) lock of some type is required or else the file needs to be organized differently. Neville Franks wrote: > A proper backup program that opene

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-08 Thread Stan Bielski
ers-boun...@sqlite.org > [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Mark Spiegel > Sent: Wednesday, July 08, 2009 2:02 PM > To: General Discussion of SQLite Database > Subject: Re: [sqlite] Repost: Accessing a DB while copying it causes Windows > to eat virtual memory >

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-08 Thread Robert Simpson
Behalf Of Mark Spiegel Sent: Wednesday, July 08, 2009 2:02 PM To: General Discussion of SQLite Database Subject: Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory Stan Bielski wrote: > Hello again, > > Copying the database with Explorer and accessi

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-08 Thread Mark Spiegel
Stan Bielski wrote: > Hello again, > > Copying the database with Explorer and accessing it via sqlite is just > a contrived example that exhibits the same problem I'm having in my > application. The app does make a copy of the DB, but it has app-layer > locking that will prevent modifications unle

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-08 Thread Stan Bielski
Hello again, Copying the database with Explorer and accessing it via sqlite is just a contrived example that exhibits the same problem I'm having in my application. The app does make a copy of the DB, but it has app-layer locking that will prevent modifications unless someone decides to start fidd

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-08 Thread Mark Spiegel
> Just to be clear, I think Windows is really the one that is at fault > here; the behavior seems very broken to me. Unfortunately, I need the > app to work on Windows without this happening, so I need to figure out > some kind of workaround in a.) Windows settings b.) the sqlite source > or c.) h

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-08 Thread Simon Slavin
On 8 Jul 2009, at 9:48am, Neville Franks wrote: > A proper backup program that opened the file for r/o, non-exclusive > use may be able to copy the file. But under many operating systems it runs the risk of copying the first half of the file, then allowing a change to be made, then copying the

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-08 Thread Neville Franks
A proper backup program that opened the file for r/o, non-exclusive use may be able to copy the file. I thought SQLite has a live backup capability now. Surely that is the best way to handle this. Wednesday, July 8, 2009, 6:16:41 PM, you wrote: JS> It looks like you are trying to copy from a pr

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-08 Thread John Stanton
It looks like you are trying to copy from a process other than the one which holds the lock. Think abnout it - the lock gives exlusive access to the file to the process which sets it and blocks all other processes. Stan Bielski wrote: > Just to make sure I understood you correctly, is this what

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-07 Thread Marcus Grimm
I might missed that but: what are you trying to acomplish by using explorer to copy the database file ? I guess you are doing it for backup purpose. For this, be aware that it might be dangerous to do a simple file copy on a running database since you might forget some temporary files used by sqlit

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-07 Thread Stan Bielski
On Tue, Jul 7, 2009 at 1:10 PM, Mark Spiegel wrote: > This should give you a picture of who is opening the file and with what > flags.  In particular, how is your SQLite app and Explorer opening the file? Thanks for the pointer to Filemon; I'm always happy to learn about a new tool. Apparently it

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-07 Thread Stan Bielski
Just to make sure I understood you correctly, is this what you are suggesting? * Open file with sqlite app (in my case sqlite3) * execute BEGIN EXCLUSIVE TRANSACTION; * initiate the file copy * COMMIT; after the copy is finished. I tried doing this, but very early into the copy Windows issues the

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-07 Thread Mark Spiegel
Stan Bielski wrote: > In the course of copying a largish (20 GB) database file while > accessing it via sqlite3, the machine became very unresponsive. I > opened task manager and found that the system was using a huge amount > of virtual memory, causing it to thrash. Per-process memory usage > look

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-07 Thread John Stanton
You should synchronize your backup (copy). Try surrounding it with an exclusive transaction. Stan Bielski wrote: > Sorry for the repost, but the original thread was hijacked by another > list user. This is a serious problem IMHO; it looks like the DB can't > be backed-up without rendering the ma

Re: [sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-07 Thread Roberto
Haven't seen your previous post but just a WAG, have you selected the VM Size column in task manager to show you per process VM usage. In XP TM, it is under Tools Menu/Select Columns/Virtual Memory Size. This should indicate which process is hogging VM. 2009/7/7 Stan Bielski : > Sorry for the repo

[sqlite] Repost: Accessing a DB while copying it causes Windows to eat virtual memory

2009-07-07 Thread Stan Bielski
Sorry for the repost, but the original thread was hijacked by another list user. This is a serious problem IMHO; it looks like the DB can't be backed-up without rendering the machine unusable if a query hits it while a copy is in progress. Hello, In the course of copying a largish (20 GB) databas