Re: [sqlite] memory wasted shm mapped file (3.7.2)

ok, I think I'll use the snapshot in the mean time. Thank you Il 16/09/2010 15.43, Richard Hipp ha scritto: > On Thu, Sep 16, 2010 at 9:02 AM, Michele Pradella< > michele.prade...@selea.com> wrote: > >> After some days of test the application work fine and with VMMap I see >> small value of

Re: [sqlite] memory wasted shm mapped file (3.7.2)

On Thu, Sep 16, 2010 at 9:02 AM, Michele Pradella < michele.prade...@selea.com> wrote: > After some days of test the application work fine and with VMMap I see > small value of mapped file and I see only 1 instance of -shm allocated, > some times 2 but it's all ok. So I'll wait for the next

Re: [sqlite] memory wasted shm mapped file (3.7.2)

After some days of test the application work fine and with VMMap I see small value of mapped file and I see only 1 instance of -shm allocated, some times 2 but it's all ok. So I'll wait for the next amalgamation release to have an official sources for that fix. When do you think will be

Re: [sqlite] memory wasted shm mapped file (3.7.2)

Ok I think that the latest patch I used to try the WAL is the patch that fix the problem. I do not get wasted memory by mapped file ;) good job. Anyway I leave the application running this day. If I have news I'll tell you. Just a question: when you plan to release a new version of sqlite

Re: [sqlite] memory wasted shm mapped file (3.7.2)

On Fri, Sep 10, 2010 at 6:32 PM, Shane Harrelson wrote: > I tried to reproduce this, and could not. > > There are some questions inline below.Additionally, I want to > verify that you've tried this with a version of SQLite containing the > previously linked fix. > > Shane,

Re: [sqlite] memory wasted shm mapped file (3.7.2)

I tried to reproduce this, and could not. There are some questions inline below.Additionally, I want to verify that you've tried this with a version of SQLite containing the previously linked fix. -Shane On Fri, Sep 10, 2010 at 12:54 AM, Max Vlasov wrote: >> But as a

Re: [sqlite] memory wasted shm mapped file (3.7.2)

The question is about the possibility, but I should be sure that this do not happen in my application, because I have to make my application running for month without restart. So because in first few test I always see the -shm Mapped File grow up (even that less than before the patch) and I

Re: [sqlite] memory wasted shm mapped file (3.7.2)

On Fri, Sep 10, 2010 at 5:07 PM, Michele Pradella < michele.prade...@selea.com> wrote: > > what I worry about is that the Addressed space of > sqlite (during an operation of DELETE or UPDATE a lot of data in WAL > mode use case described before) could grow up till 2GB. This cause the >

Re: [sqlite] memory wasted shm mapped file (3.7.2)

Ok I understand that it's only address space and not allocated: anyway if the addressed, and so the reserved, memory grow up you will see the Virutal Bytes in the windows performance monitor grow up. This is ok in the normal function: what I worry about is that the Addressed space of sqlite

Re: [sqlite] memory wasted shm mapped file (3.7.2)

On Fri, Sep 10, 2010 at 3:52 PM, Michele Pradella < michele.prade...@selea.com> wrote: > After some tests, with the new sqlite3.c source, seams that the > behavior is better than before. So I see the -shm and -wal file grow up, > and in VMMap I can see the Mapped File of -shm growing up, but not

Re: [sqlite] memory wasted shm mapped file (3.7.2)

After some tests, with the new sqlite3.c source, seams that the behavior is better than before. So I see the -shm and -wal file grow up, and in VMMap I can see the Mapped File of -shm growing up, but not so much as before. With VMMap I can see only -shm mapped files of 32K or 64K and I'm able

Re: [sqlite] memory wasted shm mapped file (3.7.2)

ok, Dan already sent me a sqlite3.c source, and I'm doing some tests...I let you know the results Il 10/09/2010 12.12, Max Vlasov ha scritto: > On Fri, Sep 10, 2010 at 12:53 PM, Dan Kennedy wrote: > >> The bug is fixed by Shane's patch linked earlier in the thread. With

Re: [sqlite] memory wasted shm mapped file (3.7.2)

On Fri, Sep 10, 2010 at 12:53 PM, Dan Kennedy wrote: > > The bug is fixed by Shane's patch linked earlier in the thread. With > the patch, an 11MB -shm file is mapped into memory in around 350 chunks > with an average size of 48KB. Total address space used is around >

Re: [sqlite] memory wasted shm mapped file (3.7.2)

I tried yesterday to apply the patch to my sqlite3.c and I think that the problem's not fixed: but perhaps I'm wrong or I miss something applying the patch. Can you provide me a sqlite3.c source with this patch applied correctly so I can test if this issue is correct? Il 10/09/2010 10.53,

Re: [sqlite] memory wasted shm mapped file (3.7.2)

> Michele, thanks for pointing out to vmmap, sysinternals made them so > fast, I > can not track them all. This is an excellent utility. > I think that the development team already knows that is going on, > just my > speculation. > > As long as I see, every next file mapping wants to see not

Re: [sqlite] memory wasted shm mapped file (3.7.2)

On Fri, Sep 10, 2010 at 11:29 AM, Michele Pradella < michele.prade...@selea.com> wrote: > I forgot to tell you something: in my situation the -shm file is > always small in size (about 6-7MB) even when I got 2GB of Virtual Bytes. > But it seams that is mapped too much times. > I see something

Re: [sqlite] memory wasted shm mapped file (3.7.2)

I forgot to tell you something: in my situation the -shm file is always small in size (about 6-7MB) even when I got 2GB of Virtual Bytes. But it seams that is mapped too much times. I see something that you could obtain if you do this: for (int i=0;i<1000;i++) "Map -shm in memory" This is

Re: [sqlite] memory wasted shm mapped file (3.7.2)

Even in my use case I got "Disk I/O error" after I reached 2GB of virtual memory. Max tell us the size of the memory mapped file in VMMap tool when you got "Disk I/O error", and check the value of Virtual Bytes in windows performance counter Il 10/09/2010 6.54, Max Vlasov ha scritto: >> But

Re: [sqlite] memory wasted shm mapped file (3.7.2)

Hi Shane, have you look at the Virtual Bytes in the performance monitor? in my situation is that counter that grow up to 2GB and than application crash because virtual space finish. I'm using Windows XP professional SP3. I can reproduce the problem in my DB with this SQL statement: DELETE

Re: [sqlite] memory wasted shm mapped file (3.7.2)

Hi Max, I'm sure that is a leak related to sqlite WAL because if I disable WAL with journal_mode=DELETE and I do not have the leak. More over if I use vmmap tool I see the size of the mapped files growing up and vmmap show me that all the memory is used by -shm files mapped a lot of times.

Re: [sqlite] memory wasted shm mapped file (3.7.2)

> But as a side effect I got not expected result in other area, when I tried > to append 1,500,000 records to this 1.7G file having 5G of free space on the > disk, I got the error a user reported recently about win7 64bit, "Disk I/O > error".

Re: [sqlite] memory wasted shm mapped file (3.7.2)

Michele- I've looked at trying to reproduce your issue on an 32-bit Windows XP system using the latest code, and could not. Even assuming the "worst case" of a 512 byte page size, starting with a 1.2gb DB file, deleting all the records would result in a WAL file of roughly the same size

Re: [sqlite] memory wasted shm mapped file (3.7.2)

On Thu, Sep 9, 2010 at 7:16 PM, Dan Kennedy wrote: > > > > > Dan, don't know whether it is related, but I detected memory leak in > > 3.7.2 > > related to handle count increasing. > > I think there will be one open handle for each 32KB of > shared-memory space in use. Or

Re: [sqlite] memory wasted shm mapped file (3.7.2)

Hi Max, I got the problem in both situations: 1. I have a journal_mode=DELETE database and I convert it to WAL. 2. I create a new database with WAL mode. I never check the handles in the task manager, but I always see the mapped files in vmmap growing up. I think it's the same. Anyway I

Re: [sqlite] memory wasted shm mapped file (3.7.2)

On Sep 9, 2010, at 10:04 PM, Max Vlasov wrote: > On Thu, Sep 9, 2010 at 11:37 AM, Dan Kennedy > wrote: > >> >> On Sep 9, 2010, at 1:12 PM, Michele Pradella wrote: >> >>> Hi, do you have some news about the wasted memory? have you found >>> the >>> reason for the

Re: [sqlite] memory wasted shm mapped file (3.7.2)

On Thu, Sep 9, 2010 at 11:37 AM, Dan Kennedy wrote: > > On Sep 9, 2010, at 1:12 PM, Michele Pradella wrote: > > > Hi, do you have some news about the wasted memory? have you found the > > reason for the windows backend? > > Fixed here: > >

Re: [sqlite] memory wasted shm mapped file (3.7.2)

I have ported the changes of the fix http://www.sqlite.org/src/ci/f213e133f6 to my code (with some difficult because I have ported directly in my sqlite3.c source). Anyway I think that the problem still exist, after few test I'll see with vmmap mapped file growing up like before the fix. I

Re: [sqlite] memory wasted shm mapped file (3.7.2)

ok thank you, today I'm going to port the difference to my source code and I'm going to try if the memory it's ok Il 09/09/2010 9.37, Dan Kennedy ha scritto: > On Sep 9, 2010, at 1:12 PM, Michele Pradella wrote: > >> Hi, do you have some news about the wasted memory? have you found the >>

Re: [sqlite] memory wasted shm mapped file (3.7.2)

On Sep 9, 2010, at 1:12 PM, Michele Pradella wrote: > Hi, do you have some news about the wasted memory? have you found the > reason for the windows backend? Fixed here: http://www.sqlite.org/src/ci/f213e133f6 Does the problem still show up for you using fossil tip? > do you think

Re: [sqlite] memory wasted shm mapped file (3.7.2)

Hi, do you have some news about the wasted memory? have you found the reason for the windows backend? do you think it could be due to the windows implementation of the mmap? Il 02/09/2010 16.46, Richard Hipp ha scritto: > On Thu, Sep 2, 2010 at 9:59 AM, Marcus

Re: [sqlite] memory wasted shm mapped file (3.7.2)

On Thu, Sep 2, 2010 at 9:59 AM, Marcus Grimm wrote: > > Michele Pradella wrote: > > ok, I'll wait for the walk around. > > I always use a BEGIN; COMMIT; transaction but often, after a COMMIT; the > > -wal file does not change in size, it seams it's not checkponted. > >

Re: [sqlite] memory wasted shm mapped file (3.7.2)

Michele Pradella wrote: > ok, I'll wait for the walk around. > I always use a BEGIN; COMMIT; transaction but often, after a COMMIT; the > -wal file does not change in size, it seams it's not checkponted. > Anyway do you think that with WAL journal mode I should continue to use > BEGIN;

Re: [sqlite] memory wasted shm mapped file (3.7.2)

ok, I'll wait for the walk around. I always use a BEGIN; COMMIT; transaction but often, after a COMMIT; the -wal file does not change in size, it seams it's not checkponted. Anyway do you think that with WAL journal mode I should continue to use BEGIN; COMMIT; statement? or not? Il

Re: [sqlite] memory wasted shm mapped file (3.7.2)

On Thu, Sep 2, 2010 at 8:34 AM, Michele Pradella wrote: > Hi, > I found a strange behavior of the sqlite 3.7.2 with WAL journal mode. > Yesterday I found my application DB with a -wal file of 1,5GB and a -shm > file of few MB (about 9MB) with a DB file of 1,2GB: in

[sqlite] memory wasted shm mapped file (3.7.2)

Hi, I found a strange behavior of the sqlite 3.7.2 with WAL journal mode. Yesterday I found my application DB with a -wal file of 1,5GB and a -shm file of few MB (about 9MB) with a DB file of 1,2GB: in this situation I got the process memory wasted by "mapped file" of the -shm file. It seams