Re: [sqlite] This is driving me nuts

y Ling > Sent: Thursday, May 31, 2018 3:57:27 PM > To: 'SQLite mailing list' > Subject: Re: [sqlite] This is driving me nuts > > > While that makes sense Clemens it again fails to explain why it does no > > caching when desc is stepped through first. I am at least satisf

Re: [sqlite] This is driving me nuts

Yes, I think you’re right. I suppose maybe it recognises the desc sequence from page access. From: sqlite-users on behalf of Andy Ling Sent: Thursday, May 31, 2018 3:57:27 PM To: 'SQLite mailing list' Subject: Re: [sqlite] This is driving me nuts > Wh

Re: [sqlite] This is driving me nuts

> While that makes sense Clemens it again fails to explain why it does no > caching when desc is stepped through first. I am at least satisfied that it’s > not > something I should dwell on or, more the point, it isn’t down to a bug in my > code which is why I got embroiled in it in the first

Re: [sqlite] This is driving me nuts

c 9534 After desc 9535 After asc 8585 After desc 8586 Tom From: sqlite-users on behalf of Clemens Ladisch Sent: Thursday, May 31, 2018 2:06:08 PM To: sqlite-users@mailinglists.sqlite.org Subject: Re: [sqlite] This is driving me nuts x wrote: &g

Re: [sqlite] This is driving me nuts

On 31 May 2018, at 2:06pm, Clemens Ladisch wrote: > Windows has several different caching strategies. Also please note that the caching strategies change between different versions of Windows, sometimes even on what looks like a trivial update. Also, that drivers can request different

Re: [sqlite] This is driving me nuts

x wrote: > Why would window’s increase it’s cache size during step ascending then > reduce it during step descending in such a consistent manner? Windows has several different caching strategies. says: |

Re: [sqlite] This is driving me nuts

>While SQLite reads more and more rows from your database, Windows caches more >and more pages from the file, explaining the physical available memory >diminishes, which does not mean SQLite, nor your program 'use' it: the OS does. Olivier, your post makes perfect sense but the above begs the

Re: [sqlite] This is driving me nuts

> Le 31 mai 2018 à 12:31, x a écrit : > >std::cout << "Ascending" << std::endl; >for (int i=0; sqlite3_step(asc)==SQLITE_ROW; i++) >if (i%Gap==0) std::cout << FreeMBs() << std::endl; >std::cout << FreeMBs() << std::endl; > >std::cout << std::endl

Re: [sqlite] This is driving me nuts

> Le 31 mai 2018 à 13:34, x a écrit : > > Clemens, when I change to ullAvailVirtual the figure stays pretty much the > same throughout the programme as you suggested. I still don’t understand the > ullAvailPhys numbers though. Why would window’s increase it’s cache size > during step

Re: [sqlite] This is driving me nuts

to any other app. From: sqlite-users on behalf of Clemens Ladisch Sent: Thursday, May 31, 2018 11:50:19 AM To: sqlite-users@mailinglists.sqlite.org Subject: Re: [sqlite] This is driving me nuts x wrote: > I’ve written the console app below to illustrate. It pri

Re: [sqlite] This is driving me nuts

x wrote: > I’ve written the console app below to illustrate. It prints out the amount of > RAM available every 20 million steps. > GlobalMemoryStatusEx(); > return status.ullAvailPhys / (1024 * 1024); This is "physical memory currently available", which is affected by other

Re: [sqlite] This is driving me nuts

> I’ve just discovered the thread in the original app decreases the > available memory by around 4 GB. Are they really that expensive? After others pointed out threads weren’t that expensive I concluded there was a bug in my code. On checking though I couldn’t find anything wrong yet the

Re: [sqlite] This is driving me nuts

> why, when you've got 16 GB ram, does a 6.4 GB vector cause any problems? Jim, it was only 3.2 GB and it was compiled as 64 bit just in case you’re thinking 32. >Either this statement is wrong, or you've misattributed the 4 GB of memory. Rowan, I’ve misattributed. There’s a bug in the

Re: [sqlite] This is driving me nuts

On May 28, 2018, at 3:35 AM, Rowan Worth wrote: > > On 28 May 2018 at 17:29, x wrote: > >> I’ve just discovered the thread in the original app decreases the >> available memory by around 4 GB. Are they really that expensive? > > A thread itself is not

Re: [sqlite] This is driving me nuts

> why, when you've got 16 GB ram, does a 6.4 GB vector cause any problems? Architecturally, 4 GB is like the sound barrier -- you may get turbulence or a sonic boom. Because 4 GB was the limit for 32 bit word size any residual 32 bit code or 32 bit assumptions anywhere in the stack (or CPU)

Re: [sqlite] This is driving me nuts

On 28 May 2018 at 17:29, x wrote: > I’ve just discovered the thread in the original app decreases the > available memory by around 4 GB. Are they really that expensive? A thread itself is not expensive in terms of memory. > It has very little data of its own Either

Re: [sqlite] This is driving me nuts

I’ve just discovered the thread in the original app decreases the available memory by around 4 GB. Are they really that expensive? It has very little data of its own and just calls a function declared in the main thread. ___ sqlite-users mailing list

Re: [sqlite] This is driving me nuts

: sqlite-users <sqlite-users-boun...@mailinglists.sqlite.org> on behalf of Abroży Nieprzełoży <abrozynieprzelozy314...@gmail.com> Sent: Monday, May 28, 2018 1:26:13 AM To: SQLite mailing list Subject: Re: [sqlite] This is driving me nuts Alternatively you can alloc physical memory using A

Re: [sqlite] This is driving me nuts

ting to mess about with windows handles and pages I’ve never heard of >> is >> beyond my pain threshold Abrozy. Thanks anyway. >> >> >> >> >> From: sqlite-users <sqlite-users-boun...@mailinglists.sqlite.org> on >&g

Re: [sqlite] This is driving me nuts

; To: SQLite mailing list > Subject: Re: [sqlite] This is driving me nuts > > I think you can experiment with changing the the working set size > limits and see what the effect will be. > https://msdn.microsoft.com/en-us/library/cc441804 > > 2018-05-27 17:09 GMT+02:00, curmudgeon <tam11

Re: [sqlite] This is driving me nuts

gmail.com> Sent: Sunday, May 27, 2018 5:23:12 PM To: SQLite mailing list Subject: Re: [sqlite] This is driving me nuts I think you can experiment with changing the the working set size limits and see what the effect will be. https://msdn.microsoft.com/en-us/library/cc441804 2018-05-27 17:

Re: [sqlite] This is driving me nuts

I think you can experiment with changing the the working set size limits and see what the effect will be. https://msdn.microsoft.com/en-us/library/cc441804 2018-05-27 17:09 GMT+02:00, curmudgeon : > It seems the array was being optimised away. I had to initialise every

Re: [sqlite] This is driving me nuts

It seems the array was being optimised away. I had to initialise every value to get the OS to claim the RAM. Once I did that the timings for the array were on a par with the vector with the second pass being slower than the first. While that clears up that part of the mystery I'm no closer to a

Re: [sqlite] This is driving me nuts

Strange. I repeated the test with v an int64 array (see below) however the FreeMB() doesn’t change. Both Memi’s return pretty much the value of Mem ??? for (int i=0; i<2; i++) { v=new int64_t[Size[i]]; int64_t Memi=FreeMBs();

Re: [sqlite] This is driving me nuts

I’ve changed the app to populate v with the query result and noted the free ram before and after each resize. I’m hoping that addresses some of the concerns re compiler optimisations even if it doesn’t supply any real answers? Results were similar to before. #include #include #pragma hdrstop

Re: [sqlite] This is driving me nuts

. From: sqlite-users <sqlite-users-boun...@mailinglists.sqlite.org> on behalf of Warren Young <war...@etr-usa.com> Sent: Saturday, May 26, 2018 8:54:33 PM To: SQLite mailing list Subject: Re: [sqlite] This is driving me nuts On May 26, 2018, at 9:43 AM, x <tam118

Re: [sqlite] This is driving me nuts

On May 26, 2018, at 9:43 AM, x wrote: > > I was sizing the vector beforehand and then timing how long it took to store > the RowIDs returned by an sqlite query. You’re not actually modifying the vector, only allocating it and then not using it. A sufficiently smart

Re: [sqlite] This is driving me nuts

m118...@hotmail.com>: > Hi Abrozy. > > > > I inserted the line > > int64_t Mem=sqlite3_memory_highwater(0); > > immediately after the while loop and added Mem to the cout. The values > returned were > > 2234704 (I==0) > > 2234768 (I==1). >

Re: [sqlite] This is driving me nuts

x wrote: > const int Size[]={11200,10}; // 112 million, 1 billion > > v.resize(Size[i]); > > // NB the exact same code is executed whether i is 0 or 1. > The only thing that > // changes is the size() of v and v isn't even used in the >

Re: [sqlite] This is driving me nuts

behalf of Abroży Nieprzełoży <abrozynieprzelozy314...@gmail.com> Sent: Saturday, May 26, 2018 5:39:03 PM To: SQLite mailing list Subject: Re: [sqlite] This is driving me nuts > I swapped Size[0] with Size[1] so that the resize was going from large to > small. The timings on each pass

Re: [sqlite] This is driving me nuts

oun...@mailinglists.sqlite.org> on behalf of Abroży Nieprzełoży <abrozynieprzelozy314...@gmail.com> Sent: Saturday, May 26, 2018 5:39:03 PM To: SQLite mailing list Subject: Re: [sqlite] This is driving me nuts > I swapped Size[0] with Size[1] so that the resize was going from large to > small. T

Re: [sqlite] This is driving me nuts

> I swapped Size[0] with Size[1] so that the resize was going from large to > small. The timings on each pass were then similar but were still around the > 24.xxx mark associated with the i==1 pass. Note the difference between size and capacity of the vector. Reducing the size does not necessarily