Re: [sqlite] Huge RAM usage when sqlite is started from another thread
Yes, I can see that SQLite is doing just small allocations using memory
methods.
I also tried to limit the size of database using following two methods
rc = sqlite3_config(SQLITE_CONFIG_HEAP, buffer, 500, 64);
if(rc != SQLITE_OK)
{
printf("Failed to set custom heap memory\n");
}
rc = sqlite3_soft_heap_limit64(100);
printf("Prev: %d\n", rc);
When I use both of them as in this example, I get usage of DB of around
22MB, what is not so bad.
Using of small heap (array in my case) should be fine for my application,
as it is going to use just small databases (1 - 3MB).
What do you think about this technique ? I think, it could be very
dangerous when size of database grows, but that should not happen.
Thank you and Regards,
Martin
2018-07-13 20:58 GMT+02:00 Richard Hipp :
> On 7/13/18, Martin Vystrčil wrote:
> >
> > But still, I would like to solve this somehow, could you please suggest
> me
> > some point, where to start ? Are there any possible compilation options,
> to
> > limit this allocation ?
>
> It is not SQLite that is doing this allocation. I suspect it is
> something happening inside of the malloc() in your libc.
>
> You can see the SQLite is never doing a huge allocation by calling
> sqlite3_memory_highwater(0) in place of sqlite3_memory_used().
>
> You are probably better off letting malloc() do whatever allocations
> it wants. However, if you really want to work around this, there are
> ways of getting SQLite to use a memory allocator other than the
> default system malloc(). See https://www.sqlite.org/malloc.html for
> further information. These are sharp techniques, so be careful. But
> if you do it right, you can limit the memory usage of SQLite to
> whatever you want.
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Huge RAM usage when sqlite is started from another thread
Thank you both for help. What Richar Hipp wrote is truth and something what I also saw using valgrind - memory allocation (overall consumption) of around 300 - 500 kB. To answer at least some one Bob's question from first e-mail, I'm trying to open in memory database (:memory:), so size of database should be almost nothing. By checking smaps, I can see big allocation there. It seems that all your assumptions are correct, memory doesn't seem to be used. 7fdd28021000-7fdd2c00 ---p 00:00 0 Size: 65404 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced:0 kB Anonymous: 0 kB AnonHugePages: 0 kB Shared_Hugetlb:0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize:4 kB MMUPageSize: 4 kB Locked:0 kB VmFlags: mr mw me nr sd But still, I would like to solve this somehow, could you please suggest me some point, where to start ? Are there any possible compilation options, to limit this allocation ? It would be completely fine for me, to allocate like 1MB or so, but this is way too much, although not used. Thank you for your help, Best Regards, Martin 2018-07-13 17:31 GMT+02:00 Bob Friesenhahn : > On Fri, 13 Jul 2018, Richard Hipp wrote: > >> >> The OP's test program (with a bug fix, various whitespace changes, and >> the addition of a call to sqlite3_memory_used()) is show below. >> sqlite3_memory_used() reports no difference in memory allocation. >> > > The usage is uninitialized/unmodified virtual memory which could be from a > heap allocation (with no subsequent writes to it) or due to memory mapping > something such as a file. The cause of the the usage may be deduced by > inspecting the /proc/[pid]/smaps content on a Linux system where '[pid]' is > the process id of the program. > > Bob > -- > Bob Friesenhahn > bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Huge RAM usage when sqlite is started from another thread
Hello everyone, I have a problem using sqlite in one of my project. When I create instance of sqlite (sqlite_open) from main thread, memory consumption is in normal (a few megabytes). But when I start sqlite from another thread, immediately around 70 - 80 MB of memory is allocated. Here is the smallest example, which can reproduce this problem. Link to pastebin where source code is: https://pastebin.com/BkU3uMCb. There is also some more info about memory usage. There are results of memory usage (from proc/pid/status) when database is opened from main thread. VmPeak:19780 kB VmSize:19636 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 1756 kB VmRSS: 1756 kB VmData: 308 kB VmStk: 132 kB VmExe: 8 kB VmLib: 4744 kB VmPTE:60 kB VmSwap:0 kB And there are results when sqlite is opened from other thread VmPeak: 158904 kB VmSize:93368 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 1636 kB VmRSS: 1636 kB VmData:74040 kB VmStk: 132 kB VmExe: 8 kB VmLib: 4744 kB VmPTE:60 kB VmSwap:0 kB Most notable difference is VmData, which is like 80MB in case when the other thread is started. Is there any reason, why sqlite consumes so much memory when started from other thread ? Or do I have any error in my application ? In parallel with this e-mail I also asked on stackoverflow. https://stackoverflow.com/questions/51317823/sqlite-high-ram-usage-when-connected-from-other-thread?noredirect=1#comment89612124_51317823 Every help is highly appreciated. Best Regards, Martin ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
