Re: [sqlite] Sqlite Step Strange Results

[D. Richard Hipp]

On Feb 25, 2010, at 10:19 AM, Rob Read wrote:

> We are using sqlite in a low memory environment and as such have  
> been using
> the MEMSYS5 memory allocator. This has been working great to confine  
> our
> memory usage to the buffer that gets passed during start time. I have
> recently run into some strange results while doing a step to insert  
> a blob
> into the database. If I set my cache size, using the  pragma, to  
> 1500 then
> the step fails with SQLITE_NOMEM, but if I do this same process and  
> decrease
> the cache size to 1000 then the step succeeds. This might sound like  
> a easy
> problem to fix in that my page cache is taking up the memory that  
> the step
> needs to complete the insert, but if I check the memory usage by  
> calling
> sqlite3_memory_used(), in both situations I have plenty of memory  
> available
> to complete the transfer. If anyone could shed some light on this  
> issue I
> would appreciate it.
>

Probably this is due to memory fragmentation.

In a low-memory environment, you should use  
sqlite3_config(SQLITE_CONFIG_PAGECACHE,...) to set up a separate  
buffer for use by the page cache.  The page cache is the primary  
memory user, so allocate most of your memory budget to it.  And since  
all page cache allocations are the same size, you won't get  
fragmentation there.  You can normally get by with much less memory  
for the general-purpose heap, depending on how many prepared  
statements you are holding at once and how complex your schema is.

>
>
> Robert Read
>
> Lead Software Engineer
>
> Saltillo Corporation
>
>
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

D. Richard Hipp
d...@hwaci.com



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Sqlite Step Strange Results

[Pavel Ivanov]

Could it be because of fragmentation issues? When you allow more cache
pages to be allocated you allow more fragmentation to be introduced.
And although in total there could be enough memory available there
could still be no solid block of sufficient size available.

>From the comment in mem5.c:

** Let n be the size of the largest allocation divided by the minimum
** allocation size (after rounding all sizes up to a power of 2.)  Let M
** be the maximum amount of memory ever outstanding at one time.  Let
** N be the total amount of memory available for allocation.  Robson
** proved that this memory allocator will never breakdown due to
** fragmentation as long as the following constraint holds:
**
**  N >=  M*(1 + log2(n)/2) - n + 1
**
** The sqlite3_status() logic tracks the maximum values of n and M so
** that an application can, at any time, verify this constraint.



Pavel

On Thu, Feb 25, 2010 at 10:19 AM, Rob Read  wrote:
> We are using sqlite in a low memory environment and as such have been using
> the MEMSYS5 memory allocator. This has been working great to confine our
> memory usage to the buffer that gets passed during start time. I have
> recently run into some strange results while doing a step to insert a blob
> into the database. If I set my cache size, using the  pragma, to 1500 then
> the step fails with SQLITE_NOMEM, but if I do this same process and decrease
> the cache size to 1000 then the step succeeds. This might sound like a easy
> problem to fix in that my page cache is taking up the memory that the step
> needs to complete the insert, but if I check the memory usage by calling
> sqlite3_memory_used(), in both situations I have plenty of memory available
> to complete the transfer. If anyone could shed some light on this issue I
> would appreciate it.
>
>
>
> Robert Read
>
> Lead Software Engineer
>
> Saltillo Corporation
>
>
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Sqlite Step Strange Results

[Rob Read]

We are using sqlite in a low memory environment and as such have been using
the MEMSYS5 memory allocator. This has been working great to confine our
memory usage to the buffer that gets passed during start time. I have
recently run into some strange results while doing a step to insert a blob
into the database. If I set my cache size, using the  pragma, to 1500 then
the step fails with SQLITE_NOMEM, but if I do this same process and decrease
the cache size to 1000 then the step succeeds. This might sound like a easy
problem to fix in that my page cache is taking up the memory that the step
needs to complete the insert, but if I check the memory usage by calling
sqlite3_memory_used(), in both situations I have plenty of memory available
to complete the transfer. If anyone could shed some light on this issue I
would appreciate it.

 

Robert Read

Lead Software Engineer

Saltillo Corporation

 

 

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users