[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi Robert, I have never used SQLite with sqlite3_prepare(), sqlite3_step(), sqlite3_reset() and sqlite3_finalize(). Do you have any code example that i can use to avoid the use of sqlite_get_table() ? Thank you, Eduardo ---Mensaje original---I don't use the sqlite_get_table() function, and don't recommend it to others to use. As I understand it, it's there for legacy application support. New programs written for SQLite should use sqlite3_prepare(), sqlite3_step(), sqlite3_reset() and sqlite3_finalize() instead. Robert - Original Message - From: To: ; Sent: Friday, March 17, 2006 7:08 AM Subject: [sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE Hi Robert, I was talking about 3 selects satements using the same connections. Anyway, thank you very much for your advice of using PRAGMA cache size=8, that solved all the problems related to sqlite3_exec memory problems with a select statement, baut the memory problems are not solved at all, stilll i get a memory leak when using sqlite3_get_table, even if i use sqlite3_free_table not all the memory is freed. Someone pointed that this might be an error on the source code becuse the ARM processor architecture is quite different from the X86 processor. I have been taking a look at the table.c file but i am not that good programer, to see if anything fails, Do you see anything on the table.c that might be generating the problem?. I have attached to this email a memory graphic of sqlite3_get_table problem El día del padre está cerca... ¿Ya tienes el regalo? http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi Robert, I was talking about 3 selects satements using the same connections. Anyway, thank you very much for your advice of using PRAGMA cache size=8, that solved all the problems related to sqlite3_exec memory problems with a select statement, baut the memory problems are not solved at all, stilll i get a memory leak when using sqlite3_get_table, even if i use sqlite3_free_table not all the memory is freed. Someone pointed that this might be an error on the source code becuse the ARM processor architecture is quite different from the X86 processor. I have been taking a look at the table.c file but i am not that good programer, to see if anything fails, Do you see anything on the table.c that might be generating the problem?. I have attached to this email a memory graphic of sqlite3_get_table problem ---Mensaje original Original Message - From: I have run your program on the CE emulator (Pocket PC 2003) and i got the same memory leak. I have inserted 2 buttons on a MFC dialog application. The first button executes your code and the second button closes the application. If you examine the memory you will discover that the program only free the memory once you exit from the apllication, meanwhile it reserves memory as its needed (on demand, but see details below). the memory behaviour of SQLite is quite strange, an example: lets say that a select sentence reserves 1000kb of memory, once this local process has finished memory keeps reserved for the program (it should be freed), if another process executes a select sentence that needs 200kb SQLite will not reserve 200k more, it will use 200k of the previous 1000k reserved. if a 3rd process executes a select sentence that needs 1300k SQlite will reserve 300kb more and those 1300kb will not be freed until the main dialog application closes (even if the 3 process where local methods or functions). Ok this is where you lost me. 3 processes? Is your program running 3 times on the CE platform? If CE is running 3 instances of your program, then they definitely won't be sharing any memory and yes you'll definitely run out. Also if I recall correctly, CE 5.0 will not let you run multiple instances of the same program. If you're talking about 3 SELECT statements in the same program using the same connection instance, then that's another story. The problem is that if a select sentence consume most of the memory it will not be freed and the program will execute very slow until you exit from the application because there will be so little memory left for other not SQLite process that the program might be unusable. SQLite's default cache size is about 3mb. After opening a connection, try executing PRAGMA cache_size=8 or some really low number and tell me if its still leaking. Robert El día del padre está cerca... ¿Ya tienes el regalo? http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Doug Nebeker wrote: It sounds to me that he isn't saying it is leaking--it simply isn't releasing memory after a SELECT statement finishes (is finalized). This might be by design. I would expect the data to be released if the database connection is closed, but not necessarily after each SELECT. It would be nice to have an API to force any cached pages to be released (maybe it already exists and I don't know about it?). -Original Message- From: Robert Simpson [mailto:[EMAIL PROTECTED] Sent: Thursday, March 16, 2006 12:13 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE - Original Message - From: <[EMAIL PROTECTED]> I have run your program on the CE emulator (Pocket PC 2003) and i got the same memory leak. I have inserted 2 buttons on a MFC dialog application. The first button executes your code and the second button closes the application. If you examine the memory you will discover that the program only free the memory once you exit from the apllication, meanwhile it reserves memory as its needed (on demand, but see details below). the memory behaviour of SQLite is quite strange, an example: lets say that a select sentence reserves 1000kb of memory, once this local process has finished memory keeps reserved for the program (it should be freed), if another process executes a select sentence that needs 200kb SQLite will not reserve 200k more, it will use 200k of the previous 1000k reserved. if a 3rd process executes a select sentence that needs 1300k SQlite will reserve 300kb more and those 1300kb will not be freed until the main dialog application closes (even if the 3 process where local methods or functions). Ok this is where you lost me. 3 processes? Is your program running 3 times on the CE platform? If CE is running 3 instances of your program, then they definitely won't be sharing any memory and yes you'll definitely run out. Also if I recall correctly, CE 5.0 will not let you run multiple instances of the same program. If you're talking about 3 SELECT statements in the same program using the same connection instance, then that's another story. The problem is that if a select sentence consume most of the memory it will not be freed and the program will execute very slow until you exit from the application because there will be so little memory left for other not SQLite process that the program might be unusable. SQLite's default cache size is about 3mb. After opening a connection, try executing "PRAGMA cache_size=8" or some really low number and tell me if its still "leaking". Robert To find out more about Reuters visit www.about.reuters.com Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Ltd. He uses the table API call to get the output. By necessity this must allocate dynamic memory to accommodate the result set. We can guess that he then runs through the table and looks at the result rows. If he were to use an sqlite3_step and process each row as it is retrieved that dynamic memory requirement would be eliminated. Not only would memory usage be better controlled, but the application would run faster by eliminating the delay while te table is created. Some thought on how malloc and free work would help in conceptualising the application. The best that can usually be achieved is that over a period of time the application hits a memory "high water mark" and does not go beyond that. Minimizing dynamic memory allocation limits that high water level. Free cannot be guaranteed to ensure that malloc'd memory is totally reused. That requires garbage collection.
[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi,
Thank you very much for your help, i have modify the line, it compiles, it
doesn´t fire any exception, but memory is not freed at all (works the same way)
Anyone has another idea?, i tought that SQLite was working since a long time
ago on Windows CE devices, it seems quite strange that no one detected this
memory problem before?.
Thank you,
Eduardo
---Mensaje original---On March 16, 2006 12:28 pm, Jose Da Silva wrote:
I forgot a line:
char x[12] = Hello world\0;
char *ptr;
ptr = x;
while (*ptr) {printf(%c,*ptr); ptr;};
If you figure out the right sizeof(???) value to use, then I think it
could be submitted as a bug-fix for \src\table.c
You want to move the pointer from azResult[1] to azResult[0] but you
want to say it without doing it like this azResult--; on line 192,
but saying it like this looks sort of ugly although it might work:
azResult -= ( azResult[1] - azResult[0] );
..so if someone has a simpler solution, thanks :-)
El día del padre está cerca... ¿Ya tienes el regalo?
http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi Robert, I was talking about 3 selects satements using the same connections. Anyway, thank you very much for your advice of using PRAGMA cache size=8, that solved all the problems related to sqlite3_exec memory problems with a select statement, baut the memory problems are not solved at all, stilll i get a memory leak when using sqlite3_get_table, even if i use sqlite3_free_table not all the memory is freed. Someone pointed that this might be an error on the source code becuse the ARM processor architecture is quite different from the X86 processor. I have been takin a look at the table.c file but i am not that good programer, to see if anything fails, Do you see anything on the table.c that might be generating the problem?. Thank you very much, Eduardo ---Mensaje original Original Message - From: I have run your program on the CE emulator (Pocket PC 2003) and i got the same memory leak. I have inserted 2 buttons on a MFC dialog application. The first button executes your code and the second button closes the application. If you examine the memory you will discover that the program only free the memory once you exit from the apllication, meanwhile it reserves memory as its needed (on demand, but see details below). the memory behaviour of SQLite is quite strange, an example: lets say that a select sentence reserves 1000kb of memory, once this local process has finished memory keeps reserved for the program (it should be freed), if another process executes a select sentence that needs 200kb SQLite will not reserve 200k more, it will use 200k of the previous 1000k reserved. if a 3rd process executes a select sentence that needs 1300k SQlite will reserve 300kb more and those 1300kb will not be freed until the main dialog application closes (even if the 3 process where local methods or functions). Ok this is where you lost me. 3 processes? Is your program running 3 times on the CE platform? If CE is running 3 instances of your program, then they definitely won't be sharing any memory and yes you'll definitely run out. Also if I recall correctly, CE 5.0 will not let you run multiple instances of the same program. If you're talking about 3 SELECT statements in the same program using the same connection instance, then that's another story. The problem is that if a select sentence consume most of the memory it will not be freed and the program will execute very slow until you exit from the application because there will be so little memory left for other not SQLite process that the program might be unusable. SQLite's default cache size is about 3mb. After opening a connection, try executing PRAGMA cache_size=8 or some really low number and tell me if its still leaking. Robert El día del padre está cerca... ¿Ya tienes el regalo? http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi Again Robert,
I have run your program on the CE emulator (Pocket PC 2003) and i got the same
memory leak.
I have inserted 2 buttons on a MFC dialog application.
The first button executes your code and the second button closes the
application.
If you examine the memory you will discover that the program only free the
memory once you exit from the apllication, meanwhile it reserves memory as its
needed (on demand, but see details below).
the memory behaviour of SQLite is quite strange, an example:
lets say that a select sentence reserves 1000kb of memory, once this local
process has finished memory keeps reserved for the program (it should be
freed), if another process executes a select sentence that needs 200kb SQLite
will not reserve 200k more, it will use 200k of the previous 1000k reserved. if
a 3rd process executes a select sentence that needs 1300k SQlite will reserve
300kb more and those 1300kb will not be freed until the main dialog application
closes (even if the 3 process where local methods or functions).
The problem is that if a select sentence consume most of the memory it will not
be freed and the program will execute very slow until you exit from the
application because there will be so little memory left for other not SQLite
process that the program might be unusable.
Any ideas,
Thank you in advance
Eduardo
---Mensaje original---Ok, here's what I did ...
On the desktop I created a SQLite database with one table and inserted
120,000 rows into it. I then copied it over to the CE emulator.
Then I ran the following code on the CE (Pocket PC 2003 SE) emulator:
#include
#include
#include sqlite3.h
int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLine,
int nCmdShow)
{
sqlite3 *pdb;
int rc;
for (int n = 0; n 1; n )
{
rc = sqlite3_open(\\test.db3, pdb);
if (rc) break;
rc = sqlite3_exec(pdb, select * from testcase, 0, 0, 0);
if (rc) break;
rc = sqlite3_close(pdb);
if (rc) break;
}
return 0;
}
On the first call to sqlite3_exec(), available program memory dropped from
an initial 10.5mb to 8.45mb. However, once that was done, available memory
remained rock solid for the duration of the loop at a constant 8.45mb and no
non-zero error codes were ever returned.
Robert
El día del padre está cerca... ¿Ya tienes el regalo?
http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi Robert,
Thank you for your test. I have not test it on the emulator but in 4 different
Windows CE devices i have at work (with different Windows CE OS versions) and
it always give me the same memory leak result.
I will run your test tomorrow at work using the emulator and i will let you
know the results.
Thank you again,
Eduardo
---Mensaje original---Ok, here's what I did ...
On the desktop I created a SQLite database with one table and inserted
120,000 rows into it. I then copied it over to the CE emulator.
Then I ran the following code on the CE (Pocket PC 2003 SE) emulator:
#include
#include
#include sqlite3.h
int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLine,
int nCmdShow)
{
sqlite3 *pdb;
int rc;
for (int n = 0; n 1; n )
{
rc = sqlite3_open(\\test.db3, pdb);
if (rc) break;
rc = sqlite3_exec(pdb, select * from testcase, 0, 0, 0);
if (rc) break;
rc = sqlite3_close(pdb);
if (rc) break;
}
return 0;
}
On the first call to sqlite3_exec(), available program memory dropped from
an initial 10.5mb to 8.45mb. However, once that was done, available memory
remained rock solid for the duration of the loop at a constant 8.45mb and no
non-zero error codes were ever returned.
Robert
El día del padre está cerca... ¿Ya tienes el regalo?
http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
[sqlite] RE:[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi again Robert,
I think i forgot to tell you that to detect the memory leak you must not close
the application testing program, the easiest way to detect the memory leak is:
create a simple MFC dialog project. Add 2 buttons, one for executing your
testing program an another one for close the application from it.
If you execute your testing program and you monitorize the memory usage with
the Remote performance monitor (from the tools menu at the EVC 4.0
IDE-tools-Remote Performance Monitor-Select your device-press
the sign (add)-change combo Box selection to Memory-add a graphic
line for memory load). You will notice that memory is only recovered once you
execute the button to close the application.
I hope this might help you.
Thank you,
Eduardo
---Mensaje original---
Hi Robert,
Thank you for your test. I have not test it on the emulator but in 4 di
fferent Windows CE devices i have at work (with different Windows CE OS
versions) and it always give me the same memory leak result.
I will run your test tomorrow at work using the emulator and i will l
et you know the results.
Thank you again,
Eduardo
---Mensaje original---Ok, here's what I did ...
On the desktop I created a SQLite database with one table and inserted
120,000 rows into it. I then copied it over to the CE emulator.
Then I ran the following code on the CE (Pocket PC 2003 SE) emulator:
#include
#include
#include sqlite3.h
int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLin
e,
int nCmdShow)
{
sqlite3 *pdb;
int rc;
for (int n = 0; n 1; n )
{
rc = sqlite3_open(\\test.db3, pdb);
if (rc) break;
rc = sqlite3_exec(pdb, select * from testcase, 0, 0, 0);
if (rc) break;
rc = sqlite3_close(pdb);
if (rc) break;
}
return 0;
}
On the first call to sqlite3_exec(), available program memory dropped f
rom
an initial 10.5mb to 8.45mb. However, once that was done, available mem
ory
remained rock solid for the duration of the loop at a constant 8.45mb a
nd no
non-zero error codes were ever returned.
Robert
El día del padre está cerca... ¿Ya tienes el regalo? http://busca
wanadoo.es/search?type=prefdestino=weborigen=homespotbuscar=
Regalos%20Día%20del%20Padre
El día del padre está cerca... ¿Ya tienes el regalo?
http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi Robert,
Thank you for your test. I have not test it on the emulator but in 4 different
Windows CE devices i have at work (with different Windows CE OS versions) and
it always give me the same memory leak result.
I will run your test tomorrow at work using the emulator and i will let you
know the results.
Thank you again,
Eduardo
---Mensaje original---Ok, here's what I did ...
On the desktop I created a SQLite database with one table and inserted
120,000 rows into it. I then copied it over to the CE emulator.
Then I ran the following code on the CE (Pocket PC 2003 SE) emulator:
#include
#include
#include sqlite3.h
int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLine,
int nCmdShow)
{
sqlite3 *pdb;
int rc;
for (int n = 0; n 1; n )
{
rc = sqlite3_open(\\test.db3, pdb);
if (rc) break;
rc = sqlite3_exec(pdb, select * from testcase, 0, 0, 0);
if (rc) break;
rc = sqlite3_close(pdb);
if (rc) break;
}
return 0;
}
On the first call to sqlite3_exec(), available program memory dropped from
an initial 10.5mb to 8.45mb. However, once that was done, available memory
remained rock solid for the duration of the loop at a constant 8.45mb and no
non-zero error codes were ever returned.
Robert
El día del padre está cerca... ¿Ya tienes el regalo?
http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
Re: [sqlite] RE: SQLite memory leak on Windows CE
Ok, here's what I did ...
On the desktop I created a SQLite database with one table and inserted
120,000 rows into it. I then copied it over to the CE emulator.
Then I ran the following code on the CE (Pocket PC 2003 SE) emulator:
#include
#include
#include "sqlite3.h"
int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLine,
int nCmdShow)
{
sqlite3 *pdb;
int rc;
for (int n = 0; n < 1; n++)
{
rc = sqlite3_open("\\test.db3", );
if (rc) break;
rc = sqlite3_exec(pdb, "select * from testcase", 0, 0, 0);
if (rc) break;
rc = sqlite3_close(pdb);
if (rc) break;
}
return 0;
}
On the first call to sqlite3_exec(), available program memory dropped from
an initial 10.5mb to 8.45mb. However, once that was done, available memory
remained rock solid for the duration of the loop at a constant 8.45mb and no
non-zero error codes were ever returned.
Robert
Re: [sqlite] RE: SQLite memory leak on Windows CE
What method(s) are you using to detect the memory leak? I'll try and test this out today and see if I can come up with anything. Robert - Original Message - From: <[EMAIL PROTECTED]> To: <sqlite-users@sqlite.org> Sent: Wednesday, March 15, 2006 6:18 AM Subject: [sqlite] RE: SQLite memory leak on Windows CE Hi again, i have tried your advice of changing line 192 at table.c and it didn´t work (it fires an exception). I do not think that the problem is related to sqlite3_get_table or sqlite3_free_table source code. To make sure, i have test it with another simple program (see the source code below), that doesn´t make use of the sqlite_x_table stuff and Windows CE doesn´t free memory properly (memory only gets freed once the main program has exited). Please see the testing code i am using (it is quite simple)
[sqlite] RE: SQLite memory leak on Windows CE
Hi again, i have tried your advice of changing line 192 at table.c and it
didn´t work (it fires an exception). I do not think that the problem is related
to sqlite3_get_table or sqlite3_free_table source code.
To make sure, i have test it with another simple program (see the source code
below), that doesn´t make use of the sqlite_x_table stuff and Windows CE
doesn´t free memory properly (memory only gets freed once the main program has
exited). Please see the testing code i am using (it is quite simple)
void CMemoryMDlg::OnButton1()
{
sqlite3 *db;
char *zErrMsg = 0;
int rc;
rc = sqlite3_open(\\articulos.db, db);
if( rc ){
MessageBox(_T(NO),_T(),MB_OK);
sqlite3_free(zErrMsg);
sqlite3_close(db);
CDialog::OnCancel();
}
rc = sqlite3_exec(db, select * from ARTIC ;, 0, 0, zErrMsg);
if( rc!=SQLITE_OK ){
sqlite3_free(zErrMsg);
MessageBox(_T(NO2),_T(),MB_OK);
}
sqlite3_free(zErrMsg);
sqlite3_close(db);
}
Thank you in advance,
Eduardo Esteban
---Mensaje original---On March 14, 2006 01:49 pm, [EMAIL PROTECTED] wrote:
Hello again,
Thank for your answers:
first of all thank you who pointed that i should use a close(db) at
line 17 and call sqlite3_free_table(result) even if there was an
error, but these solutions didn´t solve my problem (the execution
code never reached that point because ret value was always
SQLITE_OK). Do anyone of you work with SQLite on Windows CE?
No, I don't have win ce either.
However, I looked at the sqlite code and there may be a bug.
In SQLite3.3.4 go to:
src\table.c line 192 and change azResult--;
changed_from-
190 if( azResult ){
191 int i, n;
192 azResult--;
193 if( azResult==0 ) return;
194 n = (int)azResult[0];
changed_to-
190 if( azResult ){
191 int i, n;
192 azResult -= sizeof(char*); changed
193 if( azResult==0 ) return;
194 n = (int)azResult[0];
Recompile sqlite and try again.
Let us know if it works.
Automóviles, telefonía, imagen y sonido, belleza... ¡encuentra gratis todo lo
que necesitas! http://clasificados.wanadoo.es/
