[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi Robert, I have never used SQLite with sqlite3_prepare(), sqlite3_step(), sqlite3_reset() and sqlite3_finalize(). Do you have any code example that i can use to avoid the use of sqlite_get_table() ? Thank you, Eduardo ---Mensaje original---I don't use the sqlite_get_table() function, and don't recommend it to others to use. As I understand it, it's there for legacy application support. New programs written for SQLite should use sqlite3_prepare(), sqlite3_step(), sqlite3_reset() and sqlite3_finalize() instead. Robert - Original Message - From: To: ; Sent: Friday, March 17, 2006 7:08 AM Subject: [sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE Hi Robert, I was talking about 3 selects satements using the same connections. Anyway, thank you very much for your advice of using PRAGMA cache size=8, that solved all the problems related to sqlite3_exec memory problems with a select statement, baut the memory problems are not solved at all, stilll i get a memory leak when using sqlite3_get_table, even if i use sqlite3_free_table not all the memory is freed. Someone pointed that this might be an error on the source code becuse the ARM processor architecture is quite different from the X86 processor. I have been taking a look at the table.c file but i am not that good programer, to see if anything fails, Do you see anything on the table.c that might be generating the problem?. I have attached to this email a memory graphic of sqlite3_get_table problem El día del padre está cerca... ¿Ya tienes el regalo? http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi Robert, I was talking about 3 selects satements using the same connections. Anyway, thank you very much for your advice of using PRAGMA cache size=8, that solved all the problems related to sqlite3_exec memory problems with a select statement, baut the memory problems are not solved at all, stilll i get a memory leak when using sqlite3_get_table, even if i use sqlite3_free_table not all the memory is freed. Someone pointed that this might be an error on the source code becuse the ARM processor architecture is quite different from the X86 processor. I have been taking a look at the table.c file but i am not that good programer, to see if anything fails, Do you see anything on the table.c that might be generating the problem?. I have attached to this email a memory graphic of sqlite3_get_table problem ---Mensaje original Original Message - From: I have run your program on the CE emulator (Pocket PC 2003) and i got the same memory leak. I have inserted 2 buttons on a MFC dialog application. The first button executes your code and the second button closes the application. If you examine the memory you will discover that the program only free the memory once you exit from the apllication, meanwhile it reserves memory as its needed (on demand, but see details below). the memory behaviour of SQLite is quite strange, an example: lets say that a select sentence reserves 1000kb of memory, once this local process has finished memory keeps reserved for the program (it should be freed), if another process executes a select sentence that needs 200kb SQLite will not reserve 200k more, it will use 200k of the previous 1000k reserved. if a 3rd process executes a select sentence that needs 1300k SQlite will reserve 300kb more and those 1300kb will not be freed until the main dialog application closes (even if the 3 process where local methods or functions). Ok this is where you lost me. 3 processes? Is your program running 3 times on the CE platform? If CE is running 3 instances of your program, then they definitely won't be sharing any memory and yes you'll definitely run out. Also if I recall correctly, CE 5.0 will not let you run multiple instances of the same program. If you're talking about 3 SELECT statements in the same program using the same connection instance, then that's another story. The problem is that if a select sentence consume most of the memory it will not be freed and the program will execute very slow until you exit from the application because there will be so little memory left for other not SQLite process that the program might be unusable. SQLite's default cache size is about 3mb. After opening a connection, try executing PRAGMA cache_size=8 or some really low number and tell me if its still leaking. Robert El día del padre está cerca... ¿Ya tienes el regalo? http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Doug Nebeker wrote: It sounds to me that he isn't saying it is leaking--it simply isn't releasing memory after a SELECT statement finishes (is finalized). This might be by design. I would expect the data to be released if the database connection is closed, but not necessarily after each SELECT. It would be nice to have an API to force any cached pages to be released (maybe it already exists and I don't know about it?). -Original Message- From: Robert Simpson [mailto:[EMAIL PROTECTED] Sent: Thursday, March 16, 2006 12:13 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE - Original Message - From: <[EMAIL PROTECTED]> I have run your program on the CE emulator (Pocket PC 2003) and i got the same memory leak. I have inserted 2 buttons on a MFC dialog application. The first button executes your code and the second button closes the application. If you examine the memory you will discover that the program only free the memory once you exit from the apllication, meanwhile it reserves memory as its needed (on demand, but see details below). the memory behaviour of SQLite is quite strange, an example: lets say that a select sentence reserves 1000kb of memory, once this local process has finished memory keeps reserved for the program (it should be freed), if another process executes a select sentence that needs 200kb SQLite will not reserve 200k more, it will use 200k of the previous 1000k reserved. if a 3rd process executes a select sentence that needs 1300k SQlite will reserve 300kb more and those 1300kb will not be freed until the main dialog application closes (even if the 3 process where local methods or functions). Ok this is where you lost me. 3 processes? Is your program running 3 times on the CE platform? If CE is running 3 instances of your program, then they definitely won't be sharing any memory and yes you'll definitely run out. Also if I recall correctly, CE 5.0 will not let you run multiple instances of the same program. If you're talking about 3 SELECT statements in the same program using the same connection instance, then that's another story. The problem is that if a select sentence consume most of the memory it will not be freed and the program will execute very slow until you exit from the application because there will be so little memory left for other not SQLite process that the program might be unusable. SQLite's default cache size is about 3mb. After opening a connection, try executing "PRAGMA cache_size=8" or some really low number and tell me if its still "leaking". Robert To find out more about Reuters visit www.about.reuters.com Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Ltd. He uses the table API call to get the output. By necessity this must allocate dynamic memory to accommodate the result set. We can guess that he then runs through the table and looks at the result rows. If he were to use an sqlite3_step and process each row as it is retrieved that dynamic memory requirement would be eliminated. Not only would memory usage be better controlled, but the application would run faster by eliminating the delay while te table is created. Some thought on how malloc and free work would help in conceptualising the application. The best that can usually be achieved is that over a period of time the application hits a memory "high water mark" and does not go beyond that. Minimizing dynamic memory allocation limits that high water level. Free cannot be guaranteed to ensure that malloc'd memory is totally reused. That requires garbage collection.
[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi, Thank you very much for your help, i have modify the line, it compiles, it doesn´t fire any exception, but memory is not freed at all (works the same way) Anyone has another idea?, i tought that SQLite was working since a long time ago on Windows CE devices, it seems quite strange that no one detected this memory problem before?. Thank you, Eduardo ---Mensaje original---On March 16, 2006 12:28 pm, Jose Da Silva wrote: I forgot a line: char x[12] = Hello world\0; char *ptr; ptr = x; while (*ptr) {printf(%c,*ptr); ptr;}; If you figure out the right sizeof(???) value to use, then I think it could be submitted as a bug-fix for \src\table.c You want to move the pointer from azResult[1] to azResult[0] but you want to say it without doing it like this azResult--; on line 192, but saying it like this looks sort of ugly although it might work: azResult -= ( azResult[1] - azResult[0] ); ..so if someone has a simpler solution, thanks :-) El día del padre está cerca... ¿Ya tienes el regalo? http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
[sqlite] RE:Re: [sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi Robert, I was talking about 3 selects satements using the same connections. Anyway, thank you very much for your advice of using PRAGMA cache size=8, that solved all the problems related to sqlite3_exec memory problems with a select statement, baut the memory problems are not solved at all, stilll i get a memory leak when using sqlite3_get_table, even if i use sqlite3_free_table not all the memory is freed. Someone pointed that this might be an error on the source code becuse the ARM processor architecture is quite different from the X86 processor. I have been takin a look at the table.c file but i am not that good programer, to see if anything fails, Do you see anything on the table.c that might be generating the problem?. Thank you very much, Eduardo ---Mensaje original Original Message - From: I have run your program on the CE emulator (Pocket PC 2003) and i got the same memory leak. I have inserted 2 buttons on a MFC dialog application. The first button executes your code and the second button closes the application. If you examine the memory you will discover that the program only free the memory once you exit from the apllication, meanwhile it reserves memory as its needed (on demand, but see details below). the memory behaviour of SQLite is quite strange, an example: lets say that a select sentence reserves 1000kb of memory, once this local process has finished memory keeps reserved for the program (it should be freed), if another process executes a select sentence that needs 200kb SQLite will not reserve 200k more, it will use 200k of the previous 1000k reserved. if a 3rd process executes a select sentence that needs 1300k SQlite will reserve 300kb more and those 1300kb will not be freed until the main dialog application closes (even if the 3 process where local methods or functions). Ok this is where you lost me. 3 processes? Is your program running 3 times on the CE platform? If CE is running 3 instances of your program, then they definitely won't be sharing any memory and yes you'll definitely run out. Also if I recall correctly, CE 5.0 will not let you run multiple instances of the same program. If you're talking about 3 SELECT statements in the same program using the same connection instance, then that's another story. The problem is that if a select sentence consume most of the memory it will not be freed and the program will execute very slow until you exit from the application because there will be so little memory left for other not SQLite process that the program might be unusable. SQLite's default cache size is about 3mb. After opening a connection, try executing PRAGMA cache_size=8 or some really low number and tell me if its still leaking. Robert El día del padre está cerca... ¿Ya tienes el regalo? http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi Again Robert, I have run your program on the CE emulator (Pocket PC 2003) and i got the same memory leak. I have inserted 2 buttons on a MFC dialog application. The first button executes your code and the second button closes the application. If you examine the memory you will discover that the program only free the memory once you exit from the apllication, meanwhile it reserves memory as its needed (on demand, but see details below). the memory behaviour of SQLite is quite strange, an example: lets say that a select sentence reserves 1000kb of memory, once this local process has finished memory keeps reserved for the program (it should be freed), if another process executes a select sentence that needs 200kb SQLite will not reserve 200k more, it will use 200k of the previous 1000k reserved. if a 3rd process executes a select sentence that needs 1300k SQlite will reserve 300kb more and those 1300kb will not be freed until the main dialog application closes (even if the 3 process where local methods or functions). The problem is that if a select sentence consume most of the memory it will not be freed and the program will execute very slow until you exit from the application because there will be so little memory left for other not SQLite process that the program might be unusable. Any ideas, Thank you in advance Eduardo ---Mensaje original---Ok, here's what I did ... On the desktop I created a SQLite database with one table and inserted 120,000 rows into it. I then copied it over to the CE emulator. Then I ran the following code on the CE (Pocket PC 2003 SE) emulator: #include #include #include sqlite3.h int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLine, int nCmdShow) { sqlite3 *pdb; int rc; for (int n = 0; n 1; n ) { rc = sqlite3_open(\\test.db3, pdb); if (rc) break; rc = sqlite3_exec(pdb, select * from testcase, 0, 0, 0); if (rc) break; rc = sqlite3_close(pdb); if (rc) break; } return 0; } On the first call to sqlite3_exec(), available program memory dropped from an initial 10.5mb to 8.45mb. However, once that was done, available memory remained rock solid for the duration of the loop at a constant 8.45mb and no non-zero error codes were ever returned. Robert El día del padre está cerca... ¿Ya tienes el regalo? http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi Robert, Thank you for your test. I have not test it on the emulator but in 4 different Windows CE devices i have at work (with different Windows CE OS versions) and it always give me the same memory leak result. I will run your test tomorrow at work using the emulator and i will let you know the results. Thank you again, Eduardo ---Mensaje original---Ok, here's what I did ... On the desktop I created a SQLite database with one table and inserted 120,000 rows into it. I then copied it over to the CE emulator. Then I ran the following code on the CE (Pocket PC 2003 SE) emulator: #include #include #include sqlite3.h int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLine, int nCmdShow) { sqlite3 *pdb; int rc; for (int n = 0; n 1; n ) { rc = sqlite3_open(\\test.db3, pdb); if (rc) break; rc = sqlite3_exec(pdb, select * from testcase, 0, 0, 0); if (rc) break; rc = sqlite3_close(pdb); if (rc) break; } return 0; } On the first call to sqlite3_exec(), available program memory dropped from an initial 10.5mb to 8.45mb. However, once that was done, available memory remained rock solid for the duration of the loop at a constant 8.45mb and no non-zero error codes were ever returned. Robert El día del padre está cerca... ¿Ya tienes el regalo? http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
[sqlite] RE:[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi again Robert, I think i forgot to tell you that to detect the memory leak you must not close the application testing program, the easiest way to detect the memory leak is: create a simple MFC dialog project. Add 2 buttons, one for executing your testing program an another one for close the application from it. If you execute your testing program and you monitorize the memory usage with the Remote performance monitor (from the tools menu at the EVC 4.0 IDE-tools-Remote Performance Monitor-Select your device-press the sign (add)-change combo Box selection to Memory-add a graphic line for memory load). You will notice that memory is only recovered once you execute the button to close the application. I hope this might help you. Thank you, Eduardo ---Mensaje original--- Hi Robert, Thank you for your test. I have not test it on the emulator but in 4 di fferent Windows CE devices i have at work (with different Windows CE OS versions) and it always give me the same memory leak result. I will run your test tomorrow at work using the emulator and i will l et you know the results. Thank you again, Eduardo ---Mensaje original---Ok, here's what I did ... On the desktop I created a SQLite database with one table and inserted 120,000 rows into it. I then copied it over to the CE emulator. Then I ran the following code on the CE (Pocket PC 2003 SE) emulator: #include #include #include sqlite3.h int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLin e, int nCmdShow) { sqlite3 *pdb; int rc; for (int n = 0; n 1; n ) { rc = sqlite3_open(\\test.db3, pdb); if (rc) break; rc = sqlite3_exec(pdb, select * from testcase, 0, 0, 0); if (rc) break; rc = sqlite3_close(pdb); if (rc) break; } return 0; } On the first call to sqlite3_exec(), available program memory dropped f rom an initial 10.5mb to 8.45mb. However, once that was done, available mem ory remained rock solid for the duration of the loop at a constant 8.45mb a nd no non-zero error codes were ever returned. Robert El día del padre está cerca... ¿Ya tienes el regalo? http://busca wanadoo.es/search?type=prefdestino=weborigen=homespotbuscar= Regalos%20Día%20del%20Padre El día del padre está cerca... ¿Ya tienes el regalo? http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre
[sqlite] RE:Re: [sqlite] RE: SQLite memory leak on Windows CE
Hi Robert, Thank you for your test. I have not test it on the emulator but in 4 different Windows CE devices i have at work (with different Windows CE OS versions) and it always give me the same memory leak result. I will run your test tomorrow at work using the emulator and i will let you know the results. Thank you again, Eduardo ---Mensaje original---Ok, here's what I did ... On the desktop I created a SQLite database with one table and inserted 120,000 rows into it. I then copied it over to the CE emulator. Then I ran the following code on the CE (Pocket PC 2003 SE) emulator: #include #include #include sqlite3.h int WINAPI _tWinMain(HINSTANCE hInst, HINSTANCE hPrev, LPTSTR pszCmdLine, int nCmdShow) { sqlite3 *pdb; int rc; for (int n = 0; n 1; n ) { rc = sqlite3_open(\\test.db3, pdb); if (rc) break; rc = sqlite3_exec(pdb, select * from testcase, 0, 0, 0); if (rc) break; rc = sqlite3_close(pdb); if (rc) break; } return 0; } On the first call to sqlite3_exec(), available program memory dropped from an initial 10.5mb to 8.45mb. However, once that was done, available memory remained rock solid for the duration of the loop at a constant 8.45mb and no non-zero error codes were ever returned. Robert El día del padre está cerca... ¿Ya tienes el regalo? http://buscawanadoo.es/search?type=pref=web=homespot=Regalos%20Día%20del%20Padre