Re: [sqlite] Is there any memory leak in the normal routine?
Thank you very much!
I test my code enough time, and the memory using indeed stop increases when
the process's memory using is 3524 RES.
And I try to "PRAGMA default_cache_size=100". Then the memory is just 1324
and stop increase.
Black, Michael (IS) wrote:
>
> I confirmed your "memory leak". What you're seeing is the page cache
> growing. Not really a memory leak. Default page cache size is 2000 and
> indeed if I just let it run it topped out at 5404 RES in top.
>
>
> I added dmalloc to sqlite3 and found that if you let your program loop
> several times and politely exit there is no memory leak.
>
> So I added some debug statements in sqlite3.c to the alloc and free
> sections. It produced an output like this:
> alloc 664 0x2ac150a61408
> alloc 3224 0x2aab9008
> alloc 360 0x2aaab008
> alloc 1016 0x2aac7808
> free 0x2aab9008
> free 0x2ac150a61408
> free 0x2aac7808
> free 0x2aaab008
>
> I ran just two iterations of your program and edited the log to remove
> everything except the iteration between 1 & 2.
>
> "grep alloc log | wc -l" and "grep free log" show that the 1st loop
> through your program shows 811 allocs and 809 frees. So there are two
> places where memory is not getting freed.
>
> I then wrote a quick log analysis that showed
> free not found for 0x2ac150a63808
> free not found for 0x2aac7008
>
> grep for those two addresses showed
>
> alloc 1280 0x2ac150a63808
> alloc 1280 0x2aac7008
>
> There are only two 1280 alloc's in the log do I added a debug statement in
> sqlite3MemMalloc to trigger on 1280 bytes.
> Stepping through the debug then showed me at this point (line#'s are
> approximate as I've added statements) for both alloc's that occur:
> Breakpoint 1, sqlite3MemMalloc (nByte=1272) at sqlite3.c:12977
> #0 sqlite3MemMalloc (nByte=1272) at sqlite3.c:12977
> #1 0x00405b17 in mallocWithAlarm (n=1272, pp=0x7fff655c1248) at
> sqlite3.c:16343
> #2 0x00405bbc in sqlite3Malloc (n=1272) at sqlite3.c:16371
> #3 0x0040fc5c in pcache1Alloc (nByte=1272) at sqlite3.c:31163
> #4 0x0040fd39 in pcache1AllocPage (pCache=0x2acd74d1eb90) at
> sqlite3.c:31197
> #5 0x004105ca in pcache1Fetch (p=0x2acd74d1eb90, iKey=1,
> createFlag=2) at sqlite3.c:31566
> #6 0x0040f2f3 in sqlite3PcacheFetch (pCache=0x2acd74d288f0,
> pgno=1, createFlag=1,
> ppPage=0x7fff655c1400) at sqlite3.c:30645
> #7 0x0041470b in sqlite3PagerAcquire (pPager=0x2acd74d28810,
> pgno=1, ppPage=0x7fff655c1400,
> noContent=0) at sqlite3.c:36020
> #8 0x00418492 in btreeGetPage (pBt=0x2acd74d29c10, pgno=1,
> ppPage=0x7fff655c1450, noContent=0)
> at sqlite3.c:40101
> #9 0x004193bc in lockBtree (pBt=0x2acd74d29c10) at
> sqlite3.c:40811
> #10 0x004199a2 in sqlite3BtreeBeginTrans (p=0x2acd74d1ee90,
> wrflag=0) at sqlite3.c:41061
> #11 0x00455de7 in sqlite3InitOne (db=0x2acd74d28c10, iDb=0,
> pzErrMsg=0x2acd74d28420)
> at sqlite3.c:79614
> #12 0x0045629d in sqlite3Init (db=0x2acd74d28c10,
> pzErrMsg=0x2acd74d28420) at sqlite3.c:79784
> #13 0x00456384 in sqlite3ReadSchema (pParse=0x2acd74d28410) at
> sqlite3.c:79821
> #14 0x00442d4a in sqlite3StartTable (pParse=0x2acd74d28410,
> pName1=0x2aab90e8,
> pName2=0x2aab9108, isTemp=0, isView=0, isVirtual=0, noErr=0) at
> sqlite3.c:68162
> #15 0x0046bf2e in yy_reduce (yypParser=0x2aab9010,
> yyruleno=26) at sqlite3.c:94009
> #16 0x0046f67e in sqlite3Parser (yyp=0x2aab9010, yymajor=22,
> yyminor=
> {z = 0x47b36e "(id INTEGER PRIMARY KEY, d1 CHAR(16))", n = 1},
> pParse=0x2acd74d28410)
> at sqlite3.c:95191
> #17 0x0047037b in sqlite3RunParser (pParse=0x2acd74d28410,
> zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1
> CHAR(16))", pzErrMsg=0x7fff655c1980)
> at sqlite3.c:96017
> #18 0x004566fd in sqlite3Prepare (db=0x2acd74d28c10,
> zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1
> CHAR(16))", nBytes=-1,
> saveSqlFlag=0, pReprepare=0x0, ppStmt=0x7fff655c1ac8,
> pzTail=0x7fff655c1ad0) at sqlite3.c:79995
> #19 0x00456a2b in sqlite3LockAndPrepare (db=0x2acd74d28c10,
> zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1
> CHAR(16))", nBytes=-1,
> saveSqlFlag=0, pOld=0x0, ppStmt=0x7fff655c1ac8, pzTail=0x7fff655c1ad0)
> at sqlite3.c:80090
> #20 0x00456b97 in sqlite3_prepare (db=0x2acd74d28c10,
> zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1
> CHAR(16))", nBytes=-1,
> ppStmt=0x7fff655c1ac8, pzTail=0x7fff655c1ad0) at sqlite3.c:80153
> #21 0x00451bc7 in sqlite3_exec (db=0x2acd74d28c10,
> zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1
> CHAR(16))", xCallback=0, pArg=0x0,
> pzErrMsg=0x0) at sqlite3.c:76835
> #22 0x00401d8a in main () at thread.c:16
>
>
>
> Michael D. Black
> Senior Scientist
> Northrop Grumman
Re: [sqlite] Is there any memory leak in the normal routine?
I confirmed your "memory leak". What you're seeing is the page cache growing.
Not really a memory leak. Default page cache size is 2000 and indeed if I just
let it run it topped out at 5404 RES in top.
I added dmalloc to sqlite3 and found that if you let your program loop several
times and politely exit there is no memory leak.
So I added some debug statements in sqlite3.c to the alloc and free sections.
It produced an output like this:
alloc 664 0x2ac150a61408
alloc 3224 0x2aab9008
alloc 360 0x2aaab008
alloc 1016 0x2aac7808
free 0x2aab9008
free 0x2ac150a61408
free 0x2aac7808
free 0x2aaab008
I ran just two iterations of your program and edited the log to remove
everything except the iteration between 1 & 2.
"grep alloc log | wc -l" and "grep free log" show that the 1st loop through
your program shows 811 allocs and 809 frees. So there are two places where
memory is not getting freed.
I then wrote a quick log analysis that showed
free not found for 0x2ac150a63808
free not found for 0x2aac7008
grep for those two addresses showed
alloc 1280 0x2ac150a63808
alloc 1280 0x2aac7008
There are only two 1280 alloc's in the log do I added a debug statement in
sqlite3MemMalloc to trigger on 1280 bytes.
Stepping through the debug then showed me at this point (line#'s are
approximate as I've added statements) for both alloc's that occur:
Breakpoint 1, sqlite3MemMalloc (nByte=1272) at sqlite3.c:12977
#0 sqlite3MemMalloc (nByte=1272) at sqlite3.c:12977
#1 0x00405b17 in mallocWithAlarm (n=1272, pp=0x7fff655c1248) at
sqlite3.c:16343
#2 0x00405bbc in sqlite3Malloc (n=1272) at sqlite3.c:16371
#3 0x0040fc5c in pcache1Alloc (nByte=1272) at sqlite3.c:31163
#4 0x0040fd39 in pcache1AllocPage (pCache=0x2acd74d1eb90) at
sqlite3.c:31197
#5 0x004105ca in pcache1Fetch (p=0x2acd74d1eb90, iKey=1, createFlag=2)
at sqlite3.c:31566
#6 0x0040f2f3 in sqlite3PcacheFetch (pCache=0x2acd74d288f0, pgno=1,
createFlag=1,
ppPage=0x7fff655c1400) at sqlite3.c:30645
#7 0x0041470b in sqlite3PagerAcquire (pPager=0x2acd74d28810, pgno=1,
ppPage=0x7fff655c1400,
noContent=0) at sqlite3.c:36020
#8 0x00418492 in btreeGetPage (pBt=0x2acd74d29c10, pgno=1,
ppPage=0x7fff655c1450, noContent=0)
at sqlite3.c:40101
#9 0x004193bc in lockBtree (pBt=0x2acd74d29c10) at sqlite3.c:40811
#10 0x004199a2 in sqlite3BtreeBeginTrans (p=0x2acd74d1ee90, wrflag=0)
at sqlite3.c:41061
#11 0x00455de7 in sqlite3InitOne (db=0x2acd74d28c10, iDb=0,
pzErrMsg=0x2acd74d28420)
at sqlite3.c:79614
#12 0x0045629d in sqlite3Init (db=0x2acd74d28c10,
pzErrMsg=0x2acd74d28420) at sqlite3.c:79784
#13 0x00456384 in sqlite3ReadSchema (pParse=0x2acd74d28410) at
sqlite3.c:79821
#14 0x00442d4a in sqlite3StartTable (pParse=0x2acd74d28410,
pName1=0x2aab90e8,
pName2=0x2aab9108, isTemp=0, isView=0, isVirtual=0, noErr=0) at
sqlite3.c:68162
#15 0x0046bf2e in yy_reduce (yypParser=0x2aab9010, yyruleno=26) at
sqlite3.c:94009
#16 0x0046f67e in sqlite3Parser (yyp=0x2aab9010, yymajor=22,
yyminor=
{z = 0x47b36e "(id INTEGER PRIMARY KEY, d1 CHAR(16))", n = 1},
pParse=0x2acd74d28410)
at sqlite3.c:95191
#17 0x0047037b in sqlite3RunParser (pParse=0x2acd74d28410,
zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1
CHAR(16))", pzErrMsg=0x7fff655c1980)
at sqlite3.c:96017
#18 0x004566fd in sqlite3Prepare (db=0x2acd74d28c10,
zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1
CHAR(16))", nBytes=-1,
saveSqlFlag=0, pReprepare=0x0, ppStmt=0x7fff655c1ac8,
pzTail=0x7fff655c1ad0) at sqlite3.c:79995
#19 0x00456a2b in sqlite3LockAndPrepare (db=0x2acd74d28c10,
zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1
CHAR(16))", nBytes=-1,
saveSqlFlag=0, pOld=0x0, ppStmt=0x7fff655c1ac8, pzTail=0x7fff655c1ad0) at
sqlite3.c:80090
#20 0x00456b97 in sqlite3_prepare (db=0x2acd74d28c10,
zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1
CHAR(16))", nBytes=-1,
ppStmt=0x7fff655c1ac8, pzTail=0x7fff655c1ad0) at sqlite3.c:80153
#21 0x00451bc7 in sqlite3_exec (db=0x2acd74d28c10,
zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1
CHAR(16))", xCallback=0, pArg=0x0,
pzErrMsg=0x0) at sqlite3.c:76835
#22 0x00401d8a in main () at thread.c:16
Michael D. Black
Senior Scientist
Northrop Grumman Mission Systems
From: sqlite-users-boun...@sqlite.org on behalf of liubin liu
Sent: Sat 4/24/2010 1:58 AM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] Is there any memory leak in the normal routine?
And I watch the the process's run by top.
At first, the memory statistics is:
PID PPID USER STAT VSZ%MEM %CPU COMMAND
17731 15488 root S1104 5% 7% ./sqlite3multiwrite
Re: [sqlite] Is there any memory leak in the normal routine?
And I watch the the process's run by top. At first, the memory statistics is: PID PPID USER STAT VSZ%MEM %CPU COMMAND 17731 15488 root S1104 5% 7% ./sqlite3multiwrite When the printf() prints the 150, the memory statistics is: PID PPID USER STAT VSZ%MEM %CPU COMMAND 17731 15488 root S1552 5% 7% ./sqlite3multiwrite It means that after 150 for-cycles, the memory used by sqlite3multiwrite increase from 1104KB to 1552KB. What does it mean? memory leak or other thing? -- View this message in context: http://old.nabble.com/Is-there-any-memory-leak-in-the-normal-routine--tp28348648p28348725.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Is there any memory leak in the normal routine?
A routine of sqlite3_prepare_v2() + sqlite3_step() + sqlite3_finalize() could
contain leak.
It sound ridiculous. But the test code seems to say it. Or I used the
sqlite3_* wrongly.
__code__
#include
#include // for usleep()
#include
int multi_write (int j);
sqlite3 *db = NULL;
int main (void)
{
int ret = -1;
ret = sqlite3_open("test.db", );
ret = sqlite3_exec(db,"CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1
CHAR(16))", NULL,NULL,NULL);
usleep (10);
int j=0;
while (1)
{
multi_write (j++);
usleep (200);
printf (" --- %d\n", j);
}
ret = sqlite3_close (db);
return 0;
}
int multi_write (int j)
{
int ret = -1;
char *sql_f = "INSERT OR REPLACE INTO data_his VALUES (%d, %Q)";
char *sql = NULL;
sqlite3_stmt *p_stmt = NULL;
ret = sqlite3_prepare_v2 (db, "BEGIN TRANSACTION", -1, _stmt, NULL);
ret = sqlite3_step ( p_stmt );
ret = sqlite3_finalize ( p_stmt );
int i=0;
for (i=0; i<100; i++)
{
sql = sqlite3_mprintf ( sql_f, j*10 + i, "68FD");
ret = sqlite3_prepare_v2 (db, sql, -1, _stmt, NULL );
sqlite3_free ( sql );
//printf ("sqlite3_prepare_v2(): %d, %s\n", ret, sqlite3_errmsg
(db));
ret = sqlite3_step ( p_stmt );
//printf ("sqlite3_step(): %d, %s\n", ret, sqlite3_errmsg
(db));
ret = sqlite3_finalize ( p_stmt );
//printf ("sqlite3_finalize(): %d, %s\n\n", ret, sqlite3_errmsg
(db));
}
ret = sqlite3_prepare_v2 (db, "COMMIT TRANSACTION", -1, _stmt, NULL );
ret = sqlite3_step ( p_stmt );
ret = sqlite3_finalize ( p_stmt );
return 0;
}
--
View this message in context:
http://old.nabble.com/Is-there-any-memory-leak-in-the-normal-routine--tp28348648p28348648.html
Sent from the SQLite mailing list archive at Nabble.com.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
