Re: [sqlite] memory issue with large amount of inserts
I read pieces of information from other threads but I got more confused... I think I d better ask this question first: how exactly does sqlite manage memory usage, when does it attemp to release memory? Does page size/ number of pages affect insert operations? Does transactional-insert lock up memory and never release them? Because from my application it seems that as more inserts have been done in batches, memory consuption is increasing... I have tried close and reopen the connection but with no help. Any suggestions please, many thanks! zqzuk wrote: > > Hi, I am using SQLite 3.5.4, my application has encountered a memory > management issue which I think is caused by sqlite. I have searched the > forum and found this thread describing most similar situation as mine > > http://www.nabble.com/Memory-Usage-to13798003.html#a13850915 > > My application is accessing sqlite through JAVA on local machine, windows > xp sp2, there are 4 threads accessing 4 *different* databases at the same > time, each doing 2 million inserts to *different* databases. > > I have noticed that after about 3 hours, the system slowed down > dramatically, such that processing every 1000 records(then 1000 inserts) > took now 20 mins - 1 hr instead of sever seconds at the very beginning. I > then checked the resource monitor and found all of my 2 gb memory has been > used. > > I searched this forum and found talks about similar issues but solutions > seem to be re-compiling sqlite in C using new parameters, but I am using > JAVA and would like to know whether there is alternative/equivalent > solution? And also the above thread I quote mentioned that closing/reopen > connection can free up resources that sqlite occupy, i would take this as > a simple but dirty workaround? > > > Please may I have you suggestions , many thanks! > > -- View this message in context: http://www.nabble.com/memory-issue-with-large-amount-of-inserts-tp15225948p15231834.html Sent from the SQLite mailing list archive at Nabble.com. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] memory issue with large amount of inserts
just to add that I am using transaction, and the batch size is 2000 zqzuk wrote: > > Hi, I am using SQLite 3.5.4, my application has encountered a memory > management issue which I think is caused by sqlite. I have searched the > forum and found this thread describing most similar situation as mine > > http://www.nabble.com/Memory-Usage-to13798003.html#a13850915 > > My application is accessing sqlite through JAVA on local machine, windows > xp sp2, there are 4 threads accessing 4 *different* databases at the same > time, each doing 2 million inserts to *different* databases. > > I have noticed that after about 3 hours, the system slowed down > dramatically, such that processing every 1000 records(then 1000 inserts) > took now 20 mins - 1 hr instead of sever seconds at the very beginning. I > then checked the resource monitor and found all of my 2 gb memory has been > used. > > I searched this forum and found talks about similar issues but solutions > seem to be re-compiling sqlite in C using new parameters, but I am using > JAVA and would like to know whether there is alternative/equivalent > solution? And also the above thread I quote mentioned that closing/reopen > connection can free up resources that sqlite occupy, i would take this as > a simple but dirty workaround? > > > Please may I have you suggestions , many thanks! > > -- View this message in context: http://www.nabble.com/memory-issue-with-large-amount-of-inserts-tp15225948p15226252.html Sent from the SQLite mailing list archive at Nabble.com. - To unsubscribe, send email to [EMAIL PROTECTED] -
[sqlite] memory issue with large amount of inserts
Hi, I am using SQLite 3.5.4, my application has encountered a memory management issue which I think is caused by sqlite. I have searched the forum and found this thread describing most similar situation as mine http://www.nabble.com/Memory-Usage-to13798003.html#a13850915 My application is accessing sqlite through JAVA on local machine, windows xp sp2, there are 4 threads accessing 4 *different* databases at the same time, each doing 2 million inserts to *different* databases. I have noticed that after about 3 hours, the system slowed down dramatically, such that processing every 1000 records(then 1000 inserts) took now 20 mins - 1 hr instead of sever seconds at the very beginning. I then checked the resource monitor and found all of my 2 gb memory has been used. I searched this forum and found talks about similar issues but solutions seem to be re-compiling sqlite in C using new parameters, but I am using JAVA and would like to know whether there is alternative/equivalent solution? And also the above thread I quote mentioned that closing/reopen connection can free up resources that sqlite occupy, i would take this as a simple but dirty workaround? Please may I have you suggestions , many thanks! -- View this message in context: http://www.nabble.com/memory-issue-with-large-amount-of-inserts-tp15225948p15225948.html Sent from the SQLite mailing list archive at Nabble.com. - To unsubscribe, send email to [EMAIL PROTECTED] -
