Re: [Imdbpy-devel] [sql] memory consumption

2012-02-12 Thread Davide Alberani
On Sun, Feb 12, 2012 at 14:20, Emmanuel Tabard m...@webitup.fr wrote: Fair enough ! When it was selecting all the not null ids, the memory of the process grows up and the size of the .db never grows up. My theory is that dbm save on close ? Does that make sense ? Strange (even if, being

Re: [Imdbpy-devel] [sql] memory consumption

2012-02-12 Thread Emmanuel Tabard
I was wondering, why don't you use the original dbs ? Something like that takes 3 seconds: CREATE TABLE title_extract SELECT imdb_id, md5sum FROM title WHERE imdb_id IS NOT NULL CREATE TABLE name_extract SELECT imdb_id, md5sum FROM name WHERE imdb_id IS NOT NULL And use your query to

Re: [Imdbpy-devel] [sql] memory consumption

2012-02-12 Thread Emmanuel Tabard
Here is a little workarround : -- Extract imdb_id and md5sum (6sec) CREATE TABLE title_extract SELECT imdb_id, md5sum FROM title WHERE imdb_id IS NOT NULL; CREATE TABLE name_extract SELECT imdb_id, md5sum FROM name WHERE imdb_id IS NOT NULL; -- Add indexes (12sec) ALTER TABLE title_extract

Re: [Imdbpy-devel] [sql] memory consumption

2012-02-12 Thread Davide Alberani
On Sun, Feb 12, 2012 at 16:50, Emmanuel Tabard m...@webitup.fr wrote: Here is a little workarround : Well, that's a very interesting solution, thanks. :-) Anyway, I have to think a little about it: storing the data in the filesystem granted us the possibility to split the CSV mode into 2

Re: [Imdbpy-devel] [sql] memory consumption

2012-02-12 Thread Emmanuel Tabard
Well, that's a very interesting solution, thanks. :-) I made a test run with this solution. Time to save/restore : 6minutes Restoring success : - People : 99.8777% - Movies : 99.8845% Anyway, I have to think a little about it: storing the data in the filesystem granted us the