Le 12 févr. 2012 à 08:44, Davide Alberani a écrit :
>
> Well, it's not exactly nice & clean code... let's hope it works.
If prefer nice features to clean code :D
> If you believe at the theory, the return value of a 'select' method is
> a lazy object, and so it should not consume much memory.
> It's true that it's then cycled over, one item at a time, and so it
> may be slow (in which case, we can directly use a cursors and
> process the results in batches).
Fair enough !
When it was selecting all the not null ids, the memory of the process grows up
and the size of the .db never grows up.
My theory is that dbm save on close ? Does that make sense ?
>
>> It seems that you load all the datas in memory before storing it in the temp
>> databases.
>> "cls.select(ISNOTNULL(cls.q.imdbID))"
>>
>> Maybe you should save the imdbids by batch of 10000entries ?
>
> If you believe at the theory, the return value of a 'select' method is
> a lazy object, and so it should not consume much memory.
> It's true that it's then cycled over, one item at a time, and so it
> may be slow (in which case, we can directly use a cursors and
> process the results in batches).
>
>> Tell me if you need the complete database dump to test with tons of datas !
>
> It won't hurt... :)
> A dump of "SELECT md5sum, imdb_id" from the 'name', 'title', 'char_name'
> and 'company_name' will be enough (not on the public list, please :))
>
>
> Bye,
>
> --
> Davide Alberani <davide.alber...@gmail.com> [PGP KeyID: 0x465BFD47]
> http://www.mimante.net/
------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
Imdbpy-devel mailing list
Imdbpy-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/imdbpy-devel
