where very bad things (TM) is big matrices constantly swapping places with
each other, so no computing can actually happen.
Another thing to try might be numpy.memmap - I have seen some very great
savings on parallel, large matrix work using it (since the tasks I was
doing only needed one sample at a time i.e. not batch! Very wasteful to
load the whole thing into memory in my case).
TBH, sometimes the very simplest thing after trying these things is to buy
more RAM - DDR3 is pretty cheap these days. I also know that isn't always
an option, and is definitely not the scalable solution :)
On Tue, May 27, 2014 at 3:48 PM, Kyle Kastner <[email protected]> wrote:
> What is your overall memory usage like when this happens? Sounds like
> classic memory swapping/thrashing to me - what are your system specs?
>
> One quick thing to try might be to change the dtype of the matrices to
> save some space. float32 vs float64 can make a large memory difference if
> you don't need double precision. Also as far as I know, sklearn/joblib
> doesn't do any kind of scheduling or optimization based on available
> resources, though someone may correct me here. This means that if required
> memory to run n jobs is >> than your system memory, very bad things (TM)
> will happen
>
>
> On Tue, May 27, 2014 at 3:39 PM, Chris Holdgraf <[email protected]>wrote:
>
>> In particular, it seems that when I've got matrices which are too big,
>> the forked processes will hang and never finish (aka, they take up 0
>> computing time and remain that way indefinitely).
>>
>> In particular, I've noticed this problem when using cross_val_score with
>> Ridge regression. This isn't a problem when I have input matrices on the
>> order of 800,000 by 100, but it does hang when they're on the order of
>> 800,000 by 800. If I don't use parallel, then it fits fine, but setting
>> n_jobs > 1 will create a hang.
>>
>> I'm using anaconda on CentOS, and I've tried this both with and without
>> MKL optimizations.
>>
>> Anyone experience anything like this?
>>
>> (if you'd like more detail, I also opened this as an issue on the sklearn
>> repo)
>>
>> Chris
>>
>>
>> --
>> _____________________________________
>>
>> PhD Candidate in Neuroscience | UC Berkeley <http://hwni.org/>
>> Editor and Web Master | Berkeley Science
>> Review<http://sciencereview.berkeley.edu/>
>> _____________________________________
>>
>>
>> ------------------------------------------------------------------------------
>> The best possible search technologies are now affordable for all
>> companies.
>> Download your FREE open source Enterprise Search Engine today!
>> Our experts will assist you in its installation for $59/mo, no commitment.
>> Test it for FREE on our Cloud platform anytime!
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=145328191&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Scikit-learn-general mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>
------------------------------------------------------------------------------
The best possible search technologies are now affordable for all companies.
Download your FREE open source Enterprise Search Engine today!
Our experts will assist you in its installation for $59/mo, no commitment.
Test it for FREE on our Cloud platform anytime!
http://pubads.g.doubleclick.net/gampad/clk?id=145328191&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
