On Fri, Sep 7, 2012 at 4:18 PM, Olivier Grisel <[email protected]> wrote:
> 2012/9/7 Lars Buitinck <[email protected]>:
>> I just tried running the document classification example with all 20
>> classes, but the RidgeClassifier was taking so much memory that it
>> triggered the OOM killer. This didn't happen before, so it much be a
>> recent change to the codebase. Does anyone know what may be causing
>> this?
>
> No idea. This sound like a job for memory_profiler and the new
> breakpoint feature.
>
> Unfortunately it does not seem to work on my machine:
>
> $ sudo pip install -U git+https://github.com/fabianp/memory_profiler.git
> $ python -m memory_profiler --pdb-mmem=1 /tmp/tmp.py
> about to allocate
> done
> $ cat /tmp/tmp.py
> import numpy as np
> print "about to allocate"
> a = np.ones(1e8)
> print "done"
>
> Can somebody reproduce? Maybe Fabian?
Nothing like bad publicity to get things done :-). There was a bug
which is fixed now in the github. Otherwise you need to decorate the
function and give it a bigger constant as the interpreter already
takes around 8MB.
For example, the script would be
"""
@profile
def f():
import numpy as np
print "about to allocate"
a = np.ones(1e8)
print "done"
f()
"""
which now correctly stops at the hotspot:
$ (p27) python -m memory_profiler --pdb-mmem=20 tmp.py
[18:49:06]
about to allocate
Current memory 778.15 MB exceeded the maximum of 20.00 MB
Stepping into the debugger
> /Users/fabian/dev/memory_profiler/tmp.py(8)f()
-> a = np.ones(1e8)
(Pdb)
Hope that helps,
Fabian
>
> --
> Olivier
> http://twitter.com/ogrisel - http://github.com/ogrisel
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
