Hello, I have a trained ExtraTreesRegressor saved using joblib.dump
(without compress). This creates more than ten thousand files, each
weighing less than 100KB. When trying to load using joblib.load with
mmap_mode="r" I get a [Errno 24] Too many open files. Is there a way to
save my regressor without creating so many files? I know compress is an
option but then i wouldn't be able to use mmap_mode, and speed is an issue
in my implementation. Any ideas? Would it even help to use memory mapping?
On Fri, Dec 20, 2013 at 3:48 AM, Olivier Grisel <[email protected]>wrote:
> 2013/12/20 Fred Mailhot <[email protected]>:
> > On 19 December 2013 15:16, Olivier Grisel <[email protected]>
> wrote:
> >>
> >> [...]
> >> But on the other hand that makes it possible to [...] to memory map the
> >> large parameter
> >>
> >> arrays by passing mmap_mode='r' to joblib.load for instance.
> >>
> >> Memory mapping can be useful to share the memory of models loaded in
> >> several python process (e.g. gunicorn wsgi workers [...]
> >
> >
> > Is this explained in more detail in documentation anywhere?
>
> The (short) reference documentation of joblib.load is here:
>
> http://pythonhosted.org/joblib/generated/joblib.load.html#joblib.load
>
> If you want to know more about memory mapping:
>
> http://en.wikipedia.org/wiki/Memory-mapped_file
>
> And numpy support for memmory mapping that joblib is leveraging under the
> hood:
>
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html
>
> --
> Olivier
> http://twitter.com/ogrisel - http://github.com/ogrisel
>
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics
> Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
