Actually, I'd propose to turn off multiprocessing at prediction time - this
might backfire quite easily.
2013/12/20 Olivier Grisel <[email protected]>
> 2013/12/20 Vlad Niculae <[email protected]>:
> > Works exactly as you described on my machine (which doesn't mean much
> > because it's relatively close to yours, but I am just too enthusiastic
> > about this not to reply! \o/)
> >
> > Memory usage is as expected. I see a speedup in train time but a
> > slight slowdown in test time (1.7 vs 1.0), is it expected or probably
> > an artefact?
>
> Threading is not (yet) used at test time as the cython code backing
> the predict method would need to be refactored to release the GIL to
> make threading efficient.
>
> So the performance speed decrease you observe might be caused by the
> new automated memmaping feature that dumps large arrays to use share
> memory with with worker process when the multiprocessing backend is
> used. Currently the threshold to trigger the automated memmaping is
> set to 1MB arrays or larger. Maybe this is too small and we should
> trigger it only for arrays larger than 100MB for instance.
>
> How big is the data array in your case, is this the covertype benchmark?
>
> --
> Olivier
>
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics
> Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
--
Peter Prettenhofer
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
