Re: [Scikit-learn-general] Joblib compression and LFW

2011-12-29 Thread Gael Varoquaux
On Thu, Dec 29, 2011 at 10:34:16AM -0800, Josh Bleecher Snyder wrote: > If you want to experiment with more options, you might also play with > blosc (http://blosc.pytables.org/trac). The compression level is not > as good as heavier weight algorithms, but it is really zippy. I ended > up using it

Re: [Scikit-learn-general] Joblib compression and LFW

2011-12-29 Thread Josh Bleecher Snyder
> Obviously the fine-tuning that I did is not needed for the > scikit's storage of the datasets, but it general fast dump/load of Python > objects is useful for scientific computing and big data (think caching or > message passing parallel computing). If you want to experiment with more options, y

Re: [Scikit-learn-general] Joblib compression and LFW

2011-12-29 Thread Gael Varoquaux
On Wed, Dec 28, 2011 at 05:21:39PM +0100, Alexandre Gramfort wrote: > thanks Gael for the christmas present :) I just couldn't help playing more. I have pushed a new update that enables to control the compression level, and in general can achieve better compromises between speed and compression. H

Re: [Scikit-learn-general] Joblib compression and LFW

2011-12-28 Thread Alexandre Gramfort
thanks Gael for the christmas present :) Alex On Wed, Dec 28, 2011 at 11:45 AM, Olivier Grisel wrote: > Great job Gael! > > -- > Olivier > > -- > Write once. Port to many. > Get the SDK and tools to simplify cross-platfo

Re: [Scikit-learn-general] Joblib compression and LFW

2011-12-28 Thread Olivier Grisel
Great job Gael! -- Olivier -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM progr

[Scikit-learn-general] Joblib compression and LFW

2011-12-28 Thread Gael Varoquaux
Hi list, This message is not terribly informative. I just to share my current successes with joblib compression. I am a bit frustrated at the fact that the LFW cache takes 400M on my disk, for something that I never used. The disk space in the LFW cache is made of two major contributors: * The