Ronnie Ghose <ronnie.ghose@...> writes: > > > do you think it isn't saving correctly or it isn't loading correctly? > > On Thu, Jan 24, 2013 at 8:14 PM, Ark <ark_antos- /[email protected]> wrote: > Gael Varoquaux <gael.varoquaux <at> ...> writes: > > > > On Wed, Jan 23, 2013 at 12:16:32AM +0000, Afik Cohen wrote: > > Hi, I'm working with Ark on this project. Yes, that's what it looks like > > > - some investigation into this appears to show that either this is a bug > > > in zlib (the length returned is incorrect) or this is a bug in joblib.dump > > > (when called with compression set to anything other than 0, it might not > > > be writing the whole file out. We're using compress=9.) > > > > 1. Can you make a reproducible simple situation? If so I'll try to > > investigate. > > > > 2. Have you tried using compress=6? Compress=9 won't gain you much in > > size, and a good amount of trouble (it's slow). > Compress=6 also fails to create a valid(loaded correctly by joblib) classifier > object. The ~9-10gb uncompressed object is saved to ~1.6g npz > (with compress=6) file, but unfortunately the file returned seems to be > corrupted. See below for the simple script.https://gist.github.com/4630645 >
Hi Ronnie, We believe it isn't saving correctly. My hypothesis is that joblib is writing out a truncated file when using compress > 0. Whether this is related to the cStringIO issue further down this thread is not clear. It is still strange that we seem to be the only scenario with a classifier large enough (10GB) to expose problems when dumping with compression enabled. Aren't there other people out there using scikit with huge classifiers that they then save with compression? Thanks, Afik ------------------------------------------------------------------------------ Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. ON SALE this month only -- learn more at: http://p.sf.net/sfu/learnnow-d2d _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
