2012/1/2 Gael Varoquaux <gael.varoqu...@normalesup.org>: > On Fri, Dec 30, 2011 at 02:16:51PM -0800, Dav Clark wrote: >> I'll add that it would be cool to see what numbers you come up with >> (maybe with some loose specs on the machine CPU and disk you used). > > Yes, I have been working on this for a little while, and it is actually > full of surprises. I have prepared a blog post (not exactly focused on > pytables, sorry), where I discuss amongst other things the results and > the lessons learned. Before I put it on-line, I'd appreciate if the > pytables experts could give a look to the relevant parts, just to ensure > that I am not using pytables wrong, or talking too much nonsens about > pytables. > > I have uploaded a temporary version on > http://gael-varoquaux.info/blog/wp-content/uploads/2012/joblib_rel_0.6_speed/tmp.html [clip]
That's a nice blog entry indeed. In my opinion, the data regarding PyTables makes sense in general. Perhaps you may get a bit more performance if you use the `[read,write]_vl_blosc2_hdf` functions that I have sent in my earlier post, but that adds the python-blosc dependency (available at http://pypi.python.org/pypi/blosc/1.0.3), so yeah, that might be a bit 'exotic'. Also, as your datasets are pretty small, you may want to add a warning about the fact that these benchmarks are mainly doing I/O against memory, not disk. BTW, another cleaner, faster way to empty the OS filesystem cache is this: sudo echo 3 | sudo tee /proc/sys/vm/drop_caches [ from http://ubuntuforums.org/showthread.php?t=589975 ] But that requires sudo permission indeed. > I am still working on this version, as well as on commenting my > persistence code, but feel free to comment. Hopefully this can be useful > to the pytables community. Sure, it is. Thanks ! -- Francesc Alted ------------------------------------------------------------------------------ Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex infrastructure or vast IT resources to deliver seamless, secure access to virtual desktops. With this all-in-one solution, easily deploy virtual desktops for less than the cost of PCs and save 60% on VDI infrastructure costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
