Okay thank you, let me try this out! On Wednesday, September 14, 2016 at 4:42:39 AM UTC+5:30, Matias Valdenegro wrote: > > On Tuesday, 13 September 2016 15:55:13 BST Mallika Agarwal wrote: > > This is more of a Python question I suppose but considering I have to > > forward the data into a Theano CNN, I think it's ultimately relevant > here. > > > > I have 14k train and 167k test images (vectors) of size 200*200 = 40k. > > Hello, yes this might be too much for pickle, it is not very memory > efficient > at runtime. > > For these cases I have found that using HDF5 is a much better option, it > has > many desirable features and it handles terabyte-sized datasets with no > problem. It can also be read/write in many languages. > > So just use something like h5py to read/write your data. >
-- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
