On Tuesday, 13 September 2016 15:55:13 BST Mallika Agarwal wrote: > This is more of a Python question I suppose but considering I have to > forward the data into a Theano CNN, I think it's ultimately relevant here. > > I have 14k train and 167k test images (vectors) of size 200*200 = 40k.
Hello, yes this might be too much for pickle, it is not very memory efficient at runtime. For these cases I have found that using HDF5 is a much better option, it has many desirable features and it handles terabyte-sized datasets with no problem. It can also be read/write in many languages. So just use something like h5py to read/write your data. -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
