Or maybe simpler, use hdf5 file format. There is the pytables Python package that support it. It will ask an numpy array, but will load the element you select efficiently.
Le 15 sept. 2016 13:03, "Kyle Kastner" <kastnerk...@gmail.com> a écrit : > This is not really Theano specific, but I write Python iterators to > handle data loading for most ML tasks, and make sure the run loop does > something like .next() or python for loops (which expect > StopIteration() at the end of the iterator. As a bonus, if you are not > CPU bound you can fetch data for the next minibatches while the GPU > crunches. If you have some complicated preprocessing it might be > necessary to use shared memory or multiprocessing to avoid GIL issues, > but my use cases have not run into that so far. > > An example that loosely matches what I do can be seen here > https://gist.github.com/kastnerkyle/33e2172bafbb5dd794ab > > On Thu, Sep 15, 2016 at 11:27 AM, Jose Carranza > <jose.carranza...@gmail.com> wrote: > > Hi guys > > > > I have a fairly big dataset (million+ images for train) that I want to > use > > to train from scratch a model in Theano. In Caffe we use LMDB however I > > haven't seen any best practice in Theano for something bigger than MNIST > and > > stuff like that. Can somebody suggest what is the best option to pull > data > > into Theano/Lasagne? I need something that is not 100% in memory but that > > can pull in batches (hopefully also shuffled batches). > > > > Thx in advance > > > > -- > > > > --- > > You received this message because you are subscribed to the Google Groups > > "theano-users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to theano-users+unsubscr...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "theano-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to theano-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
