This is not really Theano specific, but I write Python iterators to handle data loading for most ML tasks, and make sure the run loop does something like .next() or python for loops (which expect StopIteration() at the end of the iterator. As a bonus, if you are not CPU bound you can fetch data for the next minibatches while the GPU crunches. If you have some complicated preprocessing it might be necessary to use shared memory or multiprocessing to avoid GIL issues, but my use cases have not run into that so far.
An example that loosely matches what I do can be seen here https://gist.github.com/kastnerkyle/33e2172bafbb5dd794ab On Thu, Sep 15, 2016 at 11:27 AM, Jose Carranza <[email protected]> wrote: > Hi guys > > I have a fairly big dataset (million+ images for train) that I want to use > to train from scratch a model in Theano. In Caffe we use LMDB however I > haven't seen any best practice in Theano for something bigger than MNIST and > stuff like that. Can somebody suggest what is the best option to pull data > into Theano/Lasagne? I need something that is not 100% in memory but that > can pull in batches (hopefully also shuffled batches). > > Thx in advance > > -- > > --- > You received this message because you are subscribed to the Google Groups > "theano-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
