Hi guys Thx for the suggestions. I ended up using HFD5. There is some nice framework also that uses it called Fuel. So far looks pretty good. Thx again.
On Friday, September 16, 2016 at 3:30:49 PM UTC+2, nouiz wrote: > > Or maybe simpler, use hdf5 file format. There is the pytables Python > package that support it. It will ask an numpy array, but will load the > element you select efficiently. > > Le 15 sept. 2016 13:03, "Kyle Kastner" <kastn...@gmail.com <javascript:>> > a écrit : > >> This is not really Theano specific, but I write Python iterators to >> handle data loading for most ML tasks, and make sure the run loop does >> something like .next() or python for loops (which expect >> StopIteration() at the end of the iterator. As a bonus, if you are not >> CPU bound you can fetch data for the next minibatches while the GPU >> crunches. If you have some complicated preprocessing it might be >> necessary to use shared memory or multiprocessing to avoid GIL issues, >> but my use cases have not run into that so far. >> >> An example that loosely matches what I do can be seen here >> https://gist.github.com/kastnerkyle/33e2172bafbb5dd794ab >> >> On Thu, Sep 15, 2016 at 11:27 AM, Jose Carranza >> <jose.car...@gmail.com <javascript:>> wrote: >> > Hi guys >> > >> > I have a fairly big dataset (million+ images for train) that I want to >> use >> > to train from scratch a model in Theano. In Caffe we use LMDB however I >> > haven't seen any best practice in Theano for something bigger than >> MNIST and >> > stuff like that. Can somebody suggest what is the best option to pull >> data >> > into Theano/Lasagne? I need something that is not 100% in memory but >> that >> > can pull in batches (hopefully also shuffled batches). >> > >> > Thx in advance >> > >> > -- >> > >> > --- >> > You received this message because you are subscribed to the Google >> Groups >> > "theano-users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to theano-users...@googlegroups.com <javascript:>. >> > For more options, visit https://groups.google.com/d/optout. >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "theano-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to theano-users...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
