For completeness: I was on the master branch of DataFrames and it was not working but moving DataArrays to master as well resolved the problem.
-James On Wednesday, January 15, 2014 8:54:15 PM UTC-5, John Myles White wrote: > > For now that’s the only public interface. It’s technically possible to use > any seekable IO stream, but that requires knowing implementation details at > the moment. > > — John > > On Jan 15, 2014, at 5:41 PM, James Fairbanks > <[email protected]<javascript:>> > wrote: > > > The documentation shows a constructor DataStream(). Is the only way to > get a DataStream to read it from a file using readstream? > > > > James > > > > On Thursday, January 9, 2014 11:22:18 AM UTC-5, John Myles White wrote: > > I just restored the DataStream functionality. Still need to revise its > documentation, but (assuming you’re on master for both DataArrays and > DataFrames) you can try: > > > > using DataFrames > > ds = readstream(“data.csv”, nrows = 100) > > > > for minibatch in ds > > show(minibatch, true) > > end > > > > — John > > > > On Jan 9, 2014, at 6:59 AM, Iain Dunning <[email protected]> wrote: > > > > > So we have readlines, which loads the file all at once into an array > of strings, and eachline, which allows us to iterate through a file line by > line. > > > > > > We have readcsv/readdlm, but don't have an iterating version (as far > as I know). This is a bit of a problem because it means I have to manually > parse CSV files that are too big to read in all at once, which is error > prone (e.g. quoted delimiter handling). Is there demand for this, and what > would it be called? > > > > > > Cheers, > > > Iain > > > >
