For now that’s the only public interface. It’s technically possible to use any seekable IO stream, but that requires knowing implementation details at the moment.
— John On Jan 15, 2014, at 5:41 PM, James Fairbanks <[email protected]> wrote: > The documentation shows a constructor DataStream(). Is the only way to get a > DataStream to read it from a file using readstream? > > James > > On Thursday, January 9, 2014 11:22:18 AM UTC-5, John Myles White wrote: > I just restored the DataStream functionality. Still need to revise its > documentation, but (assuming you’re on master for both DataArrays and > DataFrames) you can try: > > using DataFrames > ds = readstream(“data.csv”, nrows = 100) > > for minibatch in ds > show(minibatch, true) > end > > — John > > On Jan 9, 2014, at 6:59 AM, Iain Dunning <[email protected]> wrote: > > > So we have readlines, which loads the file all at once into an array of > > strings, and eachline, which allows us to iterate through a file line by > > line. > > > > We have readcsv/readdlm, but don't have an iterating version (as far as I > > know). This is a bit of a problem because it means I have to manually parse > > CSV files that are too big to read in all at once, which is error prone > > (e.g. quoted delimiter handling). Is there demand for this, and what would > > it be called? > > > > Cheers, > > Iain >
