I just restored the DataStream functionality. Still need to revise its
documentation, but (assuming you’re on master for both DataArrays and
DataFrames) you can try:
using DataFrames
ds = readstream(“data.csv”, nrows = 100)
for minibatch in ds
show(minibatch, true)
end
— John
On Jan 9, 2014, at 6:59 AM, Iain Dunning <[email protected]> wrote:
> So we have readlines, which loads the file all at once into an array of
> strings, and eachline, which allows us to iterate through a file line by line.
>
> We have readcsv/readdlm, but don't have an iterating version (as far as I
> know). This is a bit of a problem because it means I have to manually parse
> CSV files that are too big to read in all at once, which is error prone (e.g.
> quoted delimiter handling). Is there demand for this, and what would it be
> called?
>
> Cheers,
> Iain
