File as a stream? I think you are confusing Spark Streaming with buffer reader. Spark streaming is meant to process batches of data (files, packets, messages) as they come in, infact utilizing time of packet reception as a way to create windows etc.
In your case you are better off reading the file, partitioning it & operating on each column individually if that makes more sense to you. Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi <https://twitter.com/mayur_rustagi> On Thu, May 1, 2014 at 3:24 PM, Laeeq Ahmed <laeeqsp...@yahoo.com> wrote: > Hi all, > > Is it possible to read and process multiple streams with spark. I have > eeg(brain waves) csv file with 23 columns Each column is one stream(wave) > and each column has one million values. > > I know one way to do it is to take transpose of the file and then give it > to spark and each mapper will get one or more waves out of the 23 waves, > but then it will be non-streaming problem and I want to read the file as > stream. Please correct me if I am wrong. > > I have to apply simple operations(mean and SD) on each window of a wave. > > Regards, > Laeeq > >
