Re: streaming and piping to R, sending all data in window to pipe()

Did you try inputs.repartition(1).foreachRDD(..)? Thanks Best Regards On Fri, Jul 17, 2015 at 9:51 PM, PAULI, KEVIN CHRISTIAN [AG-Contractor/1000] wrote: > Spark newbie here, using Spark 1.3.1. > > I’m consuming a stream and trying to pipe the data from the entire > window to R for analysis.

streaming and piping to R, sending all data in window to pipe()

Spark newbie here, using Spark 1.3.1. I’m consuming a stream and trying to pipe the data from the entire window to R for analysis. The R algorithm needs the entire dataset from the stream (everything in the window) in order to function properly; it can’t be broken up. So I tried doing a coales