Re: Reusing dataframes for streaming (spark 1.6)

There is a DStream.transform() that does exactly this. On Tue, Aug 8, 2017 at 7:55 PM, Ashwin Raju wrote: > Hi, > > We've built a batch application on Spark 1.6.1. I'm looking into how to > run the same code as a streaming (DStream based) application. This is using > pyspark.

Reusing dataframes for streaming (spark 1.6)

Hi, We've built a batch application on Spark 1.6.1. I'm looking into how to run the same code as a streaming (DStream based) application. This is using pyspark. In the batch application, we have a sequence of transforms that read from file, do dataframe operations, then write to file. I was