There is a DStream.transform() that does exactly this.
On Tue, Aug 8, 2017 at 7:55 PM, Ashwin Raju wrote:
> Hi,
>
> We've built a batch application on Spark 1.6.1. I'm looking into how to
> run the same code as a streaming (DStream based) application. This is using
> pyspark.
>
> In the batch ap
Hi,
We've built a batch application on Spark 1.6.1. I'm looking into how to run
the same code as a streaming (DStream based) application. This is using
pyspark.
In the batch application, we have a sequence of transforms that read from
file, do dataframe operations, then write to file. I was hopin