Makes sense. Thanks Holden.
Alexis
On Mon, Sep 12, 2016 at 5:28 PM, Holden Karau wrote:
> Ah yes so the Py4J conversions only apply on the driver program - your
> DStream however is RDDs of pickled objects. If you want to with a transform
> function use Spark SQL transferring DataFrames back an
Ah yes so the Py4J conversions only apply on the driver program - your
DStream however is RDDs of pickled objects. If you want to with a transform
function use Spark SQL transferring DataFrames back and forth between
Python and Scala spark can be much easier.
On Monday, September 12, 2016, Alexis
Hi,
*TL;DR - I have what looks like a DStream of Strings in a PySpark
application. I want to send it as a DStream[String] to a Scala library.
Strings are not converted by Py4j, though.*
I'm working on a PySpark application that pulls data from Kafka using Spark
Streaming. My messages are string