Makes sense. Thanks Holden.
Alexis
On Mon, Sep 12, 2016 at 5:28 PM, Holden Karau wrote:
> Ah yes so the Py4J conversions only apply on the driver program - your
> DStream however is RDDs of pickled objects. If you want to with a transform
> function use Spark SQL transferring DataFrames back an
Ah yes so the Py4J conversions only apply on the driver program - your
DStream however is RDDs of pickled objects. If you want to with a transform
function use Spark SQL transferring DataFrames back and forth between
Python and Scala spark can be much easier.
On Monday, September 12, 2016, Alexis
