wordCounts.explain() -> query.explain()?
Chang Chen wrote
> Hi Guys
>
> I modified StructuredNetworkWordCount to see what the executed plan is,
> here are my codes:
>
> val wordCounts = words.groupBy("value").count()
>
> // Start running the query that prints the running counts to the console
I have used a very similar script, I think there might be some extra steps
that are needed before it could be as robust as toPandas. If you look at
_to_corrected_pandas_type in the toPandas
(https://github.com/apache/spark/blob/master/python/pyspark/sql/dataframe.py#L1869),
this would have to be im
Hi Guys
I modified StructuredNetworkWordCount to see what the executed plan is,
here are my codes:
val wordCounts = words.groupBy("value").count()
// Start running the query that prints the running counts to the console
val query = wordCounts.writeStream
.outputMode("complete")
.format("cons
