Thanks Ryan! In this case, I will have Dataset<Row> so is there a way to
convert Row to Json string?
Thanks
On Sat, Sep 9, 2017 at 5:14 PM, Shixiong(Ryan) Zhu <shixi...@databricks.com>
wrote:
> It's because "toJSON" doesn't support Structured Streaming. The current
> implementation will convert the Dataset to an RDD, which is not supported
> by streaming queries.
>
> On Sat, Sep 9, 2017 at 4:40 PM, kant kodali <kanth...@gmail.com> wrote:
>
>> yes it is a streaming dataset. so what is the problem with following code?
>>
>> Dataset<String> ds = dataset.toJSON().map(()->{some function that returns a
>> string});
>> StreamingQuery query = ds.writeStream().start();
>> query.awaitTermination();
>>
>>
>> On Sat, Sep 9, 2017 at 4:20 PM, Felix Cheung <felixcheun...@hotmail.com>
>> wrote:
>>
>>> What is newDS?
>>> If it is a Streaming Dataset/DataFrame (since you have writeStream
>>> there) then there seems to be an issue preventing toJSON to work.
>>>
>>> ------------------------------
>>> *From:* kant kodali <kanth...@gmail.com>
>>> *Sent:* Saturday, September 9, 2017 4:04:33 PM
>>> *To:* user @spark
>>> *Subject:* Queries with streaming sources must be executed with
>>> writeStream.start()
>>>
>>> Hi All,
>>>
>>> I have the following code and I am not sure what's wrong with it? I
>>> cannot call dataset.toJSON() (which returns a DataSet) ? I am using spark
>>> 2.2.0 so I am wondering if there is any work around?
>>>
>>> Dataset<String> ds = newDS.toJSON().map(()->{some function that returns a
>>> string});
>>> StreamingQuery query = ds.writeStream().start();
>>> query.awaitTermination();
>>>
>>>
>>
>
