Re: [Structured Streaming] Restarting streaming query on exception/termination

2018-04-24 Thread Arun Mahadevan 
I guess you can wait for the termination, catch exception and then restart the 
query in a loop. Something like…

while (true) {
  try {
val query = df.writeStream().
…
   .start()
query.awaitTermination()
  } catch {
case e: StreamingQueryException => // log it
  }
}

Thanks,
Arun

From:  Priyank Shrivastava <priya...@gmail.com>
Date:  Monday, April 23, 2018 at 11:27 AM
To:  formice <51296...@qq.com>, "user@spark.apache.org" <user@spark.apache.org>
Subject:  Re: [Structured Streaming] Restarting streaming query on 
exception/termination

Thanks for the reply formice.  I think that --supervise param helps to restart 
the whole spark application - what I want to be able to do is to only restart 
the structured streaming query which terminated due to error. Also, I am 
running my app in client mode. 

Thanks,
Priyank

On Sun, Apr 22, 2018 at 8:52 PM, formice <51296...@qq.com> wrote:
standlone 
  add  config:(1)--deploy-mode cluster (2)--supervise
  example:  spark-submit  --master spark://master:7077 --deploy-mode 
cluster --supervise ..


-- 原始邮件 --
发件人: "Priyank Shrivastava"<priya...@gmail.com>;
发送时间: 2018年4月21日(星期六) 凌晨5:45
收件人: "user"<user@spark.apache.org>;
主题: [Structured Streaming] Restarting streaming query on exception/termination

What's the right way of programmatically restarting a structured streaming 
query which has terminated due to an exception? Example code or reference would 
be appreciated.

Could it be done from within the onQueryTerminated() event handler of 
StreamingQueryListener class?

Priyank

Re: [Structured Streaming] Restarting streaming query on exception/termination

2018-04-23 Thread Priyank Shrivastava 
Thanks for the reply formice.  I think that --supervise param helps to
restart the whole spark application - what I want to be able to do is to
only restart the structured streaming query which terminated due to error.
Also, I am running my app in client mode.

Thanks,
Priyank

On Sun, Apr 22, 2018 at 8:52 PM, formice <51296...@qq.com> wrote:

> standlone
>   add  config:(1)--deploy-mode cluster (2)--supervise
>   example:  spark-submit  --master spark://master:7077 --deploy-mode
> cluster --supervise ..
>
>
> -- 原始邮件 --
> *发件人:* "Priyank Shrivastava"<priya...@gmail.com>;
> *发送时间:* 2018年4月21日(星期六) 凌晨5:45
> *收件人:* "user"<user@spark.apache.org>;
> *主题:* [Structured Streaming] Restarting streaming query on
> exception/termination
>
> What's the right way of programmatically restarting a structured streaming
> query which has terminated due to an exception? Example code or reference
> would be appreciated.
>
> Could it be done from within the onQueryTerminated() event handler of
> StreamingQueryListener class?
>
> Priyank
>
>

[Structured Streaming] Restarting streaming query on exception/termination

2018-04-20 Thread Priyank Shrivastava 
What's the right way of programmatically restarting a structured streaming
query which has terminated due to an exception? Example code or reference
would be appreciated.

Could it be done from within the onQueryTerminated() event handler of
StreamingQueryListener class?

Priyank