Re: Apache Spark - Structured Streaming graceful shutdown

[M Singh]

Thanks Eyal - it appears that these are the same patterns used for spark 
DStreams. 

On Wednesday, December 27, 2017 1:15 AM, Eyal Zituny 
 wrote:
 

 Hiif you're interested in stopping you're spark application externally, you 
will probably need a way to communicate with the spark driver  (which start and 
holds a ref to the spark context)this can be done by adding some code to the 
driver app, for example:   
   - you can expose a rest api that stop the query and the spark context   

   - if running in client mode you can listen to stdin   

   - you can also listen to an external system (like kafka)
Eyal
On Tue, Dec 26, 2017 at 10:37 PM, M Singh  wrote:

Thanks Diogo.  My question is how to gracefully call the stop method while the 
streaming application is running in a cluster.

 

On Monday, December 25, 2017 5:39 PM, Diogo Munaro Vieira 
 wrote:
 

 Hi M Singh! Here I'm using query.stop()
Em 25 de dez de 2017 19:19, "M Singh"  escreveu:

Hi:Are there any patterns/recommendations for gracefully stopping a structured 
streaming application ?Thanks





   



   

Re: Apache Spark - Structured Streaming graceful shutdown

[Eyal Zituny]

Hi
if you're interested in stopping you're spark application externally, you
will probably need a way to communicate with the spark driver  (which start
and holds a ref to the spark context)
this can be done by adding some code to the driver app, for example:

   - you can expose a rest api that stop the query and the spark context
   - if running in client mode you can listen to stdin
   - you can also listen to an external system (like kafka)

Eyal

On Tue, Dec 26, 2017 at 10:37 PM, M Singh 
wrote:

> Thanks Diogo.  My question is how to gracefully call the stop method while
> the streaming application is running in a cluster.
>
>
>
>
> On Monday, December 25, 2017 5:39 PM, Diogo Munaro Vieira <
> diogo.mun...@corp.globo.com> wrote:
>
>
> Hi M Singh! Here I'm using query.stop()
>
> Em 25 de dez de 2017 19:19, "M Singh" 
> escreveu:
>
> Hi:
> Are there any patterns/recommendations for gracefully stopping a
> structured streaming application ?
> Thanks
>
>
>
>
>

Re: Apache Spark - Structured Streaming graceful shutdown

[M Singh]

Thanks Diogo.  My question is how to gracefully call the stop method while the 
streaming application is running in a cluster.

 

On Monday, December 25, 2017 5:39 PM, Diogo Munaro Vieira 
 wrote:
 

 Hi M Singh! Here I'm using query.stop()
Em 25 de dez de 2017 19:19, "M Singh"  escreveu:

Hi:Are there any patterns/recommendations for gracefully stopping a structured 
streaming application ?Thanks





   

Re: Apache Spark - Structured Streaming graceful shutdown

[Diogo Munaro Vieira]

Hi M Singh! Here I'm using query.stop()

Em 25 de dez de 2017 19:19, "M Singh" 
escreveu:

> Hi:
> Are there any patterns/recommendations for gracefully stopping a
> structured streaming application ?
> Thanks
>
>
>

Apache Spark - Structured Streaming graceful shutdown

[M Singh]

Hi:Are there any patterns/recommendations for gracefully stopping a structured 
streaming application ?Thanks