Re: flink/cancel & shutdown hooks

[Ufuk Celebi]

OK, the thing is that the JVMs are not shut down when you cancel the
task. Therefore no shut down hook is executed when you cancel.

You would have to execute bin/stop-cluster.sh to stop the JVM. Does
that make sense?


On Wed, Mar 8, 2017 at 3:34 PM, Dominik Safaric
 wrote:
> I’m not using YARN but instead of starting the cluster using 
> bin/start-cluster.sh
>
>> On 8 Mar 2017, at 15:32, Ufuk Celebi  wrote:
>>
>> On Wed, Mar 8, 2017 at 3:19 PM, Dominik Safaric
>>  wrote:
>>> The cluster consists of 4 workers and a master node.
>>
>> Are you starting the cluster via bin/start-cluster.sh or are you using
>> YARN etc.?
>

Re: flink/cancel & shutdown hooks

[Dominik Safaric]

I’m not using YARN but instead of starting the cluster using 
bin/start-cluster.sh 

> On 8 Mar 2017, at 15:32, Ufuk Celebi  wrote:
> 
> On Wed, Mar 8, 2017 at 3:19 PM, Dominik Safaric
>  wrote:
>> The cluster consists of 4 workers and a master node.
> 
> Are you starting the cluster via bin/start-cluster.sh or are you using
> YARN etc.?

Re: flink/cancel & shutdown hooks

[Ufuk Celebi]

On Wed, Mar 8, 2017 at 3:19 PM, Dominik Safaric
 wrote:
> The cluster consists of 4 workers and a master node.

Are you starting the cluster via bin/start-cluster.sh or are you using
YARN etc.?

Re: flink/cancel & shutdown hooks

[Ufuk Celebi]

How are you deploying your job?

Shutdown hooks are executed when the JVM terminates whereas the cancel
command only cancels the Flink job and the JVM process potentially
keeps running. For example, running a standalone cluster would keep
the JVMs running.

On Wed, Mar 8, 2017 at 9:36 AM, Timo Walther  wrote:
> Hi Dominik,
>
> did you take a look into the logs? Maybe the exception is not shown in the
> CLI but in the logs.
>
> Timo
>
> Am 07/03/17 um 23:58 schrieb Dominik Safaric:
>
>> Hi all,
>>
>> I would appreciate for any help or advice in regard to default Java
>> runtime shutdown hooks and canceling Flink jobs.
>>
>> Namely part of my Flink application I am using a Kafka interceptor class
>> that defines a shutdown hook thread. When stopping the Flink streaming job
>> on my local machine the shutdown hook gets executed, however I do not see
>> the same behaviour when stopping the Flink application using bin/flink
>> cancel .
>>
>> Considering there are no exceptions thrown from the shutdown thread, what
>> could the root cause of this be?
>>
>> Thanks,
>> Dominik
>
>
>

Re: flink/cancel & shutdown hooks

[Dominik Safaric]

I’m deploying the job from the master node of the cluster itself using 
bin/flink run -c   . 

The cluster consists of 4 workers and a master node. 

Dominik

> On 8 Mar 2017, at 15:16, Ufuk Celebi  wrote:
> 
> How are you deploying your job?
> 
> Shutdown hooks are executed when the JVM terminates whereas the cancel
> command only cancels the Flink job and the JVM process potentially
> keeps running. For example, running a standalone cluster would keep
> the JVMs running.
> 
> On Wed, Mar 8, 2017 at 9:36 AM, Timo Walther  wrote:
>> Hi Dominik,
>> 
>> did you take a look into the logs? Maybe the exception is not shown in the
>> CLI but in the logs.
>> 
>> Timo
>> 
>> Am 07/03/17 um 23:58 schrieb Dominik Safaric:
>> 
>>> Hi all,
>>> 
>>> I would appreciate for any help or advice in regard to default Java
>>> runtime shutdown hooks and canceling Flink jobs.
>>> 
>>> Namely part of my Flink application I am using a Kafka interceptor class
>>> that defines a shutdown hook thread. When stopping the Flink streaming job
>>> on my local machine the shutdown hook gets executed, however I do not see
>>> the same behaviour when stopping the Flink application using bin/flink
>>> cancel .
>>> 
>>> Considering there are no exceptions thrown from the shutdown thread, what
>>> could the root cause of this be?
>>> 
>>> Thanks,
>>> Dominik
>> 
>> 
>> 

Re: flink/cancel & shutdown hooks

[Timo Walther]

Hi Dominik,

did you take a look into the logs? Maybe the exception is not shown in 
the CLI but in the logs.


Timo

Am 07/03/17 um 23:58 schrieb Dominik Safaric:

Hi all,

I would appreciate for any help or advice in regard to default Java runtime 
shutdown hooks and canceling Flink jobs.

Namely part of my Flink application I am using a Kafka interceptor class that defines 
a shutdown hook thread. When stopping the Flink streaming job on my local machine the 
shutdown hook gets executed, however I do not see the same behaviour when stopping 
the Flink application using bin/flink cancel .

Considering there are no exceptions thrown from the shutdown thread, what could 
the root cause of this be?

Thanks,
Dominik

flink/cancel & shutdown hooks

[Dominik Safaric]

Hi all,

I would appreciate for any help or advice in regard to default Java runtime 
shutdown hooks and canceling Flink jobs.

Namely part of my Flink application I am using a Kafka interceptor class that 
defines a shutdown hook thread. When stopping the Flink streaming job on my 
local machine the shutdown hook gets executed, however I do not see the same 
behaviour when stopping the Flink application using bin/flink cancel . 

Considering there are no exceptions thrown from the shutdown thread, what could 
the root cause of this be?   

Thanks,
Dominik