[GitHub] flink issue #2928: [FLINK-5108] Remove ClientShutdownHook during job executi...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2928
  
Thanks for your understanding :) Could you please close this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2928: [FLINK-5108] Remove ClientShutdownHook during job executi...

[Renkai]

Github user Renkai commented on the issue:

https://github.com/apache/flink/pull/2928
  
That is ok to close this issue.

Max 于2016年12月14日周三 02:31写道：

> There is one problem we overlooked. In detached mode we ensure cluster
> shutdown through a message sent by the client during job submission to 
tell
> the JobManager that this is going to be the last job it has to execute. In
> interactive execution mode, the user jar can contain multiple jobs; this 
is
> mostly useful for interactive batch jobs. Since we just execute the main
> method of the user jar, we don't know how many jobs are submitted and when
> to shutdown the cluster. That's why we chose to delegate the shutdown to
> the client for interactive jobs. Thus, I'm hesitant to remove the shutdown
> hook because it ensures that the cluster shuts down during interactive job
> executions. It prevents clusters from lingering around when the client
> shuts down.
>
> A couple of solution for this problem:
>
>1.
>
>The JobManager watches the client and shuts down a) if it looses
>connection to the client and the job it executes has completed or b) 
the
>client tells the JobManager to shut down.
>2.
>
>The JobManager drives the execution which is now part of the client
>3.
>
>We don't allow multiple jobs to execute. Then we always have a clear
>shutdown point. This is perhaps the easiest and most elegant solution. 
Most
>users only execute a single job at a time anyways. We can still allow
>interactive job executions if the user chooses to. Perhaps we can make 
this
>more explicit in the API to give a hint to the client.
>
> I'm afraid we will have to close this PR until we realize one of the above
> solutions (or another one).
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2928: [FLINK-5108] Remove ClientShutdownHook during job executi...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2928
  
There is one problem we overlooked. In detached mode we ensure cluster 
shutdown through a message sent by the client during job submission to tell the 
JobManager that this is going to be the last job it has to execute. In 
interactive execution mode, the user jar can contain multiple jobs; this is 
mostly useful for interactive batch jobs. Since we just execute the main method 
of the user jar, we don't know how many jobs are submitted and when to shutdown 
the cluster. That's why we chose to delegate the shutdown to the client for 
interactive jobs. Thus, I'm hesitant to remove the shutdown hook because it 
ensures that the cluster shuts down during interactive job executions. It 
prevents clusters from lingering around when the client shuts down.

A couple of solution for this problem:

1. The JobManager watches the client and shuts down a) if it looses 
connection to the client and the job it executes has completed or b) the client 
tells the JobManager to shut down.

2. The JobManager drives the execution which is now part of the client

3. We don't allow multiple jobs to execute. Then we always have a clear 
shutdown point. This is perhaps the easiest and most elegant solution. Most 
users only execute a single job at a time anyways. We can still allow 
interactive job executions if the user chooses to. Perhaps we can make this 
more explicit in the API to give a hint to the client. 

I'm afraid we will have to close this PR until we realize one of the above 
solutions (or another one).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2928: [FLINK-5108] Remove ClientShutdownHook during job executi...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2928
  
+1 for that approach


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2928: [FLINK-5108] Remove ClientShutdownHook during job executi...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2928
  
I will go ahead and merge this PR since there have been no further comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2928: [FLINK-5108] Remove ClientShutdownHook during job executi...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2928
  
I think the original intend of the Client shutdown hook was to make sure 
that clusters which are spawned and didn't receive jobs are cleaned up again. 
However, that behavior can be quite tricky because a job might actually have 
been submitted externally. So +1 for removing it altogether.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2928: [FLINK-5108] Remove ClientShutdownHook during job executi...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2928
  
What speaks against not having a shutdown hook at all?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2928: [FLINK-5108] Remove ClientShutdownHook during job executi...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2928
  
Thanks for the PR! This removes the shutdown hook after the cluster has 
been deployed. The original intend was to remove it only after job submission. 
Probably it makes sense to remove it all-together. I have to think about it.

Perhaps @rmetzger could comment on this as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---