Re: [Discuss] Add JobListener (hook) in flink job lifecycle

This issue is another case where we have problems figuring out the boundaries and responsibilities between the ExecutionEnvironments and the ClusterClient. I believe we should figure this out first, and decide whether the ClusterClient (or anything based on it) should be made public to

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

Hi everybody, any news on this? For us would be VERY helpful to have such a feature because we need to execute a call to a REST service once a job ends. Right now we do this after the env.execute() but this works only if the job is submitted via the CLI client, the REST client doesn't execute

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

Hi Beckett， Thanks for your feedback, See my comments inline >>> How do user specify the listener? * What I proposal is to register JobListener in ExecutionEnvironment. I don't think we should make ClusterClient as public api. >>> Where should the listener run? * I don't think it is proper to

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

Thanks for the proposal, Jeff. Adding a listener to allow users handle events during the job lifecycle makes a lot of sense to me. Here are my two cents. * How do user specify the listener? * It is not quite clear to me whether we consider ClusterClient as a public interface? From what I

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

Hi Till, IMHO, allow adding hooks involves 2 steps. 1. Provide hook interface, and call these hook in flink (ClusterClient) at the right place. This should be done by framework (flink) 2. Implement new hook implementation and add/register them into framework(flink) What I am doing is step 1

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

I think we should not expose the ClusterClient configuration via the ExecutionEnvironment (env.getClusterClient().addJobListener) because this is effectively the same as exposing the JobListener interface directly on the ExecutionEnvironment. Instead I think it could be possible to provide a

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

>>> The ExecutionEnvironment is usually used by the user who writes the code and this person (I assume) would not be really interested in these callbacks. Usually ExecutionEnvironment is used by the user who write the code, but it doesn't needs to be created and configured by this person. e.g.

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

Thanks for starting this discussion Jeff. I can see the need for additional hooks for third party integrations. The thing I'm wondering is whether we really need/want to expose a JobListener via the ExecutionEnvironment. The ExecutionEnvironment is usually used by the user who writes the code and

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

Hi Jeff, I personally like this proposal. From the perspective of programmability, the JobListener can make the third program more appreciable. The scene where I need the listener is the Flink cube engine for Apache Kylin. In the case, the Flink job program is embedded into the Kylin's

[Discuss] Add JobListener (hook) in flink job lifecycle

Hi All, I created FLINK-12214 for adding JobListener (hook) in flink job lifecycle. Since this is a new public api for flink, so I'd like to discuss it more widely in community to get more feedback. The background and motivation is that I am