aljoscha commented on issue #10474: [FLINK-15116] Make JobClient stateless, remove AutoCloseable URL: https://github.com/apache/flink/pull/10474#issuecomment-562862939 The current state in master is this: `ClusterClient` is an internal interface that is somewhat closely tied to the Flink runtime classes, we therefore cannot expose this as a user-facing interface. `JobClient` is a new interface we introduced that can be used to control one job. This allows, for example, to write things like this (from the `SocketWindowWordCount` example in this PR: ``` CompletableFuture<JobClient> jobClientFuture = env.executeAsync("Socket Window WordCount"); JobClient client = jobClientFuture.get(); long startTime = System.currentTimeMillis(); while (running) { Thread.sleep(1000); client .getJobStatus() .whenComplete((jobStatus, throwable) -> { System.out.println("JOB STATUS: " + jobStatus); if (jobStatus == null || !jobStatus.equals(JobStatus.RUNNING)) { running = false; } }); if (System.currentTimeMillis() - startTime > 10_000) { client.cancel(); } } ``` The `executeAsync()` returns the `JobClient` so that users immediately have a handle that they can work with, if `executeAsync()` returned void there would have to be some mechanism that allows to check what jobs there are and then they could create some client for "talking" to the job. If we also add the `JobListener` interface that allows frameworks to listen to newly created job clients and store them internally. If we don't require lifecycle management of the `JobClient` (as this PR introduces), this makes it very easy for the framework to interact with a job. I like this approach because there is no more potential for resource leaks. Before, the `JobClient` resources would be leaked if the user didn't call `close()` on the `JobClient`. Releasing the resource in a shutdown hook would not work, in my opinion, because there can be more long-running environments. For example shell/notebook use cases, where users submit multiple jobs, which potentially creates multiple per-job (YARN) clusters that each would have their own associated `ClusterClient` with their resources. If we only released those when the JVM shuts down resources/clients would pile up. @zjffdu Does this work for the the Zeppelin use cases you have in mind? Do you think we also need a user-facing `FlinkClient` in addition the `JobClient` or as a replacement. What would the interaction in a user program look like, i.e. how is it used?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
