Hi, Is there any way we can run a callback on job completion or failure without leaving the client running during job execution? For example, when we submit the job via the web UI the main() method's call to ExecutionEnvironment#execute() appears to by asynchronous with the job execution. Therefore, the execute() call returns before the job is completed. This is a bit confusing because the behavior is different when run from the IDE vs. run in a cluster, and because signature of the returned class JobExecutionResult implies that it can tell you how long execution took (it has getNetRuntime()). We would like to be able to detect job completion or failure so that we can monitor the success or failure of batch jobs, in particular, so that we can react to failures appropriately. It seems like the JobManager should be capable of executing callbacks like this. Otherwise, we'll have to create an external component that eg. polls the web UI/API for job status.
Does the web UI run in the same JVM as the JobManager (when deployed in YARN)? If so, I would expect logs from the main method to appear in the JobManager logs. However, for some reason I can't find log messages or System.out messages when they are logged in the main() method after execute() is called. Why is that? Edit: figured it out: OptimizerPlanEnvironment#execute() ends with "throw new ProgramAbortException()". Tricky and unexpected. That should definitely be mentioned in the javadocs of the execute() method! Even the documentation says, "The execute() method is returning a JobExecutionResult, this contains execution times and accumulator results." which isn't true, or at least isn't always true. Thanks, Shannon
