Github user sachingoel0101 commented on the pull request:
https://github.com/apache/flink/pull/1214#issuecomment-145824055
Yes. I agree. It started out as a simple issue to support detached jobs but
gotten a bit messy.
1. In detached mode, whether the program is run in interactive mode or
normal mode, if there is a call to an eager execution function, it will lead to
an exception. [As of now.]
2. This is where the problem started to appear more difficult. Since the
accumulator addition is entirely handled by Runtime Context, there can be no
check whether the job is in detached mode. Of course, having to store this
information in two places[`ExecutionEnvironment` and `RuntimeContext`] is poor
design.
If the job is being run in detached mode, the accumulators are lost and
user can never access them or use them.
But disallowing accumulators will effectively break most of existing
programs. So that's bad choice.
Eager execution functions use just accumulators actually, so, to maintain
harmony, we have to maintain them too.
If we just consider the first commit of this PR which adds support for
executing detached jobs, on a eager call, the main program throws an exception,
because the job execution result doesn't contain any accumulators.
```java
this.flatMap(new Utils.CollectHelper<T>(id, serializer)).name("collect()")
.output(new DiscardingOutputFormat<T>()).name("collect() sink");
JobExecutionResult res = getExecutionEnvironment().execute();
```
I'm really not sure how to handle accumulators in detached programs. What
we can do is:
In `JobExecutionResult`, whenever an accumulator is requested, we check for
the runtime to be -1, which indicates job was exited in detached mode. Display
a warning to the user on the Cli itself and indicate that the results of
accumulators are available on the web frontend [which will require writing them
to a file on the job manager].
One more thing: Since the detached mode is available on Yarn, it must throw
the exception too in case of `collect` or `print` calls. Is this the case? I
can't check on a yarn cluster myself.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
