[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944882#comment-14944882
]
ASF GitHub Bot commented on FLINK-2797:
---------------------------------------
Github user sachingoel0101 commented on the pull request:
https://github.com/apache/flink/pull/1214#issuecomment-145824055
Yes. I agree. It started out as a simple issue to support detached jobs but
gotten a bit messy.
1. In detached mode, whether the program is run in interactive mode or
normal mode, if there is a call to an eager execution function, it will lead to
an exception. [As of now.]
2. This is where the problem started to appear more difficult. Since the
accumulator addition is entirely handled by Runtime Context, there can be no
check whether the job is in detached mode. Of course, having to store this
information in two places[`ExecutionEnvironment` and `RuntimeContext`] is poor
design.
If the job is being run in detached mode, the accumulators are lost and
user can never access them or use them.
But disallowing accumulators will effectively break most of existing
programs. So that's bad choice.
Eager execution functions use just accumulators actually, so, to maintain
harmony, we have to maintain them too.
If we just consider the first commit of this PR which adds support for
executing detached jobs, on a eager call, the main program throws an exception,
because the job execution result doesn't contain any accumulators.
```java
this.flatMap(new Utils.CollectHelper<T>(id, serializer)).name("collect()")
.output(new DiscardingOutputFormat<T>()).name("collect() sink");
JobExecutionResult res = getExecutionEnvironment().execute();
```
I'm really not sure how to handle accumulators in detached programs. What
we can do is:
In `JobExecutionResult`, whenever an accumulator is requested, we check for
the runtime to be -1, which indicates job was exited in detached mode. Display
a warning to the user on the Cli itself and indicate that the results of
accumulators are available on the web frontend [which will require writing them
to a file on the job manager].
One more thing: Since the detached mode is available on Yarn, it must throw
the exception too in case of `collect` or `print` calls. Is this the case? I
can't check on a yarn cluster myself.
> CLI: Missing option to submit jobs in detached mode
> ---------------------------------------------------
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
> Affects Versions: 0.9, 0.10
> Reporter: Maximilian Michels
> Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
