[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004159#comment-15004159 ] ASF GitHub Bot commented on FLINK-2797: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1214 > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001938#comment-15001938
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r44641729
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/DetachedEnvironment.java
---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.InvalidProgramException;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.JobSubmissionResult;
+import org.apache.flink.api.common.Plan;
+import org.apache.flink.optimizer.plan.FlinkPlan;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URL;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Execution Environment for remote execution with the Client in detached
mode.
+ */
+public class DetachedEnvironment extends ContextEnvironment {
+
+ /** Keeps track of the program plan for the Client to access. */
+ private FlinkPlan detachedPlan;
+
+ private static final Logger LOG =
LoggerFactory.getLogger(DetachedEnvironment.class);
+
+ public DetachedEnvironment(Client remoteConnection, List jarFiles,
List classpaths, ClassLoader userCodeClassLoader) {
+ super(remoteConnection, jarFiles, classpaths,
userCodeClassLoader);
+ }
+
+ @Override
+ public JobExecutionResult execute(String jobName) throws Exception {
+ Plan p = createProgramPlan(jobName);
+ setDetachedPlan(Client.getOptimizedPlan(client.compiler, p,
getParallelism()));
+ LOG.warn("Job was executed in detached mode, the results will
be available on completion.");
+ this.lastJobExecutionResult =
DetachedJobExecutionResult.INSTANCE;
+ return this.lastJobExecutionResult;
+ }
+
+ public void setDetachedPlan(FlinkPlan plan) {
+ if (detachedPlan == null) {
+ detachedPlan = plan;
+ } else {
+ throw new
InvalidProgramException(DetachedJobExecutionResult.DETACHED_MESSAGE +
+
DetachedJobExecutionResult.EXECUTE_TWICE_MESSAGE);
+ }
+ }
+
+ /**
+* Finishes this Context Environment's execution by explicitly running
the plan constructed.
+*/
+ public JobSubmissionResult finalizeExecute() throws
ProgramInvocationException {
--- End diff --
Wouldn't it make more sense to overwrite `execute()` here?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002017#comment-15002017 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-156094351 @mxm , should be ready to merge now. :) > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001969#comment-15001969
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r44645161
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/DetachedEnvironment.java
---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.InvalidProgramException;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.JobSubmissionResult;
+import org.apache.flink.api.common.Plan;
+import org.apache.flink.optimizer.plan.FlinkPlan;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URL;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Execution Environment for remote execution with the Client in detached
mode.
+ */
+public class DetachedEnvironment extends ContextEnvironment {
+
+ /** Keeps track of the program plan for the Client to access. */
+ private FlinkPlan detachedPlan;
+
+ private static final Logger LOG =
LoggerFactory.getLogger(DetachedEnvironment.class);
+
+ public DetachedEnvironment(Client remoteConnection, List jarFiles,
List classpaths, ClassLoader userCodeClassLoader) {
+ super(remoteConnection, jarFiles, classpaths,
userCodeClassLoader);
+ }
+
+ @Override
+ public JobExecutionResult execute(String jobName) throws Exception {
+ Plan p = createProgramPlan(jobName);
+ setDetachedPlan(Client.getOptimizedPlan(client.compiler, p,
getParallelism()));
+ LOG.warn("Job was executed in detached mode, the results will
be available on completion.");
+ this.lastJobExecutionResult =
DetachedJobExecutionResult.INSTANCE;
+ return this.lastJobExecutionResult;
+ }
+
+ public void setDetachedPlan(FlinkPlan plan) {
+ if (detachedPlan == null) {
+ detachedPlan = plan;
+ } else {
+ throw new
InvalidProgramException(DetachedJobExecutionResult.DETACHED_MESSAGE +
+
DetachedJobExecutionResult.EXECUTE_TWICE_MESSAGE);
+ }
+ }
+
+ /**
+* Finishes this Context Environment's execution by explicitly running
the plan constructed.
+*/
+ public JobSubmissionResult finalizeExecute() throws
ProgramInvocationException {
--- End diff --
Ah you're overwriting`execute()` already. Sorry. Yes, please make the
finalize method package private.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001973#comment-15001973
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r44645434
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/DetachedEnvironment.java
---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.InvalidProgramException;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.JobSubmissionResult;
+import org.apache.flink.api.common.Plan;
+import org.apache.flink.optimizer.plan.FlinkPlan;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URL;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Execution Environment for remote execution with the Client in detached
mode.
+ */
+public class DetachedEnvironment extends ContextEnvironment {
+
+ /** Keeps track of the program plan for the Client to access. */
+ private FlinkPlan detachedPlan;
+
+ private static final Logger LOG =
LoggerFactory.getLogger(DetachedEnvironment.class);
+
+ public DetachedEnvironment(Client remoteConnection, List jarFiles,
List classpaths, ClassLoader userCodeClassLoader) {
+ super(remoteConnection, jarFiles, classpaths,
userCodeClassLoader);
+ }
+
+ @Override
+ public JobExecutionResult execute(String jobName) throws Exception {
+ Plan p = createProgramPlan(jobName);
+ setDetachedPlan(Client.getOptimizedPlan(client.compiler, p,
getParallelism()));
+ LOG.warn("Job was executed in detached mode, the results will
be available on completion.");
+ this.lastJobExecutionResult =
DetachedJobExecutionResult.INSTANCE;
+ return this.lastJobExecutionResult;
+ }
+
+ public void setDetachedPlan(FlinkPlan plan) {
+ if (detachedPlan == null) {
+ detachedPlan = plan;
+ } else {
+ throw new
InvalidProgramException(DetachedJobExecutionResult.DETACHED_MESSAGE +
+
DetachedJobExecutionResult.EXECUTE_TWICE_MESSAGE);
+ }
+ }
+
+ /**
+* Finishes this Context Environment's execution by explicitly running
the plan constructed.
+*/
+ public JobSubmissionResult finalizeExecute() throws
ProgramInvocationException {
--- End diff --
Already done. :)
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001955#comment-15001955
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r44643881
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/DetachedEnvironment.java
---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.client.program;
+
+import org.apache.flink.api.common.InvalidProgramException;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.JobSubmissionResult;
+import org.apache.flink.api.common.Plan;
+import org.apache.flink.optimizer.plan.FlinkPlan;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URL;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Execution Environment for remote execution with the Client in detached
mode.
+ */
+public class DetachedEnvironment extends ContextEnvironment {
+
+ /** Keeps track of the program plan for the Client to access. */
+ private FlinkPlan detachedPlan;
+
+ private static final Logger LOG =
LoggerFactory.getLogger(DetachedEnvironment.class);
+
+ public DetachedEnvironment(Client remoteConnection, List jarFiles,
List classpaths, ClassLoader userCodeClassLoader) {
+ super(remoteConnection, jarFiles, classpaths,
userCodeClassLoader);
+ }
+
+ @Override
+ public JobExecutionResult execute(String jobName) throws Exception {
+ Plan p = createProgramPlan(jobName);
+ setDetachedPlan(Client.getOptimizedPlan(client.compiler, p,
getParallelism()));
+ LOG.warn("Job was executed in detached mode, the results will
be available on completion.");
+ this.lastJobExecutionResult =
DetachedJobExecutionResult.INSTANCE;
+ return this.lastJobExecutionResult;
+ }
+
+ public void setDetachedPlan(FlinkPlan plan) {
+ if (detachedPlan == null) {
+ detachedPlan = plan;
+ } else {
+ throw new
InvalidProgramException(DetachedJobExecutionResult.DETACHED_MESSAGE +
+
DetachedJobExecutionResult.EXECUTE_TWICE_MESSAGE);
+ }
+ }
+
+ /**
+* Finishes this Context Environment's execution by explicitly running
the plan constructed.
+*/
+ public JobSubmissionResult finalizeExecute() throws
ProgramInvocationException {
--- End diff --
I'm not sure I understand. `execute()` should default to `execute(null)`,
no?
We don't really want to allow the user to trigger execution explicitly,
which also makes me realize this should be a package private method.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000161#comment-15000161
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on the pull request:
https://github.com/apache/flink/pull/1214#issuecomment-155717073
@mxm , fortunately, this time I was able to run the tests on my machine. :)
I'm seeing a strange error. This is the error in Task manager log file for
per job detached streaming test:
```java
java.io.IOException: File or directory already exists. Existing files and
directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to
overwrite existing files and directories.
at
org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:549)
at
org.apache.flink.api.common.io.FileOutputFormat.open(FileOutputFormat.java:226)
at
org.apache.flink.api.java.io.TextOutputFormat.open(TextOutputFormat.java:77)
at
org.apache.flink.streaming.api.functions.sink.FileSinkFunction.open(FileSinkFunction.java:62)
at
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:89)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:286)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:213)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
at java.lang.Thread.run(Thread.java:745)
```
This is surprising because I have made no changes to the way file writes or
overwrites are handled.
Here's the thing though: it should fail, because the folder is created at
line 475 in `YarnSessionFIFOITCase` and the program will need to overwrite it.
```java
File tmpOutFolder = null;
try{
tmpOutFolder = tmp.newFolder();
}
catch(IOException e) {
throw new RuntimeException(e);
}
```
But the tests always pass on the master branch.
The rest of the tests which follow this automatically fail because they
again check all the logs files for exception strings.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000310#comment-15000310 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155764884 Okay. Figured it out. The Parallelism in the `ContextEnvironment` wasn't being respected in `StreamContextEnvironment`. Passes on my local machine now; waiting for travis to give the green light now. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000652#comment-15000652 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155842188 There is a merge conflict. Looks good otherwise. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000790#comment-15000790 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155861149 @mxm , rebased. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998392#comment-14998392 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155385085 >>> You're probably right. It may just seem odd to the user that code gets executed even though the job won't be submitted. We might fix that in the future if users don't like it. I think most people will be aware that Flink executes lazily, and just because the program goes right till the end printing whatever it does, it doesn't guarantee a true submission. But there is no point debating on this. We can adapt to whatever people prefer. This'll likely go through a major change before 1.0 to support truly detached mode anyways. >>>Looks like your recent changes broke the YARN tests? Yes. It's also kind of hard to debug them. I can't seem to run them on my VM, and travis build takes a full one and a half hour to produce anything useful, and even then, I can't access the complete logs. It probably requires AWS keys or something. I'm not sure. Do you know of any change I can make to travis conf so all modules are built but only Yarn tests get executed? I have one more concern: There are four types of Yarn runs we can perform: 1. Yarn session in detached mode: Equivalent to `bin/start-cluster.sh` 2. Yarn session: Doesn't have a counterpart in standalone mode 3. Per job yarn in detached mode: No counterpart in standalone mode, even after we add this PR. 4. Per job yarn: No counterpart in standalone mode. Now, when `bin/flink -d` is run, user will expect there to be a single semantics; that a cluster will be brought up, job will be submitted and the cluster will be terminated. However, there doesn't seem to be any way of achieving this in standalone mode. Will this create any confusion? > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998376#comment-14998376 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155381011 Looks like your recent changes broke the YARN tests? > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998327#comment-14998327 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155373495 >This is a full fledged execution of the use code, and should show any user generated out and err messages in my opinion. You're probably right. It may just seem odd to the user that code gets executed even though the job won't be submitted. We might fix that in the future if users don't like it. >Of course, anything Flink will print, say, log messages, will already be disabled as there is no actor to receive the messages. There are still jobmanager/taskmanager log files. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998413#comment-14998413 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155391848 >Do you know of any change I can make to travis conf so all modules are built but only Yarn tests get executed? Yes change the build instructions in .travis.yml to run `mvn clean install -DskipTests` followed by a `cd flink-yarn-tests && mvn verify`. Locally, you can find the logs in the target directory of flink-yarn-tests. There is also a way to upload these log files with an AWS account. I think the different semantics for YARN and standalone are fine. The `-d` flag always starts the job detached. That's independent of YARN or standalone mode. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996451#comment-14996451 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155052238 When the main method is invoked via `PackagedProgram#invokeInteractiveModeForExecution`, it executes everything, and the flink plan it builds is stored in the `DetachedEnvironment`[now]. I just ran an example with the new code, and it prints something like this on the cli:  Is this what you mean? If not, can you check again? > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996512#comment-14996512 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155059238 Yes, the idea is suppress output during the dry-run and plan creation. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996990#comment-14996990 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155137822 I'm not sure why we should do this in Detached mode. When `info` is run, user has only directly explicitly requested the plan. This is a full fledged execution of the use code, and should show any user generated out and err messages in my opinion. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996995#comment-14996995 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155138232 Of course, anything Flink will print, say, log messages, will already be disabled as there is no actor to receive the messages. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996407#comment-14996407 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155043211 Hi @mxm , I'm not sure about directing sysout and syserr to null. Detached mode just means flink jobs are being run in detached mode; the rest of the program should go whatever way the user requested. As for refactoring into a separate environment, I agree. I've already done that. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996419#comment-14996419 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-155047586 Thanks for the timely update @sachingoel0101. I was referring to the acquisition of the plan. When we create the plan, (e.g. ./bin/flink info JAR) we suppress printing to standard out. I thought we should do similar when we create the detached plan which we submit only after we have determined whether we can run the jar in detached mode. I ran some of the examples in detached mode and found it odd that it prints to stdout during the creation of the plan. The actual execution of the plan should of course be able to print to stdout. What do you think? > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993322#comment-14993322 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-154339878 Failure on an unrelated test: JobManagerSubmittedJobGraphsRecoveryITCase > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993414#comment-14993414
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r44117601
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java
---
@@ -114,15 +115,33 @@ public Client getClient() {
public List getClasspaths(){
return classpathsToAttach;
}
-
+
+ public ClassLoader getUserCodeClassLoader() {
+ return userCodeClassLoader;
+ }
+
+ public void setCurrentPlan(FlinkPlan plan) {
--- End diff --
Isn't `setDetachedPlan` a better name? It's only used for the detached mode
and it's either set or null.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993415#comment-14993415
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r44117608
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java
---
@@ -114,15 +115,33 @@ public Client getClient() {
public List getClasspaths(){
return classpathsToAttach;
}
-
+
+ public ClassLoader getUserCodeClassLoader() {
+ return userCodeClassLoader;
+ }
+
+ public void setCurrentPlan(FlinkPlan plan) {
+ if (currentPlan == null) {
+ currentPlan = plan;
+ } else {
+ throw new DetachedProgramException(
+
DetachedJobExecutionResult.DETACHED_MESSAGE +
DetachedJobExecutionResult.EXECUTE_TWICE_MESSAGE
+ );
+ }
+ }
+
+ public FlinkPlan getCurrentPlan() {
--- End diff --
Same as above here.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993517#comment-14993517
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r44126978
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java
---
@@ -114,15 +115,33 @@ public Client getClient() {
public List getClasspaths(){
return classpathsToAttach;
}
-
+
+ public ClassLoader getUserCodeClassLoader() {
+ return userCodeClassLoader;
+ }
+
+ public void setCurrentPlan(FlinkPlan plan) {
--- End diff --
Fair enough. Changed.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993764#comment-14993764 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-154428753 There are still two issues I noticed. Printing to standard out should be disabled during the dry-run phase. Just like in the OptimizerPlanEnvironment. The other one is ContextEnvironment. Do you think you could refactor your changes in ContextEnvironment into a separate DetachedEnvironment? We don't really need the context for the dry run. Thus, we have a better separation of concerns with an extra class for the detached execution. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992051#comment-14992051
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r44040745
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java
---
@@ -81,13 +60,15 @@ public JobExecutionResult execute(String jobName)
throws Exception {
transformations.clear();
- // execute the programs
- if (wait) {
- return client.runBlocking(streamGraph, jars,
classpaths, userCodeClassLoader);
+ if (ctx.isWait()) {
+ return ctx.getClient().runBlocking(streamGraph,
ctx.getJars(), ctx.getClasspaths(),
+ ctx.getUserCodeClassLoader());
} else {
- JobSubmissionResult result =
client.runDetached(streamGraph, jars, classpaths, userCodeClassLoader);
+ ctx.setCurrentPlan(streamGraph);
--- End diff --
Here, the last call to execute() would override all the previous ones. I
think you should throw an Exception here like in ContextEnvironment. Or you
throw the Exception on setCurrentPlan. We want to allow only one job to be
submitted detached, right?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992231#comment-14992231
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r44051406
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java
---
@@ -81,13 +60,15 @@ public JobExecutionResult execute(String jobName)
throws Exception {
transformations.clear();
- // execute the programs
- if (wait) {
- return client.runBlocking(streamGraph, jars,
classpaths, userCodeClassLoader);
+ if (ctx.isWait()) {
+ return ctx.getClient().runBlocking(streamGraph,
ctx.getJars(), ctx.getClasspaths(),
+ ctx.getUserCodeClassLoader());
} else {
- JobSubmissionResult result =
client.runDetached(streamGraph, jars, classpaths, userCodeClassLoader);
+ ctx.setCurrentPlan(streamGraph);
--- End diff --
@mxm , fixed.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989372#comment-14989372 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-153690460 That's a good point. It might inadvertently break things. For the release candidate, how about including just the first part, i.e., just the detached mode for standalone clusters. That would not break any functionality, and bring the yarn and standalone mode at par as far as detached mode is concerned. It will definitely stay broken for both in case of eager jobs. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989364#comment-14989364 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-153689197 I'd like to merge this on the master but I don't think it should be included in 0.10 because it is too big of a change for a release candidate. Additionally, it's not a pressing issue. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989006#comment-14989006 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-153599182 Build passes. Let's include this in the next rc. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987402#comment-14987402 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-153377995 Looks good @sachingoel0101. Good to merge from my side. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987418#comment-14987418 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-153381184 Cool. :) Squashing commits. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987356#comment-14987356
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43753100
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java
---
@@ -81,13 +60,13 @@ public JobExecutionResult execute(String jobName)
throws Exception {
transformations.clear();
- // execute the programs
- if (wait) {
- return client.runBlocking(streamGraph, jars,
classpaths, userCodeClassLoader);
+ if (ctx.isWait()) {
+ return ctx.getClient().runBlocking(streamGraph,
ctx.getJars(), ctx.getClasspaths(),
+ ctx.getUserCodeClassLoader());
} else {
- JobSubmissionResult result =
client.runDetached(streamGraph, jars, classpaths, userCodeClassLoader);
+ ctx.setCurrentPlan(streamGraph);
LOG.warn("Job was executed in detached mode, the
results will be available on completion.");
- return
JobExecutionResult.fromJobSubmissionResult(result);
+ return
ContextEnvironment.DetachedJobExecutionResult.INSTANCE;
--- End diff --
Done.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987321#comment-14987321
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43750488
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java
---
@@ -81,13 +60,13 @@ public JobExecutionResult execute(String jobName)
throws Exception {
transformations.clear();
- // execute the programs
- if (wait) {
- return client.runBlocking(streamGraph, jars,
classpaths, userCodeClassLoader);
+ if (ctx.isWait()) {
+ return ctx.getClient().runBlocking(streamGraph,
ctx.getJars(), ctx.getClasspaths(),
+ ctx.getUserCodeClassLoader());
} else {
- JobSubmissionResult result =
client.runDetached(streamGraph, jars, classpaths, userCodeClassLoader);
+ ctx.setCurrentPlan(streamGraph);
LOG.warn("Job was executed in detached mode, the
results will be available on completion.");
- return
JobExecutionResult.fromJobSubmissionResult(result);
+ return
ContextEnvironment.DetachedJobExecutionResult.INSTANCE;
--- End diff --
Could you add a comment here which explains why you delay the execution?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985330#comment-14985330
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43635433
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java
---
@@ -165,12 +181,65 @@ public ContextEnvironmentFactory(Client client,
List jarFilesToAttach,
@Override
public ExecutionEnvironment createExecutionEnvironment() {
- ContextEnvironment env = new ContextEnvironment(client,
jarFilesToAttach, classpathsToAttach,
- userCodeClassLoader, wait);
- if (defaultParallelism > 0) {
- env.setParallelism(defaultParallelism);
+ if (wait || lastEnv == null) {
--- End diff --
Could you reformulate the if-else? That way the code would be more explicit:
```java
// We only allow one detached environment in detached mode
if (!wait && lastEnv != null) {
throw new DetachedProgramException(..);
}
// regular submission flow
lastEnv = ...
```
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984951#comment-14984951
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43609728
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java
---
@@ -75,29 +62,31 @@ public JobExecutionResult execute() throws Exception {
@Override
public JobExecutionResult execute(String jobName) throws Exception {
- JobGraph jobGraph;
- if (jobName == null) {
- jobGraph = this.getStreamGraph().getJobGraph();
- } else {
- jobGraph = this.getStreamGraph().getJobGraph(jobName);
- }
-
- transformations.clear();
-
- // attach all necessary jar files to the JobGraph
- for (URL file : jars) {
--- End diff --
I pushed a fix for this issue: bf29de981c2bcd5cb5d33c68b158c95c8820f43d
It think you have already adapted to this change. Nice :)
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984987#comment-14984987 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-152973475 Thanks for adapting the pull request to the latest master. It is in better shape now but I still think it can be improved. Please make sure there is an explicit way to submit a job in both detached and blocking execution mode. Some classes mix up the two modes (e.g. `StreamContextEnvironment`). > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984949#comment-14984949
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43609625
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java
---
@@ -36,28 +31,16 @@
private static final Logger LOG =
LoggerFactory.getLogger(StreamContextEnvironment.class);
- private final List jars;
-
- private final List classpaths;
-
- private final Client client;
+ private final ContextEnvironment ctx;
private final ClassLoader userCodeClassLoader;
-
- private final boolean wait;
- protected StreamContextEnvironment(Client client, List jars,
List classpaths, int parallelism,
- boolean wait) {
- this.client = client;
- this.jars = jars;
- this.classpaths = classpaths;
- this.wait = wait;
-
- this.userCodeClassLoader =
JobWithJars.buildUserCodeClassLoader(jars, classpaths,
- getClass().getClassLoader());
+ protected StreamContextEnvironment(ContextEnvironment ctx) {
+ this.ctx = ctx;
+ this.userCodeClassLoader =
JobWithJars.buildUserCodeClassLoader(ctx.getJars(), ctx.getClasspaths(),
getClass().getClassLoader());
--- End diff --
Classloaders rules are the same for streaming and batch. We should try to
reuse code when possible. So probably a good idea to delgate this to the Client
class.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984964#comment-14984964
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43610296
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java
---
@@ -36,28 +31,16 @@
private static final Logger LOG =
LoggerFactory.getLogger(StreamContextEnvironment.class);
- private final List jars;
-
- private final List classpaths;
-
- private final Client client;
+ private final ContextEnvironment ctx;
private final ClassLoader userCodeClassLoader;
-
- private final boolean wait;
- protected StreamContextEnvironment(Client client, List jars,
List classpaths, int parallelism,
- boolean wait) {
- this.client = client;
- this.jars = jars;
- this.classpaths = classpaths;
- this.wait = wait;
-
- this.userCodeClassLoader =
JobWithJars.buildUserCodeClassLoader(jars, classpaths,
- getClass().getClassLoader());
+ protected StreamContextEnvironment(ContextEnvironment ctx) {
+ this.ctx = ctx;
+ this.userCodeClassLoader =
JobWithJars.buildUserCodeClassLoader(ctx.getJars(), ctx.getClasspaths(),
getClass().getClassLoader());
--- End diff --
Okay. I will go through all usages of classloaders and try to delegate them
to the `Client`.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984972#comment-14984972
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43610508
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java
---
@@ -165,12 +181,65 @@ public ContextEnvironmentFactory(Client client,
List jarFilesToAttach,
@Override
public ExecutionEnvironment createExecutionEnvironment() {
- ContextEnvironment env = new ContextEnvironment(client,
jarFilesToAttach, classpathsToAttach,
- userCodeClassLoader, wait);
- if (defaultParallelism > 0) {
- env.setParallelism(defaultParallelism);
+ if (wait || lastEnv == null) {
--- End diff --
Could you at least add a comment here? This condition is not very
intuitive. Why do you even set `lastEnv` here for blocking execution mode?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985044#comment-14985044 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-152987728 @mxm , there is a lot of code duplication between the detached and blocking submission modes. I'm gonna try to clean that up while I address your concerns. Does it make sense to have a single run method which is called as `run(..., mode)` where mode is an enum? > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985057#comment-14985057 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-152989560 It's fine to have separate code for detached and blocking submission where necessary. I was more concerned with this pull request that it introduces new code branches which distort the existing submission logic. Wherever possible we should try to be explicit about the submission code path we're in instead of mixing them all together. For instance, you're setting `lastEnv´ on blocking execution but then you never access it during blocking execution. This is not a clear code style. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985075#comment-14985075 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-152990620 The lastEnv is set for blocking also, because it still makes sense. It just stores what the last environment created by the factory was. I worked on this a while back while fixing a bug(?) in `TestEnvironment` which was meant to allow accessing last created environments. [https://github.com/apache/flink/commit/166b3705c9079efafe9217d98e7edaf54e6a84cf] Whether it is accessed or not is an entirely different matter IMO. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984807#comment-14984807 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-152934449 @mxm, since you already delegated job graph creation to the Client in one of your commits, I've removed that from my commits. Rebasing required squashing all commits first. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984810#comment-14984810
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43600897
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java
---
@@ -36,28 +31,16 @@
private static final Logger LOG =
LoggerFactory.getLogger(StreamContextEnvironment.class);
- private final List jars;
-
- private final List classpaths;
-
- private final Client client;
+ private final ContextEnvironment ctx;
private final ClassLoader userCodeClassLoader;
-
- private final boolean wait;
- protected StreamContextEnvironment(Client client, List jars,
List classpaths, int parallelism,
- boolean wait) {
- this.client = client;
- this.jars = jars;
- this.classpaths = classpaths;
- this.wait = wait;
-
- this.userCodeClassLoader =
JobWithJars.buildUserCodeClassLoader(jars, classpaths,
- getClass().getClassLoader());
+ protected StreamContextEnvironment(ContextEnvironment ctx) {
+ this.ctx = ctx;
+ this.userCodeClassLoader =
JobWithJars.buildUserCodeClassLoader(ctx.getJars(), ctx.getClasspaths(),
getClass().getClassLoader());
--- End diff --
I'm not very familiar with classloaders. Is it necessary that the
classloader for Streaming jobs be built here and not via the usual
`Program#getUserCodeClassLoader` in `Client` class?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982519#comment-14982519
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43498784
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java
---
@@ -75,29 +62,31 @@ public JobExecutionResult execute() throws Exception {
@Override
public JobExecutionResult execute(String jobName) throws Exception {
- JobGraph jobGraph;
- if (jobName == null) {
- jobGraph = this.getStreamGraph().getJobGraph();
- } else {
- jobGraph = this.getStreamGraph().getJobGraph(jobName);
- }
-
- transformations.clear();
-
- // attach all necessary jar files to the JobGraph
- for (URL file : jars) {
--- End diff --
Jars and classpaths also need to be attached to the JobGraph of the
non-detached streaming job, right? Wouldn't it be nice to always do that at the
Client? It's always good to avoid multiple branches for doing the same thing.
`Client.getJobGraph(FlinkPlan,..` enables you to pass the streamgraph directly
with the jars and classpaths.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978450#comment-14978450 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-151849439 @mxm , if there are any more comments, let me know. I'll try to resolve them quickly so we can include this in the next rc (which will likely be the last one(?)). A round of testing might be needed to make sure everything works fine. I'm not much aware of how streaming jobs work, so I might have missed something while testing myself. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978611#comment-14978611
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43273230
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
@@ -271,18 +272,21 @@ public JobSubmissionResult
runDetached(PackagedProgram prog, int parallelism)
}
else if (prog.isUsingInteractiveMode()) {
LOG.info("Starting program in interactive mode");
- ContextEnvironment.setAsContext(this,
prog.getAllLibraries(), prog.getClasspaths(),
- prog.getUserCodeClassLoader(), parallelism,
false);
-
+ ContextEnvironment.ContextEnvironmentFactory factory =
ContextEnvironment.setAsContext(this,
+ prog.getAllLibraries(),
prog.getClasspaths(), prog.getUserCodeClassLoader(), parallelism, false);
// invoke here
try {
prog.invokeInteractiveModeForExecution();
+ ContextEnvironment ctx =
factory.getLastEnvironment();
+ if (ctx == null) {
+ throw new InvalidProgramException("No
execution environment was created.");
--- End diff --
Ok let's leave it like that.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975886#comment-14975886 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-151402904 @mxm I have made some changes and now the *current* plan resides in the `ContextEnvironment`. This required some changes in `StreamContextEnvironment`, which now wraps the `ContextEnvironment` which was initially created to check the existence of context environment. Furthermore, `ContextEnvironmentFactory` needs to store the last context environment created, which can be used then to access the plan, so it can be executed after the call to interactive program's `main` finishes. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976080#comment-14976080
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43098424
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
@@ -271,18 +272,21 @@ public JobSubmissionResult
runDetached(PackagedProgram prog, int parallelism)
}
else if (prog.isUsingInteractiveMode()) {
LOG.info("Starting program in interactive mode");
- ContextEnvironment.setAsContext(this,
prog.getAllLibraries(), prog.getClasspaths(),
- prog.getUserCodeClassLoader(), parallelism,
false);
-
+ ContextEnvironment.ContextEnvironmentFactory factory =
ContextEnvironment.setAsContext(this,
+ prog.getAllLibraries(),
prog.getClasspaths(), prog.getUserCodeClassLoader(), parallelism, false);
// invoke here
try {
prog.invokeInteractiveModeForExecution();
+ ContextEnvironment ctx =
factory.getLastEnvironment();
+ if (ctx == null) {
+ throw new InvalidProgramException("No
execution environment was created.");
--- End diff --
It's not strictly necessary. This should return a `JobSubmissionResult`
with null job id then.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976079#comment-14976079
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43098367
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java
---
@@ -75,29 +62,31 @@ public JobExecutionResult execute() throws Exception {
@Override
public JobExecutionResult execute(String jobName) throws Exception {
- JobGraph jobGraph;
- if (jobName == null) {
- jobGraph = this.getStreamGraph().getJobGraph();
- } else {
- jobGraph = this.getStreamGraph().getJobGraph(jobName);
- }
-
- transformations.clear();
-
- // attach all necessary jar files to the JobGraph
- for (URL file : jars) {
--- End diff --
The detached job goes through `Client#getJobGraph(FlinkPlan, List,
List)` where the jars and classpaths are added to the job graph.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976087#comment-14976087 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-151430438 Thanks. Looks much better. Let's make sure we don't break any classloader or jar dependencies during job submission. This can cause annoyances for users and we have to fix it afterwards :) > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976071#comment-14976071
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43098099
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java
---
@@ -75,29 +62,31 @@ public JobExecutionResult execute() throws Exception {
@Override
public JobExecutionResult execute(String jobName) throws Exception {
- JobGraph jobGraph;
- if (jobName == null) {
- jobGraph = this.getStreamGraph().getJobGraph();
- } else {
- jobGraph = this.getStreamGraph().getJobGraph(jobName);
- }
-
- transformations.clear();
-
- // attach all necessary jar files to the JobGraph
- for (URL file : jars) {
--- End diff --
No Jars for detached jobs?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976070#comment-14976070
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43098098
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamContextEnvironment.java
---
@@ -75,29 +62,31 @@ public JobExecutionResult execute() throws Exception {
@Override
public JobExecutionResult execute(String jobName) throws Exception {
- JobGraph jobGraph;
- if (jobName == null) {
- jobGraph = this.getStreamGraph().getJobGraph();
- } else {
- jobGraph = this.getStreamGraph().getJobGraph(jobName);
- }
-
- transformations.clear();
-
- // attach all necessary jar files to the JobGraph
- for (URL file : jars) {
- jobGraph.addJar(new Path(file.toURI()));
- }
-
- jobGraph.setClasspaths(classpaths);
--- End diff --
No classpaths for detached jobs?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976104#comment-14976104 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-151435402 Agreed. I'll build and test with the example programs in standalone mode and on yarn. It should work perfectly though since `YarnSessionFIFOITCase` checks both batch and streaming jobs on detached clusters, and the standalone mode is tested in `ClientTest#testDetached`. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976077#comment-14976077
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43098234
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
@@ -271,18 +272,21 @@ public JobSubmissionResult
runDetached(PackagedProgram prog, int parallelism)
}
else if (prog.isUsingInteractiveMode()) {
LOG.info("Starting program in interactive mode");
- ContextEnvironment.setAsContext(this,
prog.getAllLibraries(), prog.getClasspaths(),
- prog.getUserCodeClassLoader(), parallelism,
false);
-
+ ContextEnvironment.ContextEnvironmentFactory factory =
ContextEnvironment.setAsContext(this,
+ prog.getAllLibraries(),
prog.getClasspaths(), prog.getUserCodeClassLoader(), parallelism, false);
// invoke here
try {
prog.invokeInteractiveModeForExecution();
+ ContextEnvironment ctx =
factory.getLastEnvironment();
+ if (ctx == null) {
+ throw new InvalidProgramException("No
execution environment was created.");
--- End diff --
Do we really want to fail here?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976078#comment-14976078
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43098268
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
@@ -909,7 +919,21 @@ private int handleError(Throwable t) {
}
LOG.error("Error while running the command.", t);
- t.printStackTrace();
+ // check if the error was due to an invalid program in detached
mode.
+ if (t instanceof ProgramInvocationException && t.getCause()
instanceof DetachedProgramException) {
+ System.err.println(t.getCause().getMessage());
+ // now trace to the user's main method. We don't wanna
show unnecessary information
+ // in this particular case.
--- End diff --
Why in this case? Doesn't that apply to all errors that occur during
interactive execution?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976084#comment-14976084
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43098720
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
@@ -909,7 +919,21 @@ private int handleError(Throwable t) {
}
LOG.error("Error while running the command.", t);
- t.printStackTrace();
+ // check if the error was due to an invalid program in detached
mode.
+ if (t instanceof ProgramInvocationException && t.getCause()
instanceof DetachedProgramException) {
+ System.err.println(t.getCause().getMessage());
+ // now trace to the user's main method. We don't wanna
show unnecessary information
+ // in this particular case.
--- End diff --
True. I didn't wanna change the existing stack traces.
This will require maintaining a global variable for whether the program was
run in interactive mode. The `DetachedProgramException` is a special case where
we don't wanna clutter the stack trace with flink's trace, instead just show
the user program's stack trace.
In other cases, this might not be the case.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976226#comment-14976226 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-151458018 @mxm , I have tested the build with the following commands: 1. Standalone cluster: `bin/flink run -d` for both streaming and batch wordcount examples. 2. Yarn: `bin/flink run -d -m yarn-cluster` and `bin/flink run -m yarn-cluster -yd` for both streaming and batch wordcount. Lemme know if there's any other checks to be made. Also, what should I do about https://github.com/apache/flink/pull/1214#discussion_r43098424? Let it be or return a null job id? > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974487#comment-14974487
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r43015201
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -304,4 +368,60 @@ public String getDescription() {
return "TestOptimizerPlan
";
}
}
+
+ public static final class TestExecuteTwice {
+
+ public static void main(String args[]) throws Exception {
--- End diff --
Okay. Cool. Lemme know if there's any other issues to address. :smile:
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974646#comment-14974646 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-151224527 I took a look at this again because I wanted to merge it for 0.10. However, I think it still needs a bit of work. The `ContextEnvironmentFactory` shouldn't hold the state for detached batch and stream executions. Could you store the batch plan in the `ContextEnvironment` and the stream graph in the `StreamContextEnvironment` (non-static)? Please give them appropriate names. Since the `ContextEnvironmentFactory` is currently the entry point for batch and streaming execution (the streaming environment just checks whether the `ContextEnvironment` is instantiated), you may set a flag in the factory to prevent multiple executions. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971233#comment-14971233
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42882261
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -304,4 +368,60 @@ public String getDescription() {
return "TestOptimizerPlan
";
}
}
+
+ public static final class TestExecuteTwice {
+
+ public static void main(String args[]) throws Exception {
--- End diff --
> I guess then the invocation of the main method doesn't work properly...
What do you mean by that?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971285#comment-14971285
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42884467
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -304,4 +368,60 @@ public String getDescription() {
return "TestOptimizerPlan
";
}
}
+
+ public static final class TestExecuteTwice {
+
+ public static void main(String args[]) throws Exception {
--- End diff --
A `PackagedProgram` needs to have a main class with a main method with a
`public` modifier. That's why it wouldn't work.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971231#comment-14971231
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42882120
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +141,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class);
+ out.runDetached(prg, 1);
+ } catch (ProgramInvocationException e) {
--- End diff --
Aha. Good point. Will fix it.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971328#comment-14971328
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42886049
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -304,4 +368,60 @@ public String getDescription() {
return "TestOptimizerPlan
";
}
}
+
+ public static final class TestExecuteTwice {
+
+ public static void main(String args[]) throws Exception {
--- End diff --
Yes. But marking the class `private` won't interfere with that. I've
already pushed a fix and it does pass.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969503#comment-14969503
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42778575
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
--- End diff --
Actually, these arguments are not needed. I will simply remove them.
Also, in `textGetExecutionPlan` where these values are used, the locations
are not actually accessed. Only plan is tested. So shouldn't be an issue.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969512#comment-14969512 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-150299549 @mxm , I have addressed some of your comments. Two things still remain: batch + stream jobs, and better tests. Need advice on those. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969411#comment-14969411
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772489
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
@@ -277,12 +278,20 @@ else if (prog.isUsingInteractiveMode()) {
// invoke here
try {
prog.invokeInteractiveModeForExecution();
+ // if we're here, it means the program is valid
for detached mode execution.
+ JobWithJars plan =
ContextEnvironment.ContextEnvironmentFactory.getPlan();
+ JobGraph graph =
ContextEnvironment.ContextEnvironmentFactory.getGraph();
+ if (plan != null && graph == null) {
--- End diff --
What would happen if we created a streaming program and a batch program in
one jar? Then both would be set and only the batch program would be created...
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969413#comment-14969413
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772606
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java
---
@@ -172,5 +184,74 @@ public ExecutionEnvironment
createExecutionEnvironment() {
}
return env;
}
+
+ public static void setPlan(JobWithJars plan) {
+ if (_plan == null) {
+ _plan = plan;
+ } else {
+ throw new DetachedProgramException("The Job was
submitted in detached mode. Only " +
+ "one execution is allowed.
Please make sure your program doesn't call " +
+ "execute twice and/or doesn't
call an eager execution function[collect, " +
+ "print, printToErr, count].\n");
+ }
+ }
+
+ public static JobWithJars getPlan() {
+ return _plan;
+ }
+
+ public static JobGraph getGraph() {
+ return _graph;
+ }
+
+ public static void setGraph(JobGraph graph) {
+ if (_graph == null) {
+ _graph = graph;
+ } else {
+ throw new DetachedProgramException("The Job was
submitted in detached mode. Only " +
+ "one call to execute is
allowed.");
+ }
+ }
+ }
+
+ public static final class DetachedJobExecutionResult extends
JobExecutionResult {
+
+ public static final DetachedJobExecutionResult INSTANCE = new
DetachedJobExecutionResult();
+
+ public DetachedJobExecutionResult() {
--- End diff --
Constructor should be private to be a singleton.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969418#comment-14969418
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772695
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -304,4 +368,60 @@ public String getDescription() {
return "TestOptimizerPlan
";
}
}
+
+ public static final class TestExecuteTwice {
+
+ public static void main(String args[]) throws Exception {
--- End diff --
These are no proper tests.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969414#comment-14969414 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1214#discussion_r42772634 --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java --- @@ -161,6 +171,8 @@ public ContextEnvironmentFactory(Client client, List jarFilesToAttach, this.userCodeClassLoader = userCodeClassLoader; this.defaultParallelism = defaultParallelism; this.wait = wait; + _plan = null; + _graph = null; --- End diff -- Is there a reason for these odd names? `lastBatchPlan` and `lastStreamJobGraph` would be more appropriate. The first one is for batch programs, the latter for JobGraph created by the streaming API. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969416#comment-14969416
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772661
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
--- End diff --
This should write to System.getProperty("java.io.tmpdir") and have some
other Random input to be cross-platform compatible.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969420#comment-14969420 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-150283627 Thanks for the update @sachingoel0101. I've made some comments. I think the pull request needs a bit of cleanup to make things more explicit. The general approach looks good. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969432#comment-14969432
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42774255
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
@@ -277,12 +278,20 @@ else if (prog.isUsingInteractiveMode()) {
// invoke here
try {
prog.invokeInteractiveModeForExecution();
+ // if we're here, it means the program is valid
for detached mode execution.
+ JobWithJars plan =
ContextEnvironment.ContextEnvironmentFactory.getPlan();
+ JobGraph graph =
ContextEnvironment.ContextEnvironmentFactory.getGraph();
+ if (plan != null && graph == null) {
--- End diff --
Ah. I had not considered this possibility. What should be done in such a
case? Execute both programs? That does make sense, but what should be the
return value?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969435#comment-14969435 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on a diff in the pull request: https://github.com/apache/flink/pull/1214#discussion_r42774405 --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java --- @@ -161,6 +171,8 @@ public ContextEnvironmentFactory(Client client, List jarFilesToAttach, this.userCodeClassLoader = userCodeClassLoader; this.defaultParallelism = defaultParallelism; this.wait = wait; + _plan = null; + _graph = null; --- End diff -- I tend to use `_field` appended names instead of `this.field`. But yeah. `last*` makes more sense here. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969447#comment-14969447
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42774659
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
+ out.runDetached(prg, 1);
+ } catch (ProgramInvocationException e) {
+ assertEquals("The Job was submitted in detached mode.
Only one execution is allowed. " +
--- End diff --
Will do.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969446#comment-14969446
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42774635
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
--- End diff --
I was following the code in `testGetExecutionPlan`. Can you elaborate a bit
more?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969448#comment-14969448
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42774775
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -304,4 +368,60 @@ public String getDescription() {
return "TestOptimizerPlan
";
}
}
+
+ public static final class TestExecuteTwice {
+
+ public static void main(String args[]) throws Exception {
--- End diff --
I was concerned about this too. Perhaps a better idea would be have these
classes in the test-jar?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963097#comment-14963097 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-149172503 The test failures are on the flaky scala shell integration test. @mxm @StephanEwen , can you take a look at this so we can add proper documentation to the programming guide. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956617#comment-14956617 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-147997899 Failures on unrelated cases: JIRA 2847, and the scala shell test is flaky. Till is working on it. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952388#comment-14952388 ] ASF GitHub Bot commented on FLINK-2797: --- Github user hsaputra commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-147235260 Seems like each build fail in different places. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951975#comment-14951975 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-147114132 @hsaputra Yes. However, the travis build log doesn't display the exact cause of the failure. In the code, there is code to log the file with prohibited strings at error level, however, it isn't actually happening. The problem is I'm unable to test the module locally. Yarn works without any issues if I use, say, `bin/yarn-session.sh` or start a cluster using `start-yarn.sh`, but the tests are getting stuck. I was just wondering if it requires any special configuration to run the tests. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951854#comment-14951854 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-147091911 I tested this out on yarn and without yarn. Works like a charm. However, the travis builds are failing in `flink-yarn-tests`. Is there any way I can access the travis logs? I tried debugging in IntelliJ but it fails to find the jar. Running on linux is getting stuck at `Connecting to ResourceManager at ubuntu:8032`. Is there anything specific to keep in mind while running yarn tests? > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951957#comment-14951957 ] ASF GitHub Bot commented on FLINK-2797: --- Github user hsaputra commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-147112611 @sachingoel0101 , could you access this URL: https://travis-ci.org/apache/flink/jobs/84664370 You could always set your own Apache Flink forked repo. That way you could try to send PR against your own copy of the repo to kick off Travis. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951151#comment-14951151 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146980054 @mxm @StephanEwen I have an implementation ready. I have also added unit tests to verify the behavior. Let me know if this is an acceptable way. All the relevant changes are in the second commit. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950601#comment-14950601 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146909986 I would solve it pragmatically and simply make the Program interface a requirement for detached programs. Otherwise, fail with an explanation. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950616#comment-14950616 ] ASF GitHub Bot commented on FLINK-2797: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146912518 We can go with the `Program` interface ,but I think that is such a heavy restriction that this effectively renders the feature useless... > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950605#comment-14950605 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146911063 I think we need to ensure that interactive programs which can execute successfully should still be allowed. What I proposed should work. I can push a commit and we can discuss whether it's a nice solution or not. But it most certainly makes the detached mode more useful. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950581#comment-14950581 ] ASF GitHub Bot commented on FLINK-2797: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146908658 The way we do this with the Optimizer plan is a bit pragmatic. We wick the first runnable part and use that. We could do the same here. Detached mode would submit the first part of the program. If the program has more than one submission to the jobManager, the other ones would not get executed. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950589#comment-14950589 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146909628 This will fail hard when users rely on the result of the first job. So imagine your program always worked. You submit it detached and then it fails after 12 hours when the first job completes. Granted, we could alleviate the problem a bit by only allowing execute() for detached programs because then the date has been written to a proper sink. Still, the accumulator values might be used after job completion. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950629#comment-14950629 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146913514 IMO even removing the detached mode might be better than restricting to `Program` interface. Instead of failing late in the execution, or with a non-informative warning, it would be better to fail before submitting anything to the JobManager, and throw a nice exception message documenting the exact cause of failure. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950634#comment-14950634 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146914064 There are three root causes(please add if I've missed any.) 1. Call to execute twice 2. Access job execution results. 3. Executing one of the eager execution methods as the first part of program. [Second time is automatically a failure] We can handle all three easily. If a program doesn't have any of these, it shouldn't be failed. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950643#comment-14950643 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146918295 Good points. I think that should do it, Sachin. Curious, how you'll implement it :) Don't we run into the problem that we never know if another execution followed unless we execute the whole user code (might have some other side effects)? However, it's the only way to check whether we can submit the job detached. Another approach I can think of is bytecode analysis.. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950410#comment-14950410 ] ASF GitHub Bot commented on FLINK-2797: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146875588 I think it would be fine to not have HA for interactive programs. After all, Flink need no client/master interactivity for streaming and iterations, which is where the HA is most important. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950416#comment-14950416 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146877512 +1 for now, just let submission fail for detached interactive programs (with a nice exception) @sachingoel0101 Could you change the PR accordingly? This would also fix the detached YARN mode for interactive programs. +1 for having a dedicated execution container for YARN in the long run > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950397#comment-14950397 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146874010 Before interactive programs were allowed, the detached mode would have been worked fine because a single JobGraph would have been transferred to the job manager and executed there. This doesn't make sense now, unless we have another dedicated Executor (like you proposed). If that one is on the job manager, we need a fault tolerance mechanism there. Even after the HA changes, the Client is still the a single point of failure for interactive programs. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950412#comment-14950412 ] ASF GitHub Bot commented on FLINK-2797: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146876057 Having a dedicated executor on the JobManager seems like a very big thing, for a not so important issue. Why not just fail when someone tries to submit a interactive program detached? Long term, we can think about deploying the driver program to a YARN container, if desired. That way, the submission process is detached, and the execution follows its core model. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950515#comment-14950515 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146895195 @mxm, just to confirm one thing: Interactive mode is defined when the the main class isn't of the type `Program`. In that case, doesn't it mean we're essentially failing most of the programs in detached mode? Unless my understanding is wrong of course, in which case could you clarify? > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950525#comment-14950525 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146896402 Yes, that is true. We're failing them which is a similar to failing for eager executions but a much cleaner and easier solution. Essentially, we cannot detect what a user does in interactive mode, that's why we shouldn't allow detached mode for interactive programs until we have a more sophisticated solution. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950533#comment-14950533 ] ASF GitHub Bot commented on FLINK-2797: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-146897768 @sachingoel0101 you are right, interactive mode is actually pretty broad (as defined in the Client). What I have been taking about in prior comments was programs that do multiple round trips between JobManager and client (because of using `collect()` `count()` or so). > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
