[jira] [Commented] (FLINK-2313) Change Streaming Driver Execution Model
[
https://issues.apache.org/jira/browse/FLINK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17336915#comment-17336915
]
Flink Jira Bot commented on FLINK-2313:
---
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Change Streaming Driver Execution Model
> ---
>
> Key: FLINK-2313
> URL: https://issues.apache.org/jira/browse/FLINK-2313
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Stephan Ewen
>Priority: Major
> Labels: stale-major
>
> I would propose the following change to the Streaming Execution Environment:
> When calling {{env.execute()}}, it does not block, but return an
> {{ExecutionContext}} object. This execution context has various methods, that
> allow you to control the execution:
> - waitFor() -> Blocks until the job terminates, or the connection to the
> JobManager is lost.
> - stop() -> initiates a clean stop
> - cancel() -> cancels the program
> - setParallelism() -> Later: adjusts the parallelism of the program
> If this is voted as desirable, I would make a design how to do this...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-2313) Change Streaming Driver Execution Model
[
https://issues.apache.org/jira/browse/FLINK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17328979#comment-17328979
]
Flink Jira Bot commented on FLINK-2313:
---
This major issue is unassigned and itself and all of its Sub-Tasks have not
been updated for 30 days. So, it has been labeled "stale-major". If this ticket
is indeed "major", please either assign yourself or give an update. Afterwards,
please remove the label. In 7 days the issue will be deprioritized.
> Change Streaming Driver Execution Model
> ---
>
> Key: FLINK-2313
> URL: https://issues.apache.org/jira/browse/FLINK-2313
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Stephan Ewen
>Priority: Major
> Labels: stale-major
>
> I would propose the following change to the Streaming Execution Environment:
> When calling {{env.execute()}}, it does not block, but return an
> {{ExecutionContext}} object. This execution context has various methods, that
> allow you to control the execution:
> - waitFor() -> Blocks until the job terminates, or the connection to the
> JobManager is lost.
> - stop() -> initiates a clean stop
> - cancel() -> cancels the program
> - setParallelism() -> Later: adjusts the parallelism of the program
> If this is voted as desirable, I would make a design how to do this...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-2313) Change Streaming Driver Execution Model
[
https://issues.apache.org/jira/browse/FLINK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15575392#comment-15575392
]
Maximilian Michels commented on FLINK-2313:
---
+1 this also popped up recently on the Beam side:
https://issues.apache.org/jira/browse/BEAM-593
This is partially also duplicated in
https://issues.apache.org/jira/browse/FLINK-4272
> Change Streaming Driver Execution Model
> ---
>
> Key: FLINK-2313
> URL: https://issues.apache.org/jira/browse/FLINK-2313
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
>Reporter: Stephan Ewen
>
> I would propose the following change to the Streaming Execution Environment:
> When calling {{env.execute()}}, it does not block, but return an
> {{ExecutionContext}} object. This execution context has various methods, that
> allow you to control the execution:
> - waitFor() -> Blocks until the job terminates, or the connection to the
> JobManager is lost.
> - stop() -> initiates a clean stop
> - cancel() -> cancels the program
> - setParallelism() -> Later: adjusts the parallelism of the program
> If this is voted as desirable, I would make a design how to do this...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2313) Change Streaming Driver Execution Model
[
https://issues.apache.org/jira/browse/FLINK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15574877#comment-15574877
]
David Anderson commented on FLINK-2313:
---
+1
This would be super helpful when working with streams inside notebook
environments (like Jupyter and Zeppelin). In the case of a local flink, could
we still keep stdout connected to the notebook?
> Change Streaming Driver Execution Model
> ---
>
> Key: FLINK-2313
> URL: https://issues.apache.org/jira/browse/FLINK-2313
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
>Reporter: Stephan Ewen
>
> I would propose the following change to the Streaming Execution Environment:
> When calling {{env.execute()}}, it does not block, but return an
> {{ExecutionContext}} object. This execution context has various methods, that
> allow you to control the execution:
> - waitFor() -> Blocks until the job terminates, or the connection to the
> JobManager is lost.
> - stop() -> initiates a clean stop
> - cancel() -> cancels the program
> - setParallelism() -> Later: adjusts the parallelism of the program
> If this is voted as desirable, I would make a design how to do this...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2313) Change Streaming Driver Execution Model
[
https://issues.apache.org/jira/browse/FLINK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342845#comment-15342845
]
Gyula Fora commented on FLINK-2313:
---
Btw, do you have any thoughts on how should this behave in case of job
failures/restarts etc? This is also related to Robert's question regarding the
job id.
> Change Streaming Driver Execution Model
> ---
>
> Key: FLINK-2313
> URL: https://issues.apache.org/jira/browse/FLINK-2313
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
>Reporter: Stephan Ewen
>
> I would propose the following change to the Streaming Execution Environment:
> When calling {{env.execute()}}, it does not block, but return an
> {{ExecutionContext}} object. This execution context has various methods, that
> allow you to control the execution:
> - waitFor() -> Blocks until the job terminates, or the connection to the
> JobManager is lost.
> - stop() -> initiates a clean stop
> - cancel() -> cancels the program
> - setParallelism() -> Later: adjusts the parallelism of the program
> If this is voted as desirable, I would make a design how to do this...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2313) Change Streaming Driver Execution Model
[
https://issues.apache.org/jira/browse/FLINK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342789#comment-15342789
]
Robert Metzger commented on FLINK-2313:
---
Another important feature here would be to make the job id, and in the yarn
case also the yarn application id available.
> Change Streaming Driver Execution Model
> ---
>
> Key: FLINK-2313
> URL: https://issues.apache.org/jira/browse/FLINK-2313
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
>Reporter: Stephan Ewen
>
> I would propose the following change to the Streaming Execution Environment:
> When calling {{env.execute()}}, it does not block, but return an
> {{ExecutionContext}} object. This execution context has various methods, that
> allow you to control the execution:
> - waitFor() -> Blocks until the job terminates, or the connection to the
> JobManager is lost.
> - stop() -> initiates a clean stop
> - cancel() -> cancels the program
> - setParallelism() -> Later: adjusts the parallelism of the program
> If this is voted as desirable, I would make a design how to do this...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2313) Change Streaming Driver Execution Model
[
https://issues.apache.org/jira/browse/FLINK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631264#comment-14631264
]
Maximilian Michels commented on FLINK-2313:
---
+1
Why should this only be changed for the {{StreamExecutionEnvironment}}? This
could also go into the batch {{ExecutionEnvironment}}. Ultimately, it would be
nice to join the two.
> Change Streaming Driver Execution Model
> ---
>
> Key: FLINK-2313
> URL: https://issues.apache.org/jira/browse/FLINK-2313
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
>Reporter: Stephan Ewen
>
> I would propose the following change to the Streaming Execution Environment:
> When calling {{env.execute()}}, it does not block, but return an
> {{ExecutionContext}} object. This execution context has various methods, that
> allow you to control the execution:
> - waitFor() -> Blocks until the job terminates, or the connection to the
> JobManager is lost.
> - stop() -> initiates a clean stop
> - cancel() -> cancels the program
> - setParallelism() -> Later: adjusts the parallelism of the program
> If this is voted as desirable, I would make a design how to do this...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2313) Change Streaming Driver Execution Model
[
https://issues.apache.org/jira/browse/FLINK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631076#comment-14631076
]
Matthias J. Sax commented on FLINK-2313:
Cool. I like this. Hope to I find some time to work on it soon.
> Change Streaming Driver Execution Model
> ---
>
> Key: FLINK-2313
> URL: https://issues.apache.org/jira/browse/FLINK-2313
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
>Reporter: Stephan Ewen
>
> I would propose the following change to the Streaming Execution Environment:
> When calling {{env.execute()}}, it does not block, but return an
> {{ExecutionContext}} object. This execution context has various methods, that
> allow you to control the execution:
> - waitFor() -> Blocks until the job terminates, or the connection to the
> JobManager is lost.
> - stop() -> initiates a clean stop
> - cancel() -> cancels the program
> - setParallelism() -> Later: adjusts the parallelism of the program
> If this is voted as desirable, I would make a design how to do this...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2313) Change Streaming Driver Execution Model
[
https://issues.apache.org/jira/browse/FLINK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14611753#comment-14611753
]
Stephan Ewen commented on FLINK-2313:
-
Since this is API breaking, let's get at least the minimal change into the next
release. Minimal would be to return the object with the {{wairFor()}} method.
I also created a doc to collect breaking changes to the streaming API:
https://docs.google.com/document/d/1itrALuaj_mmBMFqfEI1pOt8m8R1BIsmVFS_uuabsJIE
> Change Streaming Driver Execution Model
> ---
>
> Key: FLINK-2313
> URL: https://issues.apache.org/jira/browse/FLINK-2313
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
>Reporter: Stephan Ewen
>
> I would propose the following change to the Streaming Execution Environment:
> When calling {{env.execute()}}, it does not block, but return an
> {{ExecutionContext}} object. This execution context has various methods, that
> allow you to control the execution:
> - waitFor() -> Blocks until the job terminates, or the connection to the
> JobManager is lost.
> - stop() -> initiates a clean stop
> - cancel() -> cancels the program
> - setParallelism() -> Later: adjusts the parallelism of the program
> If this is voted as desirable, I would make a design how to do this...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2313) Change Streaming Driver Execution Model
[
https://issues.apache.org/jira/browse/FLINK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14611750#comment-14611750
]
Aljoscha Krettek commented on FLINK-2313:
-
+1 this should be super nice
> Change Streaming Driver Execution Model
> ---
>
> Key: FLINK-2313
> URL: https://issues.apache.org/jira/browse/FLINK-2313
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
>Reporter: Stephan Ewen
>
> I would propose the following change to the Streaming Execution Environment:
> When calling {{env.execute()}}, it does not block, but return an
> {{ExecutionContext}} object. This execution context has various methods, that
> allow you to control the execution:
> - waitFor() -> Blocks until the job terminates, or the connection to the
> JobManager is lost.
> - stop() -> initiates a clean stop
> - cancel() -> cancels the program
> - setParallelism() -> Later: adjusts the parallelism of the program
> If this is voted as desirable, I would make a design how to do this...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2313) Change Streaming Driver Execution Model
[
https://issues.apache.org/jira/browse/FLINK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14611730#comment-14611730
]
Robert Metzger commented on FLINK-2313:
---
+1
I can actually imagine more possible methods ... for example getting metrics
from the running application or even operator state.
> Change Streaming Driver Execution Model
> ---
>
> Key: FLINK-2313
> URL: https://issues.apache.org/jira/browse/FLINK-2313
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
>Reporter: Stephan Ewen
>
> I would propose the following change to the Streaming Execution Environment:
> When calling {{env.execute()}}, it does not block, but return an
> {{ExecutionContext}} object. This execution context has various methods, that
> allow you to control the execution:
> - waitFor() -> Blocks until the job terminates, or the connection to the
> JobManager is lost.
> - stop() -> initiates a clean stop
> - cancel() -> cancels the program
> - setParallelism() -> Later: adjusts the parallelism of the program
> If this is voted as desirable, I would make a design how to do this...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2313) Change Streaming Driver Execution Model
[
https://issues.apache.org/jira/browse/FLINK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14611729#comment-14611729
]
Gyula Fora commented on FLINK-2313:
---
I think this would be a very cool change. I also don't like the blocking
execute calls, once it is removed we can nicely integrate a collect() method to
the data streams which will return the iterators over the streams like we have
now in the DataStreamUtils.
> Change Streaming Driver Execution Model
> ---
>
> Key: FLINK-2313
> URL: https://issues.apache.org/jira/browse/FLINK-2313
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
>Reporter: Stephan Ewen
>
> I would propose the following change to the Streaming Execution Environment:
> When calling {{env.execute()}}, it does not block, but return an
> {{ExecutionContext}} object. This execution context has various methods, that
> allow you to control the execution:
> - waitFor() -> Blocks until the job terminates, or the connection to the
> JobManager is lost.
> - stop() -> initiates a clean stop
> - cancel() -> cancels the program
> - setParallelism() -> Later: adjusts the parallelism of the program
> If this is voted as desirable, I would make a design how to do this...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
