[jira] [Commented] (BEAM-593) Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in FlinkRunnerResult

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618916#comment-16618916
 ] 

Maximilian Michels commented on BEAM-593:
-

Unfortunately, this is not resolved for the non-portable FlinkRunner. However, 
it works for the portable Runner. 

> Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in 
> FlinkRunnerResult
> ---
>
> Key: BEAM-593
> URL: https://issues.apache.org/jira/browse/BEAM-593
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Pei He
>Assignee: Aljoscha Krettek
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationException in Flink runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-593) Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in FlinkRunnerResult

[Etienne Chauchot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546598#comment-16546598
 ] 

Etienne Chauchot commented on BEAM-593:
---

Should be resolved, I tested it with Nexmark, pipeline.run() is not blocking 
and then pipelineResult.waitUntilFinish(TIMEOUT) blocks until TIMEOUT is 
reached in streaming mode. I think it is the expected behavior. I guess we can 
close it, [~aljoscha] WDYT?

[~borisbolvig] don't you observe the above behavior your side?

> Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in 
> FlinkRunnerResult
> ---
>
> Key: BEAM-593
> URL: https://issues.apache.org/jira/browse/BEAM-593
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Pei He
>Assignee: Aljoscha Krettek
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationException in Flink runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-593) Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in FlinkRunnerResult

[Boris Kjaer (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546236#comment-16546236
 ] 

Boris Kjaer commented on BEAM-593:
--

Has this been resolved or is there progress being made?

Still having problems with blocking pipelines with FlinkRunner.

> Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in 
> FlinkRunnerResult
> ---
>
> Key: BEAM-593
> URL: https://issues.apache.org/jira/browse/BEAM-593
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Pei He
>Assignee: Aljoscha Krettek
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationException in Flink runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-593) Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in FlinkRunnerResult

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084150#comment-16084150
 ] 

ASF GitHub Bot commented on BEAM-593:
-

Github user aljoscha closed the pull request at:

https://github.com/apache/beam/pull/2915


> Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in 
> FlinkRunnerResult
> ---
>
> Key: BEAM-593
> URL: https://issues.apache.org/jira/browse/BEAM-593
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Pei He
>Assignee: Aljoscha Krettek
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationException in Flink runner.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-593) Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in FlinkRunnerResult

[Etienne Chauchot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051640#comment-16051640
 ] 

Etienne Chauchot commented on BEAM-593:
---

Flink termination of streaming pipelines is now ok as far as Nexmark is 
concerned


> Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in 
> FlinkRunnerResult
> ---
>
> Key: BEAM-593
> URL: https://issues.apache.org/jira/browse/BEAM-593
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Pei He
>Assignee: Aljoscha Krettek
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationException in Flink runner.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-593) Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in FlinkRunnerResult

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998540#comment-15998540
 ] 

ASF GitHub Bot commented on BEAM-593:
-

GitHub user aljoscha opened a pull request:

https://github.com/apache/beam/pull/2915

[BEAM-593] Add non-blocking pipeline execution on Flink Runner 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/beam jira-593-async-execute

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2915


commit f0cb644c684b6cbe744ba6f827700895e7dee4ee
Author: Aljoscha Krettek 
Date:   2017-05-05T12:13:01Z

Add FlinkPipelineExecutor with subclasses for batch and streaming

This replaces the old FlinkPipelineExecutionEnvironment which was
responsible for both batch and stream execution, which made the code
more complicated.

commit 3de09e8b060b1d17e5789b09ac0e056c53fbc7b0
Author: Aljoscha Krettek 
Date:   2017-05-05T16:36:29Z

[BEAM-593] Add non-blocking pipeline execution on Flink Runner

This directly uses the lower level interfaces to submit Flink jobs and
to query their state and accumulators.




> Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in 
> FlinkRunnerResult
> ---
>
> Key: BEAM-593
> URL: https://issues.apache.org/jira/browse/BEAM-593
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Pei He
>Assignee: Aljoscha Krettek
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationException in Flink runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-593) Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in FlinkRunnerResult

[Etienne Chauchot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972569#comment-15972569
 ] 

Etienne Chauchot commented on BEAM-593:
---

Adding the links to the Flink related jiras:

https://issues.apache.org/jira/browse/FLINK-2313
https://issues.apache.org/jira/browse/FLINK-4272

> Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in 
> FlinkRunnerResult
> ---
>
> Key: BEAM-593
> URL: https://issues.apache.org/jira/browse/BEAM-593
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Pei He
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationException in Flink runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)