Has anyone started on https://issues.apache.org/jira/browse/BEAM-2588 
<https://issues.apache.org/jira/browse/BEAM-2588> (FlinkRunner shim for serving 
Job API). If not I would start on that.

My plan is to implement a FlinkJobService that implements JobServiceImplBase, 
similar to ReferenceRunnerJobService. This would have a lot of the 
functionality that FlinkRunner currently has. As a next step, I would add a 
JobServiceRunner that can submit Pipelines to a JobService.

For testing, I would probably add functionality that allows spinning up a 
JobService in-process with the JobServiceRunner. I can imagine for testing we 
could even eventually use something like:
"--runner=JobServiceRunner", "--streaming=true", 

Once all of this is done, we only need the python component that talks to the 
JobService to submit a pipeline.

What do you think about the plan?

Btw, I feel that the thing currently called Runner, i.e. FlinkRunner will go 
way in the long run and we will have FlinkJobService, SparkJobService and 
whatnot, what do you think?


> On 9. Feb 2018, at 01:31, Ben Sidhom <sid...@google.com> wrote:
> Hey all,
> We're working on getting the portability framework plumbed through the Flink 
> runner. The first iteration will likely only support batch and will be 
> limited in its deployment flexibility, but hopefully it shouldn't be too 
> painful to expand this.
> We have the start of a tracking doc here: 
> https://s.apache.org/portable-beam-on-flink.
> We've documented the general deployment strategy here: 
> https://s.apache.org/portable-flink-runner-overview.
> Feel free to provide comments on the docs or jump in on any of the referenced 
> bugs.
> -- 
> -Ben

Reply via email to