Hi, Has anyone started on https://issues.apache.org/jira/browse/BEAM-2588 <https://issues.apache.org/jira/browse/BEAM-2588> (FlinkRunner shim for serving Job API). If not I would start on that.
My plan is to implement a FlinkJobService that implements JobServiceImplBase, similar to ReferenceRunnerJobService. This would have a lot of the functionality that FlinkRunner currently has. As a next step, I would add a JobServiceRunner that can submit Pipelines to a JobService. For testing, I would probably add functionality that allows spinning up a JobService in-process with the JobServiceRunner. I can imagine for testing we could even eventually use something like: "--runner=JobServiceRunner", "--streaming=true", "--jobService=FlinkRunnerJobService". Once all of this is done, we only need the python component that talks to the JobService to submit a pipeline. What do you think about the plan? Btw, I feel that the thing currently called Runner, i.e. FlinkRunner will go way in the long run and we will have FlinkJobService, SparkJobService and whatnot, what do you think? Aljoscha > On 9. Feb 2018, at 01:31, Ben Sidhom <sid...@google.com> wrote: > > Hey all, > > We're working on getting the portability framework plumbed through the Flink > runner. The first iteration will likely only support batch and will be > limited in its deployment flexibility, but hopefully it shouldn't be too > painful to expand this. > > We have the start of a tracking doc here: > https://s.apache.org/portable-beam-on-flink. > > We've documented the general deployment strategy here: > https://s.apache.org/portable-flink-runner-overview. > > Feel free to provide comments on the docs or jump in on any of the referenced > bugs. > > -- > -Ben
