What's the plan for the endpoints that the Flink operator needs to provide (control/data plane, state, logging)? Is the intention to provide base implementations that can be shared across runners and then implement the Flink specific parts on top of it? Has work started on those?
If there are subtasks ready to be taken up I would be interested. Thanks, Thomas On Wed, Mar 7, 2018 at 9:35 AM, Ben Sidhom <sid...@google.com> wrote: > Yes, Axel has started work on such a shim. > > Our plan in the short term is to keep the old FlinkRunner around and to > call into it to process jobs from the job service itself. That way we can > keep the non-portable runner fully-functional while working on portability. > Eventually, I think it makes sense for this to go away, but we haven't > given much thought to that. The translator layer will likely stay the same, > and the FlinkRunner bits are a relatively simple wrapper around > translation, so it should be simple enough to factor this out. > > Much of the service code from the Universal Local Runner (ULR) should be > composed and reused with other runner implementations. Thomas and Axel have > more context around that. > > > On Wed, Mar 7, 2018 at 8:47 AM Aljoscha Krettek <aljos...@apache.org> > wrote: > >> Hi, >> >> Has anyone started on https://issues.apache.org/jira/browse/BEAM-2588 >> (FlinkRunner >> shim for serving Job API). If not I would start on that. >> >> My plan is to implement a FlinkJobService that implements JobServiceImplBase, >> similar to ReferenceRunnerJobService. This would have a lot of the >> functionality that FlinkRunner currently has. As a next step, I would add a >> JobServiceRunner that can submit Pipelines to a JobService. >> >> For testing, I would probably add functionality that allows spinning up a >> JobService in-process with the JobServiceRunner. I can imagine for testing >> we could even eventually use something like: >> "--runner=JobServiceRunner", "--streaming=true", "--jobService= >> FlinkRunnerJobService". >> >> Once all of this is done, we only need the python component that talks to >> the JobService to submit a pipeline. >> >> What do you think about the plan? >> >> Btw, I feel that the thing currently called Runner, i.e. FlinkRunner will >> go way in the long run and we will have FlinkJobService, SparkJobService >> and whatnot, what do you think? >> >> Aljoscha >> >> >> On 9. Feb 2018, at 01:31, Ben Sidhom <sid...@google.com> wrote: >> >> Hey all, >> >> We're working on getting the portability framework plumbed through the >> Flink runner. The first iteration will likely only support batch and will >> be limited in its deployment flexibility, but hopefully it shouldn't be too >> painful to expand this. >> >> We have the start of a tracking doc here: https://s.apache.org/ >> portable-beam-on-flink. >> >> We've documented the general deployment strategy here: >> https://s.apache.org/portable-flink-runner-overview. >> >> Feel free to provide comments on the docs or jump in on any of the >> referenced bugs. >> >> -- >> -Ben >> >> >> > > -- > -Ben >
