I'm curious if pipelines that are exclusively Java will be executed (when running on Flink or other JVM based runnner) in separate harness containers also? This would impose a significant penalty compared to the current execution model. Will this be something the user can control?
Thanks, Thomas On Wed, Mar 7, 2018 at 2:09 PM, Aljoscha Krettek <aljos...@apache.org> wrote: > @Axel I assigned https://issues.apache.org/jira/browse/BEAM-2588 to you. > It might make sense to also grab other issues that you're already working > on. > > > On 7. Mar 2018, at 21:18, Aljoscha Krettek <aljos...@apache.org> wrote: > > Cool, so we had the same ideas. I think this indicates that we're not > completely on the wrong track with this! ;-) > > Aljoscha > > On 7. Mar 2018, at 21:14, Thomas Weise <t...@apache.org> wrote: > > Ben, > > Looks like we hit the send button at the same time. Is the plan the to > derive the Flink implementation of the various execution services from > those under org.apache.beam.runners.fnexecution ? > > Thanks > > On Wed, Mar 7, 2018 at 11:02 AM, Thomas Weise <t...@apache.org> wrote: > >> What's the plan for the endpoints that the Flink operator needs to >> provide (control/data plane, state, logging)? Is the intention to provide >> base implementations that can be shared across runners and then implement >> the Flink specific parts on top of it? Has work started on those? >> >> If there are subtasks ready to be taken up I would be interested. >> >> Thanks, >> Thomas >> >> >> On Wed, Mar 7, 2018 at 9:35 AM, Ben Sidhom <sid...@google.com> wrote: >> >>> Yes, Axel has started work on such a shim. >>> >>> Our plan in the short term is to keep the old FlinkRunner around and to >>> call into it to process jobs from the job service itself. That way we can >>> keep the non-portable runner fully-functional while working on portability. >>> Eventually, I think it makes sense for this to go away, but we haven't >>> given much thought to that. The translator layer will likely stay the same, >>> and the FlinkRunner bits are a relatively simple wrapper around >>> translation, so it should be simple enough to factor this out. >>> >>> Much of the service code from the Universal Local Runner (ULR) should be >>> composed and reused with other runner implementations. Thomas and Axel have >>> more context around that. >>> >>> >>> On Wed, Mar 7, 2018 at 8:47 AM Aljoscha Krettek <aljos...@apache.org> >>> wrote: >>> >>>> Hi, >>>> >>>> Has anyone started on https://issues.apache.org/jira/browse/BEAM-2588 >>>> (FlinkRunner shim for serving Job API). If not I would start on that. >>>> >>>> My plan is to implement a FlinkJobService that implements >>>> JobServiceImplBase, >>>> similar to ReferenceRunnerJobService. This would have a lot of the >>>> functionality that FlinkRunner currently has. As a next step, I would add a >>>> JobServiceRunner that can submit Pipelines to a JobService. >>>> >>>> For testing, I would probably add functionality that allows spinning up >>>> a JobService in-process with the JobServiceRunner. I can imagine for >>>> testing we could even eventually use something like: >>>> "--runner=JobServiceRunner", "--streaming=true", >>>> "--jobService=FlinkRunnerJobService". >>>> >>>> Once all of this is done, we only need the python component that talks >>>> to the JobService to submit a pipeline. >>>> >>>> What do you think about the plan? >>>> >>>> Btw, I feel that the thing currently called Runner, i.e. FlinkRunner >>>> will go way in the long run and we will have FlinkJobService, >>>> SparkJobService and whatnot, what do you think? >>>> >>>> Aljoscha >>>> >>>> >>>> On 9. Feb 2018, at 01:31, Ben Sidhom <sid...@google.com> wrote: >>>> >>>> Hey all, >>>> >>>> We're working on getting the portability framework plumbed through the >>>> Flink runner. The first iteration will likely only support batch and will >>>> be limited in its deployment flexibility, but hopefully it shouldn't be too >>>> painful to expand this. >>>> >>>> We have the start of a tracking doc here: https://s.apache.org/por >>>> table-beam-on-flink. >>>> >>>> We've documented the general deployment strategy here: >>>> https://s.apache.org/portable-flink-runner-overview. >>>> >>>> Feel free to provide comments on the docs or jump in on any of the >>>> referenced bugs. >>>> >>>> -- >>>> -Ben >>>> >>>> >>>> >>> >>> -- >>> -Ben >>> >> >> > > >
