Open question: did you think to a way to run the portable api on top of any
runner to implement it once? Since runners have primitive it should be
doable and avoid a per runner codebase, no? Other benefit: no direct
portable api code in runners, yeah :). (Im thinking to a runner decorator
or orchestrator or previsitor which rewrites the pipeline before the actual
translations)

Le 7 mars 2018 18:35, "Ben Sidhom" <sid...@google.com> a écrit :

> Yes, Axel has started work on such a shim.
>
> Our plan in the short term is to keep the old FlinkRunner around and to
> call into it to process jobs from the job service itself. That way we can
> keep the non-portable runner fully-functional while working on portability.
> Eventually, I think it makes sense for this to go away, but we haven't
> given much thought to that. The translator layer will likely stay the same,
> and the FlinkRunner bits are a relatively simple wrapper around
> translation, so it should be simple enough to factor this out.
>
> Much of the service code from the Universal Local Runner (ULR) should be
> composed and reused with other runner implementations. Thomas and Axel have
> more context around that.
>
>
> On Wed, Mar 7, 2018 at 8:47 AM Aljoscha Krettek <aljos...@apache.org>
> wrote:
>
>> Hi,
>>
>> Has anyone started on https://issues.apache.org/jira/browse/BEAM-2588 
>> (FlinkRunner
>> shim for serving Job API). If not I would start on that.
>>
>> My plan is to implement a FlinkJobService that implements JobServiceImplBase,
>> similar to ReferenceRunnerJobService. This would have a lot of the
>> functionality that FlinkRunner currently has. As a next step, I would add a
>> JobServiceRunner that can submit Pipelines to a JobService.
>>
>> For testing, I would probably add functionality that allows spinning up a
>> JobService in-process with the JobServiceRunner. I can imagine for testing
>> we could even eventually use something like:
>> "--runner=JobServiceRunner", "--streaming=true", "--jobService=
>> FlinkRunnerJobService".
>>
>> Once all of this is done, we only need the python component that talks to
>> the JobService to submit a pipeline.
>>
>> What do you think about the plan?
>>
>> Btw, I feel that the thing currently called Runner, i.e. FlinkRunner will
>> go way in the long run and we will have FlinkJobService, SparkJobService
>> and whatnot, what do you think?
>>
>> Aljoscha
>>
>>
>> On 9. Feb 2018, at 01:31, Ben Sidhom <sid...@google.com> wrote:
>>
>> Hey all,
>>
>> We're working on getting the portability framework plumbed through the
>> Flink runner. The first iteration will likely only support batch and will
>> be limited in its deployment flexibility, but hopefully it shouldn't be too
>> painful to expand this.
>>
>> We have the start of a tracking doc here: https://s.apache.org/
>> portable-beam-on-flink.
>>
>> We've documented the general deployment strategy here:
>> https://s.apache.org/portable-flink-runner-overview.
>>
>> Feel free to provide comments on the docs or jump in on any of the
>> referenced bugs.
>>
>> --
>> -Ben
>>
>>
>>
>
> --
> -Ben
>

Reply via email to