Regarding the discussion, I agree with JB to keep the runner in the Nemo codebase for now, as there are a number of ongoing developments related to supporting the streaming functionalities. Also, as Nemo relies on programming layers like Apache Beam and Apache Spark, it feels better to let the runner live in the Nemo codebase, if both ways work to list the Nemo runner as an official runner in Beam.
With the directions set, I'll send an email to the Beam-dev mailing list inquiring the process required to list the Nemo runner as an official runner in Beam, and let you know how it goes on the thread! 😀 Wonook 2018년 11월 12일 (월) 오후 3:32, John Yang <johnya...@gmail.com>님이 작성: > Thanks JB for your mail. > > I think the Beam portability framework is an exciting feature that Nemo > wants to support in the future like Flink, although my personal focus at > the moment is on streaming and overall performance/reliability. > > A quick update on the "When"/"How" parts of the capability matrix: Taegeon > confirmed (on his private development branch) that NEXMark Q0-Q7 streaming > queries run on Nemo except for Q3(timer, stateful) and Q7(sideinput), so I > think we have a good coverage, although we haven't yet compared the > outputs/performance with Flink. > > Best, > John > > > On Mon, Nov 12, 2018 at 2:06 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > Thanks for the update. > > > > Regarding the governance, it's also related to visibility. > > > > My preference would be to have the runner in nemo codebase. It gives us > > a complete control and managed our own release cycle. > > > > However, it reduces the visibility for the beam community (or we have to > > send update on the beam mailing list). The nemo runner can be listed as > > official runner in Beam wherever the code is located. > > > > I have a question about the runner: do we plan to support the Beam > > portability layer (Job API, ...) in the nemo runner ? > > > > Regards > > JB > > > > On 12/11/2018 01:23, John Yang wrote: > > > Thanks all for the great discussion. > > > > > > My take on the current status of Beam support in Nemo from the > > perspective > > > of the Beam Capability Matrix[1]: > > > - What: Full support except for SDF / Metrics / Stateful Processing > (Side > > > Inputs for streaming pipelines are not supported yet though) > > > - Where: Probably full support > > > - When: I think we support event-time/processing-time/count triggers, > > but I > > > am not sure we have tested the other triggers. > > > - How: I don't have a good idea on this one, as we haven't had tests > for > > > this. > > > > > > I would say that we have good fundamentals to be on par with many of > the > > > existing official Beam runners. Taegeon and I are experimenting with > the > > > NEXMark benchmark, which I believe cover almost all of the Beam > features, > > > on Nemo to understand in more detail and fix the remaining issues. > > > > > > Regarding the governance I would also like to hear from other members. > > > Taegeon and I are also happy to prioritize things differently depending > > on > > > the plan. > > > > > > Thanks, > > > John > > > > > > [1] Beam Capability Matrix: > > > > > > https://beam.apache.org/documentation/runners/capability-matrix/#cap-summary-what > > > > > > > > > On Sun, Nov 11, 2018 at 1:44 PM Byung-Gon Chun <bgc...@gmail.com> > wrote: > > > > > >> Thanks for the valuable input, Davor! > > >> > > >> Nemo PPMC members, which direction would you like to take? > > >> > > >> -Gon > > >> > > >> On Thu, Nov 8, 2018 at 3:06 AM Davor Bonaci <da...@apache.org> wrote: > > >> > > >>> I think the first decision you have to make is around the governance > of > > >> the > > >>> runner. It can live in the Beam project (and be governed by the Beam > > >> PMC), > > >>> or in the Nemo project (and be governed by the Nemo PPMC). Both are > > >> viable > > >>> paths, and different folks have chosen different paths. > > >>> > > >>> On the technical side, the discussion revolves around API stability > and > > >>> versioning: (1) whether the API surface between the runner and Nemo > > >>> internals is more stable than the API surface between the runner and > > >> Beam's > > >>> runner-facing APIs; and (2) how to version Beam and Nemo, which pairs > > can > > >>> work together, etc. > > >>> > > >>> On the organizational side, delegating governance to Beam PMC would > > mean > > >>> that they control release cadence, contribution process, and > committer > > >>> access. None of it is a particular issue, I think. These are > reasonable > > >>> people, but certainly requires more consensus building and process. > > >>> > > >>> * * * > > >>> > > >>> I don't want to steer you in any direction, and happy to support > > >> whichever > > >>> decision you make. Also, happy to help on the Beam side and make > things > > >>> happen quickly. (But, I'd love to see a meaningful discussion and > > >> consensus > > >>> decision before proceeding.) > > >>> > > >>> On Wed, Nov 7, 2018 at 1:19 AM Jean-Baptiste Onofré <j...@nanthrax.net > > > > >>> wrote: > > >>> > > >>>> Hi guys, > > >>>> > > >>>> definitely happy to help on that front. > > >>>> > > >>>> Regards > > >>>> JB > > >>>> > > >>>> On 07/11/2018 09:54, Byung-Gon Chun wrote: > > >>>>> Thanks for initiating this discussion, Wonwook! > > >>>>> > > >>>>> Davor and JB, it’d be great to get your guide. > > >>>>> > > >>>>> Thanks. > > >>>>> - Gon > > >>>>> > > >>>>> Sent from my iPhone > > >>>>> > > >>>>> 2018. 11. 7. 오후 3:57, 송원욱 <won...@apache.org> 작성: > > >>>>> > > >>>>>> Hi all, > > >>>>>> > > >>>>>> It's nice to hear that the first release is coming up pretty soon > > >> with > > >>>> the > > >>>>>> progress that we are making! > > >>>>>> With the first release and the current development for supporting > > >>> stream > > >>>>>> processing, I think it's time for us to consider sending a request > > >> to > > >>>>>> the *Apache > > >>>>>> Beam* community to include the *support for the Nemo Runner* for > > >> Beam > > >>>>>> applications, as our frontend provides the support for running > Beam > > >>>>>> applications. > > >>>>>> > > >>>>>> Any opinions regarding the issue are welcome! > > >>>>>> I think a word from Davor would greatly help this issue, as he is > a > > >>> PMC > > >>>>>> member of the Apache Beam community and our mentor. Would there > some > > >>>>>> information that you could provide us with? > > >>>>>> > > >>>>>> Thanks a lot! > > >>>>>> Wonook > > >>>> > > >>>> -- > > >>>> Jean-Baptiste Onofré > > >>>> jbono...@apache.org > > >>>> http://blog.nanthrax.net > > >>>> Talend - http://www.talend.com > > >>>> > > >>> > > >> > > >> > > >> -- > > >> Byung-Gon Chun > > >> > > > > > > > -- > > Jean-Baptiste Onofré > > jbono...@apache.org > > http://blog.nanthrax.net > > Talend - http://www.talend.com > > >
