This sounds great to me :) +1 On Tue, Nov 13, 2018 at 1:13 PM 송원욱 <won...@apache.org> wrote:
> Regarding the discussion, I agree with JB to keep the runner in the Nemo > codebase for now, as there are a number of ongoing developments related to > supporting the streaming functionalities. Also, as Nemo relies on > programming layers like Apache Beam and Apache Spark, it feels better to > let the runner live in the Nemo codebase, if both ways work to list the > Nemo runner as an official runner in Beam. > > With the directions set, I'll send an email to the Beam-dev mailing list > inquiring the process required to list the Nemo runner as an official > runner in Beam, and let you know how it goes on the thread! 😀 > Wonook > > > 2018년 11월 12일 (월) 오후 3:32, John Yang <johnya...@gmail.com>님이 작성: > > > Thanks JB for your mail. > > > > I think the Beam portability framework is an exciting feature that Nemo > > wants to support in the future like Flink, although my personal focus at > > the moment is on streaming and overall performance/reliability. > > > > A quick update on the "When"/"How" parts of the capability matrix: > Taegeon > > confirmed (on his private development branch) that NEXMark Q0-Q7 > streaming > > queries run on Nemo except for Q3(timer, stateful) and Q7(sideinput), so > I > > think we have a good coverage, although we haven't yet compared the > > outputs/performance with Flink. > > > > Best, > > John > > > > > > On Mon, Nov 12, 2018 at 2:06 PM Jean-Baptiste Onofré <j...@nanthrax.net> > > wrote: > > > > > Thanks for the update. > > > > > > Regarding the governance, it's also related to visibility. > > > > > > My preference would be to have the runner in nemo codebase. It gives us > > > a complete control and managed our own release cycle. > > > > > > However, it reduces the visibility for the beam community (or we have > to > > > send update on the beam mailing list). The nemo runner can be listed as > > > official runner in Beam wherever the code is located. > > > > > > I have a question about the runner: do we plan to support the Beam > > > portability layer (Job API, ...) in the nemo runner ? > > > > > > Regards > > > JB > > > > > > On 12/11/2018 01:23, John Yang wrote: > > > > Thanks all for the great discussion. > > > > > > > > My take on the current status of Beam support in Nemo from the > > > perspective > > > > of the Beam Capability Matrix[1]: > > > > - What: Full support except for SDF / Metrics / Stateful Processing > > (Side > > > > Inputs for streaming pipelines are not supported yet though) > > > > - Where: Probably full support > > > > - When: I think we support event-time/processing-time/count triggers, > > > but I > > > > am not sure we have tested the other triggers. > > > > - How: I don't have a good idea on this one, as we haven't had tests > > for > > > > this. > > > > > > > > I would say that we have good fundamentals to be on par with many of > > the > > > > existing official Beam runners. Taegeon and I are experimenting with > > the > > > > NEXMark benchmark, which I believe cover almost all of the Beam > > features, > > > > on Nemo to understand in more detail and fix the remaining issues. > > > > > > > > Regarding the governance I would also like to hear from other > members. > > > > Taegeon and I are also happy to prioritize things differently > depending > > > on > > > > the plan. > > > > > > > > Thanks, > > > > John > > > > > > > > [1] Beam Capability Matrix: > > > > > > > > > > https://beam.apache.org/documentation/runners/capability-matrix/#cap-summary-what > > > > > > > > > > > > On Sun, Nov 11, 2018 at 1:44 PM Byung-Gon Chun <bgc...@gmail.com> > > wrote: > > > > > > > >> Thanks for the valuable input, Davor! > > > >> > > > >> Nemo PPMC members, which direction would you like to take? > > > >> > > > >> -Gon > > > >> > > > >> On Thu, Nov 8, 2018 at 3:06 AM Davor Bonaci <da...@apache.org> > wrote: > > > >> > > > >>> I think the first decision you have to make is around the > governance > > of > > > >> the > > > >>> runner. It can live in the Beam project (and be governed by the > Beam > > > >> PMC), > > > >>> or in the Nemo project (and be governed by the Nemo PPMC). Both are > > > >> viable > > > >>> paths, and different folks have chosen different paths. > > > >>> > > > >>> On the technical side, the discussion revolves around API stability > > and > > > >>> versioning: (1) whether the API surface between the runner and Nemo > > > >>> internals is more stable than the API surface between the runner > and > > > >> Beam's > > > >>> runner-facing APIs; and (2) how to version Beam and Nemo, which > pairs > > > can > > > >>> work together, etc. > > > >>> > > > >>> On the organizational side, delegating governance to Beam PMC would > > > mean > > > >>> that they control release cadence, contribution process, and > > committer > > > >>> access. None of it is a particular issue, I think. These are > > reasonable > > > >>> people, but certainly requires more consensus building and process. > > > >>> > > > >>> * * * > > > >>> > > > >>> I don't want to steer you in any direction, and happy to support > > > >> whichever > > > >>> decision you make. Also, happy to help on the Beam side and make > > things > > > >>> happen quickly. (But, I'd love to see a meaningful discussion and > > > >> consensus > > > >>> decision before proceeding.) > > > >>> > > > >>> On Wed, Nov 7, 2018 at 1:19 AM Jean-Baptiste Onofré < > j...@nanthrax.net > > > > > > >>> wrote: > > > >>> > > > >>>> Hi guys, > > > >>>> > > > >>>> definitely happy to help on that front. > > > >>>> > > > >>>> Regards > > > >>>> JB > > > >>>> > > > >>>> On 07/11/2018 09:54, Byung-Gon Chun wrote: > > > >>>>> Thanks for initiating this discussion, Wonwook! > > > >>>>> > > > >>>>> Davor and JB, it’d be great to get your guide. > > > >>>>> > > > >>>>> Thanks. > > > >>>>> - Gon > > > >>>>> > > > >>>>> Sent from my iPhone > > > >>>>> > > > >>>>> 2018. 11. 7. 오후 3:57, 송원욱 <won...@apache.org> 작성: > > > >>>>> > > > >>>>>> Hi all, > > > >>>>>> > > > >>>>>> It's nice to hear that the first release is coming up pretty > soon > > > >> with > > > >>>> the > > > >>>>>> progress that we are making! > > > >>>>>> With the first release and the current development for > supporting > > > >>> stream > > > >>>>>> processing, I think it's time for us to consider sending a > request > > > >> to > > > >>>>>> the *Apache > > > >>>>>> Beam* community to include the *support for the Nemo Runner* for > > > >> Beam > > > >>>>>> applications, as our frontend provides the support for running > > Beam > > > >>>>>> applications. > > > >>>>>> > > > >>>>>> Any opinions regarding the issue are welcome! > > > >>>>>> I think a word from Davor would greatly help this issue, as he > is > > a > > > >>> PMC > > > >>>>>> member of the Apache Beam community and our mentor. Would there > > some > > > >>>>>> information that you could provide us with? > > > >>>>>> > > > >>>>>> Thanks a lot! > > > >>>>>> Wonook > > > >>>> > > > >>>> -- > > > >>>> Jean-Baptiste Onofré > > > >>>> jbono...@apache.org > > > >>>> http://blog.nanthrax.net > > > >>>> Talend - http://www.talend.com > > > >>>> > > > >>> > > > >> > > > >> > > > >> -- > > > >> Byung-Gon Chun > > > >> > > > > > > > > > > -- > > > Jean-Baptiste Onofré > > > jbono...@apache.org > > > http://blog.nanthrax.net > > > Talend - http://www.talend.com > > > > > >
