It's been a while, but just to let you know that there's a PR up regarding the issue! Anyone who's interested can take a look at https://github.com/apache/beam/pull/7236. Wonook
2018년 11월 16일 (금) 오후 1:16, Jangho Seo <jan...@apache.org>님이 작성: > That also sounds good to me. +1 > > Best, > Jangho > > On 11/15/18 7:29 PM, Byung-Gon Chun wrote: > > Sounds good! > > > > On Thu, Nov 15, 2018 at 6:09 PM Joo Yeon Kim <jooy...@apache.org> wrote: > > > >> This sounds great to me :) +1 > >> > >> On Tue, Nov 13, 2018 at 1:13 PM 송원욱 <won...@apache.org> wrote: > >> > >>> Regarding the discussion, I agree with JB to keep the runner in the > Nemo > >>> codebase for now, as there are a number of ongoing developments related > >> to > >>> supporting the streaming functionalities. Also, as Nemo relies on > >>> programming layers like Apache Beam and Apache Spark, it feels better > to > >>> let the runner live in the Nemo codebase, if both ways work to list the > >>> Nemo runner as an official runner in Beam. > >>> > >>> With the directions set, I'll send an email to the Beam-dev mailing > list > >>> inquiring the process required to list the Nemo runner as an official > >>> runner in Beam, and let you know how it goes on the thread! 😀 > >>> Wonook > >>> > >>> > >>> 2018년 11월 12일 (월) 오후 3:32, John Yang <johnya...@gmail.com>님이 작성: > >>> > >>>> Thanks JB for your mail. > >>>> > >>>> I think the Beam portability framework is an exciting feature that > Nemo > >>>> wants to support in the future like Flink, although my personal focus > >> at > >>>> the moment is on streaming and overall performance/reliability. > >>>> > >>>> A quick update on the "When"/"How" parts of the capability matrix: > >>> Taegeon > >>>> confirmed (on his private development branch) that NEXMark Q0-Q7 > >>> streaming > >>>> queries run on Nemo except for Q3(timer, stateful) and Q7(sideinput), > >> so > >>> I > >>>> think we have a good coverage, although we haven't yet compared the > >>>> outputs/performance with Flink. > >>>> > >>>> Best, > >>>> John > >>>> > >>>> > >>>> On Mon, Nov 12, 2018 at 2:06 PM Jean-Baptiste Onofré <j...@nanthrax.net > > > >>>> wrote: > >>>> > >>>>> Thanks for the update. > >>>>> > >>>>> Regarding the governance, it's also related to visibility. > >>>>> > >>>>> My preference would be to have the runner in nemo codebase. It gives > >> us > >>>>> a complete control and managed our own release cycle. > >>>>> > >>>>> However, it reduces the visibility for the beam community (or we have > >>> to > >>>>> send update on the beam mailing list). The nemo runner can be listed > >> as > >>>>> official runner in Beam wherever the code is located. > >>>>> > >>>>> I have a question about the runner: do we plan to support the Beam > >>>>> portability layer (Job API, ...) in the nemo runner ? > >>>>> > >>>>> Regards > >>>>> JB > >>>>> > >>>>> On 12/11/2018 01:23, John Yang wrote: > >>>>>> Thanks all for the great discussion. > >>>>>> > >>>>>> My take on the current status of Beam support in Nemo from the > >>>>> perspective > >>>>>> of the Beam Capability Matrix[1]: > >>>>>> - What: Full support except for SDF / Metrics / Stateful Processing > >>>> (Side > >>>>>> Inputs for streaming pipelines are not supported yet though) > >>>>>> - Where: Probably full support > >>>>>> - When: I think we support event-time/processing-time/count > >> triggers, > >>>>> but I > >>>>>> am not sure we have tested the other triggers. > >>>>>> - How: I don't have a good idea on this one, as we haven't had > >> tests > >>>> for > >>>>>> this. > >>>>>> > >>>>>> I would say that we have good fundamentals to be on par with many > >> of > >>>> the > >>>>>> existing official Beam runners. Taegeon and I are experimenting > >> with > >>>> the > >>>>>> NEXMark benchmark, which I believe cover almost all of the Beam > >>>> features, > >>>>>> on Nemo to understand in more detail and fix the remaining issues. > >>>>>> > >>>>>> Regarding the governance I would also like to hear from other > >>> members. > >>>>>> Taegeon and I are also happy to prioritize things differently > >>> depending > >>>>> on > >>>>>> the plan. > >>>>>> > >>>>>> Thanks, > >>>>>> John > >>>>>> > >>>>>> [1] Beam Capability Matrix: > >>>>>> > >> > https://beam.apache.org/documentation/runners/capability-matrix/#cap-summary-what > >>>>>> > >>>>>> On Sun, Nov 11, 2018 at 1:44 PM Byung-Gon Chun <bgc...@gmail.com> > >>>> wrote: > >>>>>>> Thanks for the valuable input, Davor! > >>>>>>> > >>>>>>> Nemo PPMC members, which direction would you like to take? > >>>>>>> > >>>>>>> -Gon > >>>>>>> > >>>>>>> On Thu, Nov 8, 2018 at 3:06 AM Davor Bonaci <da...@apache.org> > >>> wrote: > >>>>>>>> I think the first decision you have to make is around the > >>> governance > >>>> of > >>>>>>> the > >>>>>>>> runner. It can live in the Beam project (and be governed by the > >>> Beam > >>>>>>> PMC), > >>>>>>>> or in the Nemo project (and be governed by the Nemo PPMC). Both > >> are > >>>>>>> viable > >>>>>>>> paths, and different folks have chosen different paths. > >>>>>>>> > >>>>>>>> On the technical side, the discussion revolves around API > >> stability > >>>> and > >>>>>>>> versioning: (1) whether the API surface between the runner and > >> Nemo > >>>>>>>> internals is more stable than the API surface between the runner > >>> and > >>>>>>> Beam's > >>>>>>>> runner-facing APIs; and (2) how to version Beam and Nemo, which > >>> pairs > >>>>> can > >>>>>>>> work together, etc. > >>>>>>>> > >>>>>>>> On the organizational side, delegating governance to Beam PMC > >> would > >>>>> mean > >>>>>>>> that they control release cadence, contribution process, and > >>>> committer > >>>>>>>> access. None of it is a particular issue, I think. These are > >>>> reasonable > >>>>>>>> people, but certainly requires more consensus building and > >> process. > >>>>>>>> * * * > >>>>>>>> > >>>>>>>> I don't want to steer you in any direction, and happy to support > >>>>>>> whichever > >>>>>>>> decision you make. Also, happy to help on the Beam side and make > >>>> things > >>>>>>>> happen quickly. (But, I'd love to see a meaningful discussion and > >>>>>>> consensus > >>>>>>>> decision before proceeding.) > >>>>>>>> > >>>>>>>> On Wed, Nov 7, 2018 at 1:19 AM Jean-Baptiste Onofré < > >>> j...@nanthrax.net > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi guys, > >>>>>>>>> > >>>>>>>>> definitely happy to help on that front. > >>>>>>>>> > >>>>>>>>> Regards > >>>>>>>>> JB > >>>>>>>>> > >>>>>>>>> On 07/11/2018 09:54, Byung-Gon Chun wrote: > >>>>>>>>>> Thanks for initiating this discussion, Wonwook! > >>>>>>>>>> > >>>>>>>>>> Davor and JB, it’d be great to get your guide. > >>>>>>>>>> > >>>>>>>>>> Thanks. > >>>>>>>>>> - Gon > >>>>>>>>>> > >>>>>>>>>> Sent from my iPhone > >>>>>>>>>> > >>>>>>>>>> 2018. 11. 7. 오후 3:57, 송원욱 <won...@apache.org> 작성: > >>>>>>>>>> > >>>>>>>>>>> Hi all, > >>>>>>>>>>> > >>>>>>>>>>> It's nice to hear that the first release is coming up pretty > >>> soon > >>>>>>> with > >>>>>>>>> the > >>>>>>>>>>> progress that we are making! > >>>>>>>>>>> With the first release and the current development for > >>> supporting > >>>>>>>> stream > >>>>>>>>>>> processing, I think it's time for us to consider sending a > >>> request > >>>>>>> to > >>>>>>>>>>> the *Apache > >>>>>>>>>>> Beam* community to include the *support for the Nemo Runner* > >> for > >>>>>>> Beam > >>>>>>>>>>> applications, as our frontend provides the support for running > >>>> Beam > >>>>>>>>>>> applications. > >>>>>>>>>>> > >>>>>>>>>>> Any opinions regarding the issue are welcome! > >>>>>>>>>>> I think a word from Davor would greatly help this issue, as he > >>> is > >>>> a > >>>>>>>> PMC > >>>>>>>>>>> member of the Apache Beam community and our mentor. Would > >> there > >>>> some > >>>>>>>>>>> information that you could provide us with? > >>>>>>>>>>> > >>>>>>>>>>> Thanks a lot! > >>>>>>>>>>> Wonook > >>>>>>>>> -- > >>>>>>>>> Jean-Baptiste Onofré > >>>>>>>>> jbono...@apache.org > >>>>>>>>> http://blog.nanthrax.net > >>>>>>>>> Talend - http://www.talend.com > >>>>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Byung-Gon Chun > >>>>>>> > >>>>> -- > >>>>> Jean-Baptiste Onofré > >>>>> jbono...@apache.org > >>>>> http://blog.nanthrax.net > >>>>> Talend - http://www.talend.com > >>>>> > > >
