Thanks for the update. Regarding the governance, it's also related to visibility.
My preference would be to have the runner in nemo codebase. It gives us a complete control and managed our own release cycle. However, it reduces the visibility for the beam community (or we have to send update on the beam mailing list). The nemo runner can be listed as official runner in Beam wherever the code is located. I have a question about the runner: do we plan to support the Beam portability layer (Job API, ...) in the nemo runner ? Regards JB On 12/11/2018 01:23, John Yang wrote: > Thanks all for the great discussion. > > My take on the current status of Beam support in Nemo from the perspective > of the Beam Capability Matrix[1]: > - What: Full support except for SDF / Metrics / Stateful Processing (Side > Inputs for streaming pipelines are not supported yet though) > - Where: Probably full support > - When: I think we support event-time/processing-time/count triggers, but I > am not sure we have tested the other triggers. > - How: I don't have a good idea on this one, as we haven't had tests for > this. > > I would say that we have good fundamentals to be on par with many of the > existing official Beam runners. Taegeon and I are experimenting with the > NEXMark benchmark, which I believe cover almost all of the Beam features, > on Nemo to understand in more detail and fix the remaining issues. > > Regarding the governance I would also like to hear from other members. > Taegeon and I are also happy to prioritize things differently depending on > the plan. > > Thanks, > John > > [1] Beam Capability Matrix: > https://beam.apache.org/documentation/runners/capability-matrix/#cap-summary-what > > > On Sun, Nov 11, 2018 at 1:44 PM Byung-Gon Chun <bgc...@gmail.com> wrote: > >> Thanks for the valuable input, Davor! >> >> Nemo PPMC members, which direction would you like to take? >> >> -Gon >> >> On Thu, Nov 8, 2018 at 3:06 AM Davor Bonaci <da...@apache.org> wrote: >> >>> I think the first decision you have to make is around the governance of >> the >>> runner. It can live in the Beam project (and be governed by the Beam >> PMC), >>> or in the Nemo project (and be governed by the Nemo PPMC). Both are >> viable >>> paths, and different folks have chosen different paths. >>> >>> On the technical side, the discussion revolves around API stability and >>> versioning: (1) whether the API surface between the runner and Nemo >>> internals is more stable than the API surface between the runner and >> Beam's >>> runner-facing APIs; and (2) how to version Beam and Nemo, which pairs can >>> work together, etc. >>> >>> On the organizational side, delegating governance to Beam PMC would mean >>> that they control release cadence, contribution process, and committer >>> access. None of it is a particular issue, I think. These are reasonable >>> people, but certainly requires more consensus building and process. >>> >>> * * * >>> >>> I don't want to steer you in any direction, and happy to support >> whichever >>> decision you make. Also, happy to help on the Beam side and make things >>> happen quickly. (But, I'd love to see a meaningful discussion and >> consensus >>> decision before proceeding.) >>> >>> On Wed, Nov 7, 2018 at 1:19 AM Jean-Baptiste Onofré <j...@nanthrax.net> >>> wrote: >>> >>>> Hi guys, >>>> >>>> definitely happy to help on that front. >>>> >>>> Regards >>>> JB >>>> >>>> On 07/11/2018 09:54, Byung-Gon Chun wrote: >>>>> Thanks for initiating this discussion, Wonwook! >>>>> >>>>> Davor and JB, it’d be great to get your guide. >>>>> >>>>> Thanks. >>>>> - Gon >>>>> >>>>> Sent from my iPhone >>>>> >>>>> 2018. 11. 7. 오후 3:57, 송원욱 <won...@apache.org> 작성: >>>>> >>>>>> Hi all, >>>>>> >>>>>> It's nice to hear that the first release is coming up pretty soon >> with >>>> the >>>>>> progress that we are making! >>>>>> With the first release and the current development for supporting >>> stream >>>>>> processing, I think it's time for us to consider sending a request >> to >>>>>> the *Apache >>>>>> Beam* community to include the *support for the Nemo Runner* for >> Beam >>>>>> applications, as our frontend provides the support for running Beam >>>>>> applications. >>>>>> >>>>>> Any opinions regarding the issue are welcome! >>>>>> I think a word from Davor would greatly help this issue, as he is a >>> PMC >>>>>> member of the Apache Beam community and our mentor. Would there some >>>>>> information that you could provide us with? >>>>>> >>>>>> Thanks a lot! >>>>>> Wonook >>>> >>>> -- >>>> Jean-Baptiste Onofré >>>> jbono...@apache.org >>>> http://blog.nanthrax.net >>>> Talend - http://www.talend.com >>>> >>> >> >> >> -- >> Byung-Gon Chun >> > -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
