(3) In particular, I see a lot of value for (quoting the proposal)
""" Since then, there have been numerous updates, increased Python parity, and new features that do not have accompanying examples employing best practices and demonstrating an end-to-end experience for new users. We would like to leverage the existing examples by raising their visibility and auditing them. """ and I think the situation would become *worse* on all these fronts with a separate repo (as well as the other issues mentioned, especially complexity). We should consider lowering the bar to liking to user-maintained examples that don't merit being in the main repo, as well as guidelines for adding examples in the main repo itself. On Thu, Aug 9, 2018 at 1:44 PM Ismaël Mejía <ieme...@gmail.com> wrote: > > 3 for all the reasons discussed above. I think there are better ways to > improve the status quo without the extra maintenance of having a new repo for > this. > > On Thu, Aug 9, 2018 at 7:00 PM Ahmet Altay <al...@google.com> wrote: >> >> If we go forward with (3), could we actually update our documentation on how >> we will support casual example contributions? I think we will need to have >> information on how to add links to the new examples people want to add to >> the set, what examples would be good additions to the Beam repo and what >> examples would be better maintained somewhere else by their owners, and what >> could they expect from our community when they work on such examples. >> >> On Thu, Aug 9, 2018 at 9:41 AM, Mikhail Gryzykhin <mig...@google.com> wrote: >>> >>> 3 (if contributors are up for voting) - We want to have beam maintained >>> examples in main repo. This will give good man to users and allow us to >>> test those easily with minimal maintenance. >>> >>> We can add links to opensource user repositories to our documentation/wiki. >>> This will be flexible enough to provide external examples on one hand, and >>> avoid responsibility of maintaining user code on the other hand. >>> >>> --Mikhail >>> >>> Have feedback? >>> >>> >>> On Thu, Aug 9, 2018 at 8:57 AM Rafael Fernandez <rfern...@google.com> wrote: >>>> >>>> Here is the Rose', David's, and Gris' proposal in text form, I hope >>>> the copy/paste helps: >>>> >>>> >>>> Apache Beam Examples Repository >>>> >>>> Authors: Rose Nguyen (rtngu...@google.com), David Cavazos >>>> (dcava...@google.com), Gris Cuevas (g...@apache.org) >>>> >>>> Status: Proposal >>>> Created: 2018-07-30 >>>> Updated: 2018-07-30 >>>> >>>> Summary >>>> >>>> The Apache Beam Community creates and contributes examples to the core >>>> Apache Beam Github repository. We want to make the process easier and >>>> less dependent in the core repository by creating a separate repo, >>>> dedicated solely to Community examples, contribution guidelines and >>>> add the examples to the website. >>>> >>>> Background >>>> >>>> The original batch of examples on the Apache Beam GitHub repository >>>> was donated by Cloud Dataflow at the time of Java SDK 1.x to >>>> demonstrate the capability of this programming model. These initial >>>> examples were intended to demonstrate how a user can put together >>>> their code components and try out Beam. Since then, there have been >>>> numerous updates, increased Python parity, and new features that do >>>> not have accompanying examples employing best practices and >>>> demonstrating an end-to-end experience for new users. We would like to >>>> leverage the existing examples by raising their visibility and >>>> auditing them. This is also an opportunity to establish >>>> contribution/maintenance guidelines for community contributions and to >>>> start hosting the examples on the Beam site in an official repository. >>>> Attracting and retaining new users necessitates updated, concrete >>>> examples that exhibit the range of capabilities of Beam. >>>> >>>> Proposed Tasks >>>> >>>> We would like to create a new GitHub Repository under the Apache >>>> Software Foundation Org page for Apache Beam Community Examples. This >>>> repo would be similar to apache/beam-site. The name we’d like to have >>>> is apache/beam-examples. We will also move all current examples to >>>> this repo, perform an audit to outline best practices and guidelines >>>> and then publish them in the Apache Beam website. >>>> >>>> >>>> Here is an outlined list of tasks we propose: >>>> >>>> Send Apache Beam Example Repository proposal to the mailing list >>>> (David) - July 31 >>>> >>>> Create the GitHub Repo (PMC would need to do this) - Request help >>>> after proposal is refined/accepted >>>> >>>> Move current examples to new repo (David) -- 2 weeks after item 2 is >>>> completed >>>> >>>> Add a note to let people we need to audit for best practices >>>> >>>> Audit current examples and define best practices (David, Rose, Gris) >>>> -- Target date: week of 8/20 >>>> >>>> Write guidelines on adding new examples and maintaining them (Gris, >>>> Rose) -- Week after audit is completed >>>> >>>> Add examples to website (Rose) -- 1 week after guidelines are written >>>> >>>> Publish guidelines in website (Rose) -- 1 week after guidelines are written >>>> >>>> >>>> >>>> On Thu, Aug 9, 2018 at 6:22 AM Łukasz Gajowy <lukasz.gaj...@gmail.com> >>>> wrote: >>>> > >>>> > I'd also vote for 3: I don't see much added value in separating the >>>> > repos and I see much additional effort to be done in maintaining extra >>>> > repo(s) (updating examples when new version of beam sdk comes out) and >>>> > their infrastructure (jenkins, etc). What Lukasz Cwik said about mvn >>>> > archetypes and how easy the examples can be to get starter examples from >>>> > a common repo only strengthens my opinion. >>>> > >>>> > Regarding 2: I think it's not good to have some official examples here >>>> > and some there - IMO it can make a false impression (user experience) >>>> > that some examples are less important than the others. Maybe a good idea >>>> > is to encourage users to share their (independent, non official) >>>> > examples and create a list of such on the beam site instead of 2? >>>> > >>>> > Łukasz >>>> > >>>> > czw., 9 sie 2018 o 11:35 Alexey Romanenko <aromanenko....@gmail.com> >>>> > napisał(a): >>>> >> >>>> >> 3 - I agree with JB, Charles and Lukasz arguments above saying why we >>>> >> need to have examples and main code in the same repository (+ website >>>> >> code base will move there soon). I don’t see any huge benefits to have >>>> >> examples aside and, at the same time, it will bring additional >>>> >> complexity and burden for project support. >>>> >> >>>> >> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: >>>> >> >>>> >> Hi guys, >>>> >> >>>> >> For this kind of discussion, I would prefer to avoid Google Doc and >>>> >> directly put the point/proposal on the mailing list. >>>> >> >>>> >> It's easier for the community to follow. >>>> >> >>>> >> The statement is more for 3 because it's more convenient for users to >>>> >> easily find the examples and include in the distribution. >>>> >> >>>> >> Regards >>>> >> JB >>>> >> >>>> >> On 08/08/2018 23:25, Charles Chen wrote: >>>> >> >>>> >> It looks like the main claim is that 1 and 2 have the benefit of >>>> >> increasing visibility for examples on the Beam site. I agree with >>>> >> Robert's comments on the doc which claim that this is orthogonal to >>>> >> whether a separate repository is created (the comments are unresolved: >>>> >> https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY). >>>> >> >>>> >> I would add that the maintenance and testing burden has not been >>>> >> adequately addressed in the proposal (i.e. are we creating new Jenkins >>>> >> jobs?; will postcommits on the main Beam repo run examples tests?; are >>>> >> we releasing artifacts--if so, is this together with the main package or >>>> >> separately in new packages?). If we go with the half-way solution in >>>> >> (2), there is also the issue of where the threshold is--for example, if >>>> >> a user-contributed example is particularly useful, do we move it to the >>>> >> main repo? >>>> >> >>>> >> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <g...@google.com >>>> >> <mailto:g...@google.com>> wrote: >>>> >> >>>> >> I'd vote for 2. >>>> >> >>>> >> Giving independence to an example repository and creating the right >>>> >> infrastructure to maintain them will give visibility to the efforts >>>> >> our users are creating to solve their uses cases with Beam. I also >>>> >> want to make the process of sharing common work more easily. >>>> >> >>>> >> Re:The examples that will remain in core, I agree that it's crucial >>>> >> to keep some examples for testing. >>>> >> >>>> >> >>>> >> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lc...@google.com >>>> >> <mailto:lc...@google.com>> wrote: >>>> >> >>>> >> I would vote for 3. >>>> >> >>>> >> My reasoning is that Java has a good mechanism to get a >>>> >> starter/example project going by using the the maven archetypes >>>> >> already. Our quickstart guide for Apache Beam for the Java SDK >>>> >> already covers generating the examples archetype. >>>> >> We could point users to the starter project at the end of the >>>> >> java quickstart. >>>> >> >>>> >> If python/go have a similar mechanism that is commonly used, I >>>> >> would go with those over creating a separate repo for examples >>>> >> and adding the maintenance burden involved. >>>> >> >>>> >> >>>> >> >>>> >> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <ruw...@google.com >>>> >> <mailto:ruw...@google.com>> wrote: >>>> >> >>>> >> 2 - examples that rely on experimental API can still stay in >>>> >> where they are because such examples could be changed. >>>> >> >>>> >> -Rui >>>> >> >>>> >> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen <c...@google.com >>>> >> <mailto:c...@google.com>> wrote: >>>> >> >>>> >> 3 - We benefit from increased test coverage by having >>>> >> examples together with the rest of the code. As Robert >>>> >> mentions in the doc, hosting the Beam examples in the >>>> >> main repository is the best way to keep the examples >>>> >> visible, tested and maintained. Given that we recently >>>> >> moved to a single repository for the website since that >>>> >> previously caused a lot of pain, it makes sense to be >>>> >> consistent here. >>>> >> >>>> >> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay >>>> >> <al...@google.com <mailto:al...@google.com>> wrote: >>>> >> >>>> >> 2 - Similar to Huygaa, I see value in keeping a core >>>> >> set of examples tested and maintained against head. >>>> >> At the same time I understand the value of a growing >>>> >> set of community grown examples that are targeted >>>> >> against a pre-defined versions of Beam and not >>>> >> necessarily updated at every release. >>>> >> >>>> >> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan >>>> >> <bat...@google.com <mailto:bat...@google.com>> wrote: >>>> >> >>>> >> 2 - I like the idea of having a separate repo >>>> >> where we can have more freedom to check in >>>> >> examples. However, we benefit from having >>>> >> immediate core examples in Beam for testing >>>> >> purposes. >>>> >> >>>> >> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos >>>> >> <dcava...@google.com >>>> >> <mailto:dcava...@google.com>> wrote: >>>> >> >>>> >> Hi everyone! >>>> >> >>>> >> We discussed several options as well as some >>>> >> of the implications of each option. Please >>>> >> vote for your favorite option, feel free to >>>> >> back it up with any reasons that make you >>>> >> feel that way. >>>> >> >>>> >> 1) Move *all* samples to a *new >>>> >> *examples*repository* >>>> >> 2) Move *some* samples to a *new >>>> >> *examples*repository* >>>> >> 3) Leave samples where they are >>>> >> >>>> >> Some implications to creating a new >>>> >> repository: >>>> >> - Every example would be independent from >>>> >> every other example, so tests can be run in >>>> >> parallel >>>> >> - Examples would now show how to use Beam >>>> >> /externally/ >>>> >> - The examples repository would need a >>>> >> testing infrastructure >>>> >> - Decoupling makes examples easier to test >>>> >> on different versions >>>> >> - Easier to copy-paste an existing example >>>> >> and start from there, almost like a template >>>> >> - Smaller size for the core Beam library >>>> >> - Two different repositories to maintain >>>> >> - Versioning could mirror Beam's current >>>> >> version >>>> >> >>>> >> Link to proposal >>>> >> >>>> >> <https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing> >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Jean-Baptiste Onofré >>>> >> jbono...@apache.org >>>> >> http://blog.nanthrax.net >>>> >> Talend - http://www.talend.com >>>> >> >>>> >> >> >>
