+1 for 2, it would be nice to have a example project in it's own GitHub repo. Might not need to be an "official" repo thought. Could we provide links to community supplied examples?
On Thu, Aug 9, 2018, 2:30 PM Robert Bradshaw <[email protected]> wrote: > (3) > > In particular, I see a lot of value for (quoting the proposal) > > """ > Since then, there have been > numerous updates, increased Python parity, and new features that do > not have accompanying examples employing best practices and > demonstrating an end-to-end experience for new users. We would like to > leverage the existing examples by raising their visibility and > auditing them. > """ > > and I think the situation would become *worse* on all these fronts > with a separate repo (as well as the other issues mentioned, > especially complexity). We should consider lowering the bar to liking > to user-maintained examples that don't merit being in the main repo, > as well as guidelines for adding examples in the main repo itself. > On Thu, Aug 9, 2018 at 1:44 PM Ismaël Mejía <[email protected]> wrote: > > > > 3 for all the reasons discussed above. I think there are better ways to > improve the status quo without the extra maintenance of having a new repo > for this. > > > > On Thu, Aug 9, 2018 at 7:00 PM Ahmet Altay <[email protected]> wrote: > >> > >> If we go forward with (3), could we actually update our documentation > on how we will support casual example contributions? I think we will need > to have information on how to add links to the new examples people want to > add to the set, what examples would be good additions to the Beam repo and > what examples would be better maintained somewhere else by their owners, > and what could they expect from our community when they work on such > examples. > >> > >> On Thu, Aug 9, 2018 at 9:41 AM, Mikhail Gryzykhin <[email protected]> > wrote: > >>> > >>> 3 (if contributors are up for voting) - We want to have beam > maintained examples in main repo. This will give good man to users and > allow us to test those easily with minimal maintenance. > >>> > >>> We can add links to opensource user repositories to our > documentation/wiki. This will be flexible enough to provide external > examples on one hand, and avoid responsibility of maintaining user code on > the other hand. > >>> > >>> --Mikhail > >>> > >>> Have feedback? > >>> > >>> > >>> On Thu, Aug 9, 2018 at 8:57 AM Rafael Fernandez <[email protected]> > wrote: > >>>> > >>>> Here is the Rose', David's, and Gris' proposal in text form, I hope > >>>> the copy/paste helps: > >>>> > >>>> > >>>> Apache Beam Examples Repository > >>>> > >>>> Authors: Rose Nguyen ([email protected]), David Cavazos > >>>> ([email protected]), Gris Cuevas ([email protected]) > >>>> > >>>> Status: Proposal > >>>> Created: 2018-07-30 > >>>> Updated: 2018-07-30 > >>>> > >>>> Summary > >>>> > >>>> The Apache Beam Community creates and contributes examples to the core > >>>> Apache Beam Github repository. We want to make the process easier and > >>>> less dependent in the core repository by creating a separate repo, > >>>> dedicated solely to Community examples, contribution guidelines and > >>>> add the examples to the website. > >>>> > >>>> Background > >>>> > >>>> The original batch of examples on the Apache Beam GitHub repository > >>>> was donated by Cloud Dataflow at the time of Java SDK 1.x to > >>>> demonstrate the capability of this programming model. These initial > >>>> examples were intended to demonstrate how a user can put together > >>>> their code components and try out Beam. Since then, there have been > >>>> numerous updates, increased Python parity, and new features that do > >>>> not have accompanying examples employing best practices and > >>>> demonstrating an end-to-end experience for new users. We would like to > >>>> leverage the existing examples by raising their visibility and > >>>> auditing them. This is also an opportunity to establish > >>>> contribution/maintenance guidelines for community contributions and to > >>>> start hosting the examples on the Beam site in an official repository. > >>>> Attracting and retaining new users necessitates updated, concrete > >>>> examples that exhibit the range of capabilities of Beam. > >>>> > >>>> Proposed Tasks > >>>> > >>>> We would like to create a new GitHub Repository under the Apache > >>>> Software Foundation Org page for Apache Beam Community Examples. This > >>>> repo would be similar to apache/beam-site. The name we’d like to have > >>>> is apache/beam-examples. We will also move all current examples to > >>>> this repo, perform an audit to outline best practices and guidelines > >>>> and then publish them in the Apache Beam website. > >>>> > >>>> > >>>> Here is an outlined list of tasks we propose: > >>>> > >>>> Send Apache Beam Example Repository proposal to the mailing list > >>>> (David) - July 31 > >>>> > >>>> Create the GitHub Repo (PMC would need to do this) - Request help > >>>> after proposal is refined/accepted > >>>> > >>>> Move current examples to new repo (David) -- 2 weeks after item 2 is > completed > >>>> > >>>> Add a note to let people we need to audit for best practices > >>>> > >>>> Audit current examples and define best practices (David, Rose, Gris) > >>>> -- Target date: week of 8/20 > >>>> > >>>> Write guidelines on adding new examples and maintaining them (Gris, > >>>> Rose) -- Week after audit is completed > >>>> > >>>> Add examples to website (Rose) -- 1 week after guidelines are written > >>>> > >>>> Publish guidelines in website (Rose) -- 1 week after guidelines are > written > >>>> > >>>> > >>>> > >>>> On Thu, Aug 9, 2018 at 6:22 AM Łukasz Gajowy <[email protected]> > wrote: > >>>> > > >>>> > I'd also vote for 3: I don't see much added value in separating the > repos and I see much additional effort to be done in maintaining extra > repo(s) (updating examples when new version of beam sdk comes out) and > their infrastructure (jenkins, etc). What Lukasz Cwik said about mvn > archetypes and how easy the examples can be to get starter examples from a > common repo only strengthens my opinion. > >>>> > > >>>> > Regarding 2: I think it's not good to have some official examples > here and some there - IMO it can make a false impression (user experience) > that some examples are less important than the others. Maybe a good idea is > to encourage users to share their (independent, non official) examples and > create a list of such on the beam site instead of 2? > >>>> > > >>>> > Łukasz > >>>> > > >>>> > czw., 9 sie 2018 o 11:35 Alexey Romanenko <[email protected]> > napisał(a): > >>>> >> > >>>> >> 3 - I agree with JB, Charles and Lukasz arguments above saying why > we need to have examples and main code in the same repository (+ website > code base will move there soon). I don’t see any huge benefits to have > examples aside and, at the same time, it will bring additional complexity > and burden for project support. > >>>> >> > >>>> >> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <[email protected]> > wrote: > >>>> >> > >>>> >> Hi guys, > >>>> >> > >>>> >> For this kind of discussion, I would prefer to avoid Google Doc and > >>>> >> directly put the point/proposal on the mailing list. > >>>> >> > >>>> >> It's easier for the community to follow. > >>>> >> > >>>> >> The statement is more for 3 because it's more convenient for users > to > >>>> >> easily find the examples and include in the distribution. > >>>> >> > >>>> >> Regards > >>>> >> JB > >>>> >> > >>>> >> On 08/08/2018 23:25, Charles Chen wrote: > >>>> >> > >>>> >> It looks like the main claim is that 1 and 2 have the benefit of > >>>> >> increasing visibility for examples on the Beam site. I agree with > >>>> >> Robert's comments on the doc which claim that this is orthogonal to > >>>> >> whether a separate repository is created (the comments are > unresolved: > >>>> >> > https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY > ). > >>>> >> > >>>> >> I would add that the maintenance and testing burden has not been > >>>> >> adequately addressed in the proposal (i.e. are we creating new > Jenkins > >>>> >> jobs?; will postcommits on the main Beam repo run examples tests?; > are > >>>> >> we releasing artifacts--if so, is this together with the main > package or > >>>> >> separately in new packages?). If we go with the half-way solution > in > >>>> >> (2), there is also the issue of where the threshold is--for > example, if > >>>> >> a user-contributed example is particularly useful, do we move it > to the > >>>> >> main repo? > >>>> >> > >>>> >> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <[email protected] > >>>> >> <mailto:[email protected]>> wrote: > >>>> >> > >>>> >> I'd vote for 2. > >>>> >> > >>>> >> Giving independence to an example repository and creating the > right > >>>> >> infrastructure to maintain them will give visibility to the > efforts > >>>> >> our users are creating to solve their uses cases with Beam. I > also > >>>> >> want to make the process of sharing common work more easily. > >>>> >> > >>>> >> Re:The examples that will remain in core, I agree that it's > crucial > >>>> >> to keep some examples for testing. > >>>> >> > >>>> >> > >>>> >> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <[email protected] > >>>> >> <mailto:[email protected]>> wrote: > >>>> >> > >>>> >> I would vote for 3. > >>>> >> > >>>> >> My reasoning is that Java has a good mechanism to get a > >>>> >> starter/example project going by using the the maven > archetypes > >>>> >> already. Our quickstart guide for Apache Beam for the Java > SDK > >>>> >> already covers generating the examples archetype. > >>>> >> We could point users to the starter project at the end of > the > >>>> >> java quickstart. > >>>> >> > >>>> >> If python/go have a similar mechanism that is commonly > used, I > >>>> >> would go with those over creating a separate repo for > examples > >>>> >> and adding the maintenance burden involved. > >>>> >> > >>>> >> > >>>> >> > >>>> >> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <[email protected] > >>>> >> <mailto:[email protected]>> wrote: > >>>> >> > >>>> >> 2 - examples that rely on experimental API can still > stay in > >>>> >> where they are because such examples could be changed. > >>>> >> > >>>> >> -Rui > >>>> >> > >>>> >> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen < > [email protected] > >>>> >> <mailto:[email protected]>> wrote: > >>>> >> > >>>> >> 3 - We benefit from increased test coverage by > having > >>>> >> examples together with the rest of the code. As > Robert > >>>> >> mentions in the doc, hosting the Beam examples in > the > >>>> >> main repository is the best way to keep the examples > >>>> >> visible, tested and maintained. Given that we > recently > >>>> >> moved to a single repository for the website since > that > >>>> >> previously caused a lot of pain, it makes sense to > be > >>>> >> consistent here. > >>>> >> > >>>> >> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay > >>>> >> <[email protected] <mailto:[email protected]>> wrote: > >>>> >> > >>>> >> 2 - Similar to Huygaa, I see value in keeping a > core > >>>> >> set of examples tested and maintained against > head. > >>>> >> At the same time I understand the value of a > growing > >>>> >> set of community grown examples that are > targeted > >>>> >> against a pre-defined versions of Beam and not > >>>> >> necessarily updated at every release. > >>>> >> > >>>> >> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa > Batsaikhan > >>>> >> <[email protected] <mailto:[email protected]>> > wrote: > >>>> >> > >>>> >> 2 - I like the idea of having a separate > repo > >>>> >> where we can have more freedom to check in > >>>> >> examples. However, we benefit from having > >>>> >> immediate core examples in Beam for testing > >>>> >> purposes. > >>>> >> > >>>> >> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos > >>>> >> <[email protected] > >>>> >> <mailto:[email protected]>> wrote: > >>>> >> > >>>> >> Hi everyone! > >>>> >> > >>>> >> We discussed several options as well as > some > >>>> >> of the implications of each option. > Please > >>>> >> vote for your favorite option, feel > free to > >>>> >> back it up with any reasons that make > you > >>>> >> feel that way. > >>>> >> > >>>> >> 1) Move *all* samples to a *new > >>>> >> *examples*repository* > >>>> >> 2) Move *some* samples to a *new > >>>> >> *examples*repository* > >>>> >> 3) Leave samples where they are > >>>> >> > >>>> >> Some implications to creating a new > repository: > >>>> >> - Every example would be independent > from > >>>> >> every other example, so tests can be > run in > >>>> >> parallel > >>>> >> - Examples would now show how to use > Beam > >>>> >> /externally/ > >>>> >> - The examples repository would need a > >>>> >> testing infrastructure > >>>> >> - Decoupling makes examples easier to > test > >>>> >> on different versions > >>>> >> - Easier to copy-paste an existing > example > >>>> >> and start from there, almost like a > template > >>>> >> - Smaller size for the core Beam library > >>>> >> - Two different repositories to maintain > >>>> >> - Versioning could mirror Beam's > current version > >>>> >> > >>>> >> Link to proposal > >>>> >> < > https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing > > > >>>> >> > >>>> >> > >>>> >> > >>>> >> -- > >>>> >> Jean-Baptiste Onofré > >>>> >> [email protected] > >>>> >> http://blog.nanthrax.net > >>>> >> Talend - http://www.talend.com > >>>> >> > >>>> >> > >> > >> >
