And I've created the repos just now. Kenn
On Mon, Feb 7, 2022 at 10:39 AM Kenneth Knowles <[email protected]> wrote: > Legal question asked at https://issues.apache.org/jira/browse/LEGAL-601 > > Kenn > > On Fri, Feb 4, 2022 at 7:58 AM Danny McCormick <[email protected]> > wrote: > >> Sure - I'm happy to help out with the Actions setup (and/or with the Go >> template). I will say though, the Actions config should be pretty darn >> simple for these examples - >> https://github.com/davidcavazos/beam-java/blob/main/.github/workflows/test.yaml >> seems right, for each language configuration we're targeting we basically >> just want a job with: >> >> - checkout >> - setup-<language> >> - inlined script to run tests >> >> Always happy to help with or consult on any actions issues 🙂 >> >> Thanks, >> Danny >> >> On Fri, Feb 4, 2022 at 10:21 AM Kerry Donny-Clark <[email protected]> >> wrote: >> >>> Danny has extensive experience with GitHub actions, and may be able to >>> help out. >>> Kerry >>> >>> On Thu, Feb 3, 2022, 11:47 PM Kenneth Knowles <[email protected]> wrote: >>> >>>> I'm convinced on all points. My main motivation was to keep it simple. >>>> But of course we should keep it simple for users, not us :-) >>>> >>>> I can take on the task of asking about MIT license and requesting the >>>> repos be created. Not sure if it needs my level of privileges but I'm happy >>>> to do it anyhow. >>>> >>>> Kenn >>>> >>>> On Wed, Feb 2, 2022 at 10:30 AM Robert Bradshaw <[email protected]> >>>> wrote: >>>> >>>>> On Wed, Feb 2, 2022 at 10:12 AM David Cavazos <[email protected]> >>>>> wrote: >>>>> > >>>>> > MIT is much more permissive, but I also don't have any problems >>>>> changing it to Apache license. In any case, how about we create the >>>>> following repos? >>>>> >>>>> For these starter projects, we don't want to encumber any users of >>>>> these templates with any particular licensing requirements (right?) >>>>> and we don't even care about attribution. We want these to be pretty >>>>> much as close to public domain as possible. That's not what the Apache >>>>> licence does. (If it's even relevant, a good argument could likely be >>>>> made for de minis or fair use, but I think it's best to be explicit >>>>> about this. Perhaps this'd be a good question for apache legal? >>>>> >>>>> > apache/beam-starter-java >>>>> > apache/beam-starter-python >>>>> > apache/beam-starter-go >>>>> > apache/beam-starter-kotlin >>>>> > apache/beam-starter-scala >>>>> > >>>>> > We'll start by populating the Java one which is the most pressing >>>>> one and the one that is ready, but the rest should be simpler. >>>>> > >>>>> > +David Huntsperger, tldr; these are minimal starter projects for >>>>> every language. Once we have Java, Python and Go, it might be a good idea >>>>> to change the quickstarts to use these instead of the word count. There is >>>>> already a dedicated word count walkthrough so I think that is already >>>>> covered. >>>>> > >>>>> > If we all agree on the repo names, who can help us create them? >>>>> > >>>>> > On Thu, Jan 27, 2022 at 12:58 PM Robert Bradshaw < >>>>> [email protected]> wrote: >>>>> >> >>>>> >> On Tue, Jan 18, 2022 at 6:17 AM Kenneth Knowles <[email protected]> >>>>> wrote: >>>>> >> > >>>>> >> > Agree with Luke here. "Just git clone and go" is a big part of it. >>>>> >> > >>>>> >> > But also the answer to "I simply don't know what one would put in >>>>> a Python repo than, other than a bare setup.py that lists a dependency on >>>>> apache_beam" is answered by David's initial email and his repo, namely: >>>>> >> > >>>>> >> > - GitHub Actions configuration >>>>> >> > - README.md >>>>> >> > - example that already runs >>>>> >> >>>>> >> OK, fair enough. >>>>> >> >>>>> >> > - LICENSE (notably you've got it as MIT but to be part of Apache >>>>> software it needs to be ASL2) >>>>> >> >>>>> >> On the topic of licence, it's a bit tricky because one doesn't want >>>>> to >>>>> >> bind the users of such a template as being a derivative work of a >>>>> >> too-restrictive licence. The licence of the template itself should >>>>> >> generally be very permissive. >>>>> >> >>>>> >> > On Fri, Jan 14, 2022 at 2:34 PM Luke Cwik <[email protected]> >>>>> wrote: >>>>> >> >> >>>>> >> >> I think for consistency it makes sense to users to be told to >>>>> checkout this git repo for the language of your choice and run. Some repos >>>>> will have more/less than others when it comes to setup necessary. >>>>> >> >> >>>>> >> >> On Fri, Jan 14, 2022 at 2:26 PM Robert Bradshaw < >>>>> [email protected]> wrote: >>>>> >> >>> >>>>> >> >>> +1 for doing this for Java, as setting up a project there is >>>>> quite >>>>> >> >>> complicated. I simply don't know what one would put in a Python >>>>> repo >>>>> >> >>> than, other than a bare setup.py that lists a dependency on >>>>> >> >>> apache_beam. We don't have recommendations on file layout, etc. >>>>> more >>>>> >> >>> than that (though there's plenty of generic advice to be found >>>>> out >>>>> >> >>> there on the topic). I have a hunch go is similar, and >>>>> javascript >>>>> >> >>> would be as well (npm install apache-beam and your package.json >>>>> file >>>>> >> >>> gets updated). >>>>> >> >>> >>>>> >> >>> On Fri, Jan 14, 2022 at 2:17 PM Luke Cwik <[email protected]> >>>>> wrote: >>>>> >> >>> > >>>>> >> >>> > There are several examples already within the Beam repo found >>>>> in: >>>>> >> >>> > https://github.com/apache/beam/tree/master/examples >>>>> >> >>> > https://github.com/apache/beam/tree/master/sdks/go/examples >>>>> >> >>> > >>>>> https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> > On Fri, Jan 14, 2022 at 11:07 AM Sachin Agarwal < >>>>> [email protected]> wrote: >>>>> >> >>> >> >>>>> >> >>> >> I'd love to do something other than Wordcount just for >>>>> novelty/freshness but agreed with the suggestion that having an example in >>>>> each quickstart would be ideal. >>>>> >> >>> >> >>>>> >> >>> >> On Fri, Jan 14, 2022 at 11:06 AM David Huntsperger < >>>>> [email protected]> wrote: >>>>> >> >>> >>> >>>>> >> >>> >>> + 1 to a separate repo for each language. >>>>> >> >>> >>> >>>>> >> >>> >>> Would it make sense to include the Wordcount example in >>>>> each repo? I know that makes the repos less minimal, but we could rewrite >>>>> the quickstarts around these repos instead of the current Wordcount >>>>> examples. Or maybe we don't need to use the Wordcount example in the >>>>> quickstarts... >>>>> >> >>> >>> >>>>> >> >>> >>> On Wed, Jan 12, 2022 at 1:54 PM David Cavazos < >>>>> [email protected]> wrote: >>>>> >> >>> >>>> >>>>> >> >>> >>>> I agree with dropping the archetypes. Less maintenance is >>>>> preferable, and the github repos are more flexible and maintainable. >>>>> >> >>> >>>> >>>>> >> >>> >>>> How about we create: >>>>> >> >>> >>>> >>>>> >> >>> >>>> apache/beam-starter-java >>>>> >> >>> >>>> apache/beam-starter-python >>>>> >> >>> >>>> apache/beam-starter-go >>>>> >> >>> >>>> >>>>> >> >>> >>>> During our OKR planning, +Keith Malvetti would prefer >>>>> having repos for all languages. It makes sense for consistency as well. >>>>> >> >>> >>>> >>>>> >> >>> >>>> On Mon, Jan 10, 2022 at 5:14 PM Luke Cwik < >>>>> [email protected]> wrote: >>>>> >> >>> >>>>> >>>>> >> >>> >>>>> As long as we have tags so that people can pull out a >>>>> specific version of the examples that coincides with a specific SDK >>>>> version >>>>> then we could drop the archetypes. >>>>> >> >>> >>>>> >>>>> >> >>> >>>>> On Mon, Jan 10, 2022 at 4:09 PM Brian Hulette < >>>>> [email protected]> wrote: >>>>> >> >>> >>>>>> >>>>> >> >>> >>>>>> > Being such minimal examples, I don't expect them to >>>>> break commonly, but I think it would be good to make sure tests aren't >>>>> failing when a release is published. >>>>> >> >>> >>>>>> >>>>> >> >>> >>>>>> Yeah it would be very unfortunate if we discovered a >>>>> breakage after the release. Agree we should verify RCs (document as part >>>>> of >>>>> the release process), or even better, add automation to verify the repo >>>>> against snapshots. The automation could be nice to have anyway since it >>>>> provides an example for users to follow if they want to test against >>>>> snapshots and report issues to us sooner. >>>>> >> >>> >>>>>> >>>>> >> >>> >>>>>> >>>>> >> >>> >>>>>> If we move forward with this can we drop the archetype? >>>>> >> >>> >>>>>> >>>>> >> >>> >>>>>> On Fri, Jan 7, 2022 at 3:54 PM Luke Cwik < >>>>> [email protected]> wrote: >>>>> >> >>> >>>>>>> >>>>> >> >>> >>>>>>> Sounds reasonable. >>>>> >> >>> >>>>>>> >>>>> >> >>> >>>>>>> On Wed, Jan 5, 2022 at 12:47 PM David Cavazos < >>>>> [email protected]> wrote: >>>>> >> >>> >>>>>>>> >>>>> >> >>> >>>>>>>> I personally like the idea of a separate repo since we >>>>> can see how a true minimal project looks like. Having it in the main repo >>>>> would inherit build file configurations and other settings that would be >>>>> different from a clean project, so it could be non-trivial to adapt. Also >>>>> as its own repo, it's easier to clone and modify, or create an instance of >>>>> the template. >>>>> >> >>> >>>>>>>> >>>>> >> >>> >>>>>>>> Dependabot can take care of updating the Beam version >>>>> and other dependencies automatically. Testing is already set up via GitHub >>>>> actions for every pull request, so it would automatically be tested as >>>>> soon >>>>> as there is a new dependency version available. >>>>> >> >>> >>>>>>>> >>>>> >> >>> >>>>>>>> Being such minimal examples, I don't expect them to >>>>> break commonly, but I think it would be good to make sure tests aren't >>>>> failing when a release is published. >>>>> >> >>> >>>>>>>> >>>>> >> >>> >>>>>>>> I'm okay with having one repo per language, and having >>>>> all the build systems we want to support for them. As long as we document >>>>> which files are for which build system. That way there are less repos to >>>>> maintain. >>>>> >> >>> >>>>>>>> >>>>> >> >>> >>>>>>>> On Mon, Dec 13, 2021 at 9:25 AM Luke Cwik < >>>>> [email protected]> wrote: >>>>> >> >>> >>>>>>>>> >>>>> >> >>> >>>>>>>>> The github repo is definitely more flexible then the >>>>> archetypes but the archetypes have a few conveniences since they are >>>>> integrated with apache/beam repo. For example, updates/testing are done at >>>>> the same time a corresponding change to the main repo is done (like >>>>> library >>>>> version updates), they are released when the SDK is released. >>>>> >> >>> >>>>>>>>> >>>>> >> >>> >>>>>>>>> Should these be part of the main repo, or a single >>>>> starter repo containing all the starters or one per language or one per >>>>> build system? >>>>> >> >>> >>>>>>>>> >>>>> >> >>> >>>>>>>>> When should updates to the starter happen? >>>>> >> >>> >>>>>>>>> How as a community do we get them to happen (e.g. >>>>> release manager owns it)? >>>>> >> >>> >>>>>>>>> >>>>> >> >>> >>>>>>>>> >>>>> >> >>> >>>>>>>>> On Sun, Dec 12, 2021 at 4:06 PM David Cavazos < >>>>> [email protected]> wrote: >>>>> >> >>> >>>>>>>>>> >>>>> >> >>> >>>>>>>>>> We could do the Maven archetype, but that wouldn't >>>>> work very well for Gradle and SBT users. I think a GitHub template might >>>>> be >>>>> the more flexible option, and we could have something similar for other >>>>> languages as well. Having said that, we could still create a Maven >>>>> archetype. If someone is familiar with that process, please let me know >>>>> since I'm not too familiar with Maven and its ecosystem. >>>>> >> >>> >>>>>>>>>> >>>>> >> >>> >>>>>>>>>> @Ahmet Altay I think right now we only need to pin >>>>> down the name of the repo, create it, and move the code there. I was >>>>> thinking either `apache/beam-java-template` or `apache/beam-java-starter`. >>>>> What do you think? >>>>> >> >>> >>>>>>>>>> >>>>> >> >>> >>>>>>>>>> What would be the next steps on creating the repo? >>>>> >> >>> >>>>>>>>>> >>>>> >> >>> >>>>>>>>>> On Thu, Dec 9, 2021 at 11:09 AM Ahmet Altay < >>>>> [email protected]> wrote: >>>>> >> >>> >>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>> This is great David. Was there any progress on >>>>> this? Do you need help? >>>>> >> >>> >>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>> On Wed, Dec 1, 2021 at 3:54 PM Brian Hulette < >>>>> [email protected]> wrote: >>>>> >> >>> >>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>> This is cool, thanks! >>>>> >> >>> >>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>> We do have a template in apache/beam already, >>>>> built with Maven Archetype [1]. It's what powers the Java quickstart [2]. >>>>> Could we de-dupe these (e.g. reference the GitHub template in the >>>>> quickstart, or co-locate the archetype with the GitHub template)? >>>>> >> >>> >>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>> As far as creating an Apache repo, would we put >>>>> this somewhere like apache/beam-java-template? I think apache repositories >>>>> like beam-* are allowed. >>>>> >> >>> >>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>> Brian >>>>> >> >>> >>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>> [1] https://maven.apache.org/archetype/index.html >>>>> >> >>> >>>>>>>>>>>> [2] >>>>> https://beam.apache.org/get-started/quickstart-java/#get-the-example-code >>>>> >> >>> >>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>> On Wed, Dec 1, 2021 at 11:30 AM David Cavazos < >>>>> [email protected]> wrote: >>>>> >> >>> >>>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>>> +Ahmet Altay >>>>> >> >>> >>>>>>>>>>>>> +Valentyn Tymofieiev >>>>> >> >>> >>>>>>>>>>>>> +Kenneth Knowles >>>>> >> >>> >>>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>>> Please feel free to include anyone else! >>>>> >> >>> >>>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>>> On Mon, Oct 25, 2021 at 11:31 AM David Cavazos < >>>>> [email protected]> wrote: >>>>> >> >>> >>>>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>>>> Hi Beam community! >>>>> >> >>> >>>>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>>>> To make it easier to create a new Beam Java >>>>> project, I've been working on a GitHub template containing a minimal Beam >>>>> Java pipeline for people to start with. >>>>> >> >>> >>>>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>>>> Link to the GitHub template: >>>>> https://github.com/davidcavazos/beam-java >>>>> >> >>> >>>>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>>>> So far, here's what the template contains: >>>>> >> >>> >>>>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>>>> Minimal "Hello World" Beam pipeline >>>>> >> >>> >>>>>>>>>>>>>> Minimal test file >>>>> >> >>> >>>>>>>>>>>>>> Build files for Gradle, sbt, and Maven (Direct >>>>> runner) >>>>> >> >>> >>>>>>>>>>>>>> Continuous integration via GitHub actions >>>>> (around 1-2 minutes to run) >>>>> >> >>> >>>>>>>>>>>>>> README with instructions on how to build, run, >>>>> test, and add other runners >>>>> >> >>> >>>>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>>>> It's easy to create a new GitHub repo from a >>>>> template. >>>>> >> >>> >>>>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>>>> Next steps >>>>> >> >>> >>>>>>>>>>>>>> >>>>> >> >>> >>>>>>>>>>>>>> Some reviewers to make sure everyone is happy >>>>> with it 🙂 >>>>> >> >>> >>>>>>>>>>>>>> Right now it lives in my personal GitHub >>>>> account, so we need to create an Apache repo to host it >>>>> >> >>> >>>>>>>>>>>>>> Update/create docs with instructions on how to >>>>> create a new Beam Java pipeline >>>>> >>>>
