+1 to starters for Kotlin and Scala too. Scala at least as sbt as the idiomatic build tool (as of like 10 years ago...) so a fully idiomatic repo would look different and be helpful.
On Tue, Jan 25, 2022 at 10:54 AM David Cavazos <[email protected]> wrote: > We could also have Kotlin and Scala starter projects. I was already > experimenting with those. It's low hanging fruit since they use the same > Java SDK, and many people are already using and liking Kotlin (from > Android) and Scala (from Spark). > > On Tue, Jan 25, 2022 at 10:52 AM David Cavazos <[email protected]> > wrote: > >> I can change the license to ASL2, no problem. I might have copied and >> pasted it from somewhere else. >> >> As for Python and Go, I think the main advantage would be having a >> testing infrastructure setup including GitHub actions configuration. But >> other than that I agree that the actual Beam setup would be pretty minimal. >> But I think the main advantage is that the quickstarts for all >> languages would be more consistent: git clone and then run. >> >> On Tue, Jan 18, 2022 at 5:25 PM Robert Burke <[email protected]> wrote: >> >>> Can confirm that Go would be very minimal as well. But I agree there's >>> value in users not needing to start entirely from scratch. Some users will >>> find it easier to mutate and expand to what you want vs writing it from a >>> blank page, even if the boiler plate is negligible. >>> >>> Most Go tooling is pretty good at doing package lookups and similar, but >>> only after the a module has been loaded into your cache. >>> >>> On Tue, Jan 18, 2022 at 6:20 AM Kenneth Knowles <[email protected]> wrote: >>> >>>> I want to clarify one thing: I am not certain the requirement of ASL2 >>>> applies to example code snippets. I am also not sure if it makes a material >>>> difference to users. I _am_ sure we would need to deal with some process to >>>> use something other than ASL2, so I'd rather not. >>>> >>>> Kenn >>>> >>>> On Tue, Jan 18, 2022 at 6:17 AM Kenneth Knowles <[email protected]> >>>> wrote: >>>> >>>>> Agree with Luke here. "Just git clone and go" is a big part of it. >>>>> >>>>> But also the answer to "I simply don't know what one would put in a >>>>> Python repo than, other than a bare setup.py that lists a dependency on >>>>> apache_beam" is answered by David's initial email and his repo, namely: >>>>> >>>>> - GitHub Actions configuration >>>>> - README.md >>>>> - example that already runs >>>>> - LICENSE (notably you've got it as MIT but to be part of Apache >>>>> software it needs to be ASL2) >>>>> >>>>> Kenn >>>>> >>>>> On Fri, Jan 14, 2022 at 2:34 PM Luke Cwik <[email protected]> wrote: >>>>> >>>>>> I think for consistency it makes sense to users to be told to >>>>>> checkout this git repo for the language of your choice and run. Some >>>>>> repos >>>>>> will have more/less than others when it comes to setup necessary. >>>>>> >>>>>> On Fri, Jan 14, 2022 at 2:26 PM Robert Bradshaw <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> +1 for doing this for Java, as setting up a project there is quite >>>>>>> complicated. I simply don't know what one would put in a Python repo >>>>>>> than, other than a bare setup.py that lists a dependency on >>>>>>> apache_beam. We don't have recommendations on file layout, etc. more >>>>>>> than that (though there's plenty of generic advice to be found out >>>>>>> there on the topic). I have a hunch go is similar, and javascript >>>>>>> would be as well (npm install apache-beam and your package.json file >>>>>>> gets updated). >>>>>>> >>>>>>> On Fri, Jan 14, 2022 at 2:17 PM Luke Cwik <[email protected]> wrote: >>>>>>> > >>>>>>> > There are several examples already within the Beam repo found in: >>>>>>> > https://github.com/apache/beam/tree/master/examples >>>>>>> > https://github.com/apache/beam/tree/master/sdks/go/examples >>>>>>> > >>>>>>> https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples >>>>>>> > >>>>>>> > >>>>>>> > On Fri, Jan 14, 2022 at 11:07 AM Sachin Agarwal < >>>>>>> [email protected]> wrote: >>>>>>> >> >>>>>>> >> I'd love to do something other than Wordcount just for >>>>>>> novelty/freshness but agreed with the suggestion that having an example >>>>>>> in >>>>>>> each quickstart would be ideal. >>>>>>> >> >>>>>>> >> On Fri, Jan 14, 2022 at 11:06 AM David Huntsperger < >>>>>>> [email protected]> wrote: >>>>>>> >>> >>>>>>> >>> + 1 to a separate repo for each language. >>>>>>> >>> >>>>>>> >>> Would it make sense to include the Wordcount example in each >>>>>>> repo? I know that makes the repos less minimal, but we could rewrite the >>>>>>> quickstarts around these repos instead of the current Wordcount >>>>>>> examples. >>>>>>> Or maybe we don't need to use the Wordcount example in the >>>>>>> quickstarts... >>>>>>> >>> >>>>>>> >>> On Wed, Jan 12, 2022 at 1:54 PM David Cavazos < >>>>>>> [email protected]> wrote: >>>>>>> >>>> >>>>>>> >>>> I agree with dropping the archetypes. Less maintenance is >>>>>>> preferable, and the github repos are more flexible and maintainable. >>>>>>> >>>> >>>>>>> >>>> How about we create: >>>>>>> >>>> >>>>>>> >>>> apache/beam-starter-java >>>>>>> >>>> apache/beam-starter-python >>>>>>> >>>> apache/beam-starter-go >>>>>>> >>>> >>>>>>> >>>> During our OKR planning, +Keith Malvetti would prefer having >>>>>>> repos for all languages. It makes sense for consistency as well. >>>>>>> >>>> >>>>>>> >>>> On Mon, Jan 10, 2022 at 5:14 PM Luke Cwik <[email protected]> >>>>>>> wrote: >>>>>>> >>>>> >>>>>>> >>>>> As long as we have tags so that people can pull out a specific >>>>>>> version of the examples that coincides with a specific SDK version then >>>>>>> we >>>>>>> could drop the archetypes. >>>>>>> >>>>> >>>>>>> >>>>> On Mon, Jan 10, 2022 at 4:09 PM Brian Hulette < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>> >>>>>>> >>>>>> > Being such minimal examples, I don't expect them to break >>>>>>> commonly, but I think it would be good to make sure tests aren't failing >>>>>>> when a release is published. >>>>>>> >>>>>> >>>>>>> >>>>>> Yeah it would be very unfortunate if we discovered a breakage >>>>>>> after the release. Agree we should verify RCs (document as part of the >>>>>>> release process), or even better, add automation to verify the repo >>>>>>> against >>>>>>> snapshots. The automation could be nice to have anyway since it >>>>>>> provides an >>>>>>> example for users to follow if they want to test against snapshots and >>>>>>> report issues to us sooner. >>>>>>> >>>>>> >>>>>>> >>>>>> >>>>>>> >>>>>> If we move forward with this can we drop the archetype? >>>>>>> >>>>>> >>>>>>> >>>>>> On Fri, Jan 7, 2022 at 3:54 PM Luke Cwik <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Sounds reasonable. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jan 5, 2022 at 12:47 PM David Cavazos < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> I personally like the idea of a separate repo since we can >>>>>>> see how a true minimal project looks like. Having it in the main repo >>>>>>> would >>>>>>> inherit build file configurations and other settings that would be >>>>>>> different from a clean project, so it could be non-trivial to adapt. >>>>>>> Also >>>>>>> as its own repo, it's easier to clone and modify, or create an instance >>>>>>> of >>>>>>> the template. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> Dependabot can take care of updating the Beam version and >>>>>>> other dependencies automatically. Testing is already set up via GitHub >>>>>>> actions for every pull request, so it would automatically be tested as >>>>>>> soon >>>>>>> as there is a new dependency version available. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> Being such minimal examples, I don't expect them to break >>>>>>> commonly, but I think it would be good to make sure tests aren't failing >>>>>>> when a release is published. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> I'm okay with having one repo per language, and having all >>>>>>> the build systems we want to support for them. As long as we document >>>>>>> which >>>>>>> files are for which build system. That way there are less repos to >>>>>>> maintain. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> On Mon, Dec 13, 2021 at 9:25 AM Luke Cwik <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> The github repo is definitely more flexible then the >>>>>>> archetypes but the archetypes have a few conveniences since they are >>>>>>> integrated with apache/beam repo. For example, updates/testing are done >>>>>>> at >>>>>>> the same time a corresponding change to the main repo is done (like >>>>>>> library >>>>>>> version updates), they are released when the SDK is released. >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> Should these be part of the main repo, or a single starter >>>>>>> repo containing all the starters or one per language or one per build >>>>>>> system? >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> When should updates to the starter happen? >>>>>>> >>>>>>>>> How as a community do we get them to happen (e.g. release >>>>>>> manager owns it)? >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> On Sun, Dec 12, 2021 at 4:06 PM David Cavazos < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> We could do the Maven archetype, but that wouldn't work >>>>>>> very well for Gradle and SBT users. I think a GitHub template might be >>>>>>> the >>>>>>> more flexible option, and we could have something similar for other >>>>>>> languages as well. Having said that, we could still create a Maven >>>>>>> archetype. If someone is familiar with that process, please let me know >>>>>>> since I'm not too familiar with Maven and its ecosystem. >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> @Ahmet Altay I think right now we only need to pin down >>>>>>> the name of the repo, create it, and move the code there. I was thinking >>>>>>> either `apache/beam-java-template` or `apache/beam-java-starter`. What >>>>>>> do >>>>>>> you think? >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> What would be the next steps on creating the repo? >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> On Thu, Dec 9, 2021 at 11:09 AM Ahmet Altay < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>>>>>> This is great David. Was there any progress on this? Do >>>>>>> you need help? >>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>>>>>> On Wed, Dec 1, 2021 at 3:54 PM Brian Hulette < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> This is cool, thanks! >>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> We do have a template in apache/beam already, built >>>>>>> with Maven Archetype [1]. It's what powers the Java quickstart [2]. >>>>>>> Could >>>>>>> we de-dupe these (e.g. reference the GitHub template in the quickstart, >>>>>>> or >>>>>>> co-locate the archetype with the GitHub template)? >>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> As far as creating an Apache repo, would we put this >>>>>>> somewhere like apache/beam-java-template? I think apache repositories >>>>>>> like >>>>>>> beam-* are allowed. >>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> Brian >>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> [1] https://maven.apache.org/archetype/index.html >>>>>>> >>>>>>>>>>>> [2] >>>>>>> https://beam.apache.org/get-started/quickstart-java/#get-the-example-code >>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> On Wed, Dec 1, 2021 at 11:30 AM David Cavazos < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>> +Ahmet Altay >>>>>>> >>>>>>>>>>>>> +Valentyn Tymofieiev >>>>>>> >>>>>>>>>>>>> +Kenneth Knowles >>>>>>> >>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>> Please feel free to include anyone else! >>>>>>> >>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>> On Mon, Oct 25, 2021 at 11:31 AM David Cavazos < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> Hi Beam community! >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> To make it easier to create a new Beam Java project, >>>>>>> I've been working on a GitHub template containing a minimal Beam Java >>>>>>> pipeline for people to start with. >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> Link to the GitHub template: >>>>>>> https://github.com/davidcavazos/beam-java >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> So far, here's what the template contains: >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> Minimal "Hello World" Beam pipeline >>>>>>> >>>>>>>>>>>>>> Minimal test file >>>>>>> >>>>>>>>>>>>>> Build files for Gradle, sbt, and Maven (Direct runner) >>>>>>> >>>>>>>>>>>>>> Continuous integration via GitHub actions (around 1-2 >>>>>>> minutes to run) >>>>>>> >>>>>>>>>>>>>> README with instructions on how to build, run, test, >>>>>>> and add other runners >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> It's easy to create a new GitHub repo from a template. >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> Next steps >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> Some reviewers to make sure everyone is happy with it >>>>>>> 🙂 >>>>>>> >>>>>>>>>>>>>> Right now it lives in my personal GitHub account, so >>>>>>> we need to create an Apache repo to host it >>>>>>> >>>>>>>>>>>>>> Update/create docs with instructions on how to create >>>>>>> a new Beam Java pipeline >>>>>>> >>>>>>
