Le 9 avr. 2018 21:04, "Lukasz Cwik" <[email protected]> a écrit :
Romain, can you clarify by the weird task naming? (Give some examples using our current project and Gradle and what you would have expected.) Sure https://github.com/apache/beam/blob/fb6ba3bfd43605a0f4944828b5f19e75840fa7aa/examples/java/build.gradle#L78 for instance It is way more natural to have one generic command you can select a runner (including a csv value) than N commands for this kind of things. Gradle allows to define custom tasks and "scopes". This must not be used at Xtrem until justifiable. We now have a lot of configurations not making much sense too until you scripted yourself the script or know beam well. This doesnt enable contribs :(. Hope it is clearer On Mon, Apr 9, 2018 at 2:49 PM Romain Manni-Bucau <[email protected]> wrote: > +1 gradle not being mainstream - and even more in beam scope than other > java scope - it must stay simple. I know that personally it takes me x2 or > 3 to say "ok ill fork" a project using gradle, in particular with custom > scripts and not only parent/module scripts. > > If needed we can write custom plugins to simply it and hide the complexity > and weird task naming can move to "standard" names with options/parameters > maybe? > > Le 9 avr. 2018 20:27, "Kenneth Knowles" <[email protected]> a écrit : > >> My phrasing sounded a bit too much like adding a blocking condition. I >> absolutely don't want to do that. Our maven build was far from clear, and >> had lots of tech debt and spooky action at a distance, >> >> I just want to emphatically agree with JB's sentiment that we have an >> opportunity to improve it before it ossifies in the current state. Maybe we >> won't really have time, and that's OK too and we can just improve as we go. >> >> Luke - I know we have different standards of clarity but I doubt we'll >> have specific disagreements on cleanups to the build as they happen.* I was >> mostly listing elementary programming concepts that apply to our build and >> tend to get forgotten in configs & build files. Being "natural gradle" is >> also important. Let's see how it goes. >> >> Kenn >> >> *FWIW I did know that we have some string-munged identifiers already, but >> mostly I am speaking from experience beyond & before Beam too. I'll likely >> tidy the examples build.gradle as I finish up nexmark in a similar way. >> >> On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik <[email protected]> wrote: >> >>> JB, learning the build system in a project is hopefully avoided by most >>> users if the contribution guide can clearly explain what users need to do. >>> But for everyone else who wants to change a dependency version or add a >>> dependency it should be as simple as copy/paste (which I believe it already >>> is). Note that people who want to do anything more complicated like add new >>> jenkins job configurations for new integration test targets needs to learn >>> how the build system works, how maven profiles work, how to plumb the >>> required set of attributes from Jenkins into the test target via the build >>> system. >>> >>> Kenn, I have to disagree with a lot in 1 and 2. For users who are >>> unfamiliar with Maven, we didn't setup the Maven build in such a way that >>> anyone unfamiliar with Maven knew what was going on or try to use concepts >>> unnatural to Maven in an attempt to make it seem easier. I believe we >>> should stick with Gradle/Groovy conventions. Users who are not familiar >>> with how Gradle/Groovy works will either ask the community or look at >>> StackOverflow and doing things like passing the project around into >>> functions is extremely uncommon compared to using the current context and >>> applying closures to it. For users who are familiar with Gradle, the builds >>> should be natural Gradle. >>> >>> *- If you are going to refer to an identifier, it should have an >>> explicit point of origin (not strings smashed together in a loop)* >>> I assume your referring to how the examples java precommit is done. It >>> is the only case of it to my knowledge and extra hands to migrate it to be >>> like how the validates runner tests run would be useful. Filed >>> https://issues.apache.org/jira/browse/BEAM-4033 >>> >>> >>> On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles <[email protected]> wrote: >>> >>>> Huge +1 to the idea of investing in simplification and polish of the >>>> Gradle files before considering the migration complete. >>>> >>>> 1. build.gradle files should be as close to straight-line configuration >>>> as possible: >>>> - You should be able to understand a module's build easily, locally, >>>> without knowing Groovy >>>> - As little Groovy programming as possible, focused on providing >>>> well-defined utilities >>>> >>>> 2. Explicit > implicit, especially for config files >>>> - Ditto about being able to understand a module's build locally >>>> - Minimize globally inherited config, in favor of explicitly calling >>>> utilities >>>> - Passing current project to a utility is better than trying to make >>>> something the "closure" >>>> - If you are going to refer to an identifier, it should have an >>>> explicit point of origin (not strings smashed together in a loop) >>>> >>>> When in doubt, a little redundancy to improve readability is worth it >>>> in this context. >>>> >>>> Kenn >>>> >>>> On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I did multiple gradle build since last week and I would like to share >>>>> one of my concern: it's about the communities. >>>>> >>>>> If I think our users won't see any change for them due to Gradle build >>>>> (I think that most of our users will still use Maven with artifacts >>>>> provided by Gradle), I'm more concerned by the dev community and the >>>>> contribution. >>>>> >>>>> Maven is well known and straight forward for a large part of potential >>>>> contributors. I think we have to keep in mind that we still have to >>>>> grow >>>>> up our contributors community. >>>>> >>>>> Today, maybe I'm wrong, but I have the feeling that gradle build is not >>>>> straight forward (build.gradle includes build_rules.gradle, gathering >>>>> all taks all together). >>>>> >>>>> I would like to add a task in the gradle "migration" process: simplify >>>>> the gradle structure and files, and document this. >>>>> >>>>> I know we already have a Jira about the documentation part, but I would >>>>> like to "polish" and use a clean structure for the Gradle resources. As >>>>> already quickly discussed, I think that having one gradle file per >>>>> tasks >>>>> in the .gradle directory would be helpful. >>>>> >>>>> The goal is really to simplify the contribution. >>>>> >>>>> Do you agree if I add a Jira about "Gradle polish" ? >>>>> Thoughts ? >>>>> >>>>> Regards >>>>> JB >>>>> >>>>> On 07/04/2018 04:52, Scott Wegner wrote: >>>>> > Here's an end-of-day update on migration work: >>>>> > >>>>> > * Snapshot unsigned dailies and signed release builds are working >>>>> (!!). >>>>> > PR/5048 [1] merges changes from Luke's branch >>>>> > * python precommit failing... will investigate python precommit >>>>> Monday >>>>> > * All Precommits are gradle only >>>>> > * All Postcommits except performance tests and Java_JDK_Versions_Test >>>>> > use gradle (after PR/5047 [2] merged) >>>>> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be >>>>> > merged before switching >>>>> > * ValidatesRunner_Spark failing consistently; investigating >>>>> > >>>>> > Thanks for another productive day of hacking. I'll pick up again on >>>>> Monday. >>>>> > >>>>> > [1] https://github.com/apache/beam/pull/5048 >>>>> > [2] https://github.com/apache/beam/pull/5047 >>>>> > >>>>> > >>>>> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau >>>>> > <[email protected] <mailto:[email protected]>> wrote: >>>>> > >>>>> > Why building a zip per runner which its stack and just pointing >>>>> out >>>>> > on that zip and let beam lazy load the runner: >>>>> > >>>>> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... >>>>> (or >>>>> > the fromSystemProperties() if it gets merged a day ;)) >>>>> > >>>>> > Le 6 avr. 2018 20:21, "Kenneth Knowles" <[email protected] >>>>> > <mailto:[email protected]>> a écrit : >>>>> > >>>>> > I'm working on finding a solution for launching the Nexmark >>>>> > suite with each runner. This doesn't have to be done via >>>>> Gradle, >>>>> > but we anyhow need built artifacts that don't require user >>>>> > classpath intervention. >>>>> > >>>>> > It looks to me like the examples are also missing this - they >>>>> > have separate configuration e.g. sparkRunnerPreCommit but >>>>> that >>>>> > is overspecified compared to a free-form launching of a >>>>> main() >>>>> > program with a runner profile. >>>>> > >>>>> > On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik < >>>>> [email protected] >>>>> > <mailto:[email protected]>> wrote: >>>>> > >>>>> > Romain, are you talking about the profiles that exist as >>>>> > part of the archetype examples? >>>>> > >>>>> > If so, then those still exist and haven't been changed. >>>>> If >>>>> > not, can you provide a link to the profile in a pom file >>>>> to >>>>> > be clearer? >>>>> > >>>>> > On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau >>>>> > <[email protected] <mailto:[email protected]>> >>>>> wrote: >>>>> > >>>>> > Hi Scott, >>>>> > >>>>> > is it right that 2 doesn't handle the hierachy >>>>> anymore >>>>> > and that it doesn't handle profiles for runners as >>>>> it is >>>>> > currently with maven? >>>>> > >>>>> > >>>>> > Romain Manni-Bucau >>>>> > @rmannibucau <https://twitter.com/rmannibucau> | >>>>> Blog >>>>> > <https://rmannibucau.metawerx.net/> | Old Blog >>>>> > <http://rmannibucau.wordpress.com> | Github >>>>> > <https://github.com/rmannibucau> | LinkedIn >>>>> > <https://www.linkedin.com/in/rmannibucau> | Book >>>>> > <https://www.packtpub.com/ >>>>> application-development/java-ee-8-high-performance> >>>>> > >>>>> > 2018-04-06 18:32 GMT+02:00 Scott Wegner >>>>> > <[email protected] <mailto:[email protected]>>: >>>>> > >>>>> > I wanted to start a thread to summarize the >>>>> current >>>>> > state of Gradle migration. We've made lots of >>>>> good >>>>> > progress so far this week. Here's the status from >>>>> > what I can tell-- please add or correct anything >>>>> I >>>>> > missed: >>>>> > >>>>> > * Release artifacts can be built and published >>>>> for >>>>> > Snapshot and officlal releases [1] >>>>> > * Gradle-generated releases have been validated >>>>> with >>>>> > the the Apache Beam archetype generation >>>>> quickstart; >>>>> > still needs additional validation. >>>>> > * Generated release pom files have correct >>>>> project >>>>> > metadata [2] >>>>> > * The python pre-commits are now working in >>>>> Gradle [3] >>>>> > * Ismaël has started a collaborative doc of >>>>> Gradle >>>>> > tips [4] as we all learn the new system-- please >>>>> add >>>>> > your own. This will eventually feed into official >>>>> > documentation on the website. >>>>> > * Łukasz Gajowy is working on migrating >>>>> performance >>>>> > testing framework [5] >>>>> > * Daniel is working on updating documentation to >>>>> > refer to Gradle instead of maven >>>>> > >>>>> > If I missed anything, please add it to this >>>>> thread. >>>>> > >>>>> > The general roadmap we're working towards is: >>>>> > (a) Publish release artifacts with Gradle >>>>> (SNAPSHOT >>>>> > and signed releases) >>>>> > (b) Postcommits migrated to Gradle >>>>> > (c) Migrate documentation from maven to Gradle >>>>> > (d) Migrate perfkit suites to use Gradle >>>>> > >>>>> > For those of you that are hacking: thanks for >>>>> your >>>>> > help so far! Progress is being roughly tracked on >>>>> > the Kanban [6]; please make sure the issues >>>>> assigned >>>>> > to you are up-to-date. Many of the changes are >>>>> > staged on lukecwik's local branch [7]; we'll >>>>> work on >>>>> > merging them back soon. >>>>> > >>>>> > >>>>> > [1] https://github.com/lukecwik/ >>>>> incubator-beam/pull/7 >>>>> > [2] https://github.com/lukecwik/ >>>>> incubator-beam/pull/3 >>>>> > [3] https://github.com/apache/beam/pull/5032 >>>>> > [4] >>>>> > https://docs.google.com/document/d/ >>>>> 1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit >>>>> > [5] https://github.com/apache/beam/pull/5003 >>>>> > [6] >>>>> > https://issues.apache.org/ >>>>> jira/secure/RapidBoard.jspa?rapidView=242 >>>>> > >>>>> > [7] >>>>> > https://github.com/lukecwik/ >>>>> incubator-beam/tree/gradle >>>>> > -- >>>>> > >>>>> > >>>>> > Got feedback? http://go/swegner-feedback >>>>> > >>>>> > >>>>> > -- >>>>> > >>>>> > >>>>> > Got feedback? http://go/swegner-feedback >>>>> >>>>
