My phrasing sounded a bit too much like adding a blocking condition. I absolutely don't want to do that. Our maven build was far from clear, and had lots of tech debt and spooky action at a distance,
I just want to emphatically agree with JB's sentiment that we have an opportunity to improve it before it ossifies in the current state. Maybe we won't really have time, and that's OK too and we can just improve as we go. Luke - I know we have different standards of clarity but I doubt we'll have specific disagreements on cleanups to the build as they happen.* I was mostly listing elementary programming concepts that apply to our build and tend to get forgotten in configs & build files. Being "natural gradle" is also important. Let's see how it goes. Kenn *FWIW I did know that we have some string-munged identifiers already, but mostly I am speaking from experience beyond & before Beam too. I'll likely tidy the examples build.gradle as I finish up nexmark in a similar way. On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik <lc...@google.com> wrote: > JB, learning the build system in a project is hopefully avoided by most > users if the contribution guide can clearly explain what users need to do. > But for everyone else who wants to change a dependency version or add a > dependency it should be as simple as copy/paste (which I believe it already > is). Note that people who want to do anything more complicated like add new > jenkins job configurations for new integration test targets needs to learn > how the build system works, how maven profiles work, how to plumb the > required set of attributes from Jenkins into the test target via the build > system. > > Kenn, I have to disagree with a lot in 1 and 2. For users who are > unfamiliar with Maven, we didn't setup the Maven build in such a way that > anyone unfamiliar with Maven knew what was going on or try to use concepts > unnatural to Maven in an attempt to make it seem easier. I believe we > should stick with Gradle/Groovy conventions. Users who are not familiar > with how Gradle/Groovy works will either ask the community or look at > StackOverflow and doing things like passing the project around into > functions is extremely uncommon compared to using the current context and > applying closures to it. For users who are familiar with Gradle, the builds > should be natural Gradle. > > *- If you are going to refer to an identifier, it should have an explicit > point of origin (not strings smashed together in a loop)* > I assume your referring to how the examples java precommit is done. It is > the only case of it to my knowledge and extra hands to migrate it to be > like how the validates runner tests run would be useful. Filed > https://issues.apache.org/jira/browse/BEAM-4033 > > > On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles <k...@google.com> wrote: > >> Huge +1 to the idea of investing in simplification and polish of the >> Gradle files before considering the migration complete. >> >> 1. build.gradle files should be as close to straight-line configuration >> as possible: >> - You should be able to understand a module's build easily, locally, >> without knowing Groovy >> - As little Groovy programming as possible, focused on providing >> well-defined utilities >> >> 2. Explicit > implicit, especially for config files >> - Ditto about being able to understand a module's build locally >> - Minimize globally inherited config, in favor of explicitly calling >> utilities >> - Passing current project to a utility is better than trying to make >> something the "closure" >> - If you are going to refer to an identifier, it should have an explicit >> point of origin (not strings smashed together in a loop) >> >> When in doubt, a little redundancy to improve readability is worth it in >> this context. >> >> Kenn >> >> On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >>> Hi all, >>> >>> I did multiple gradle build since last week and I would like to share >>> one of my concern: it's about the communities. >>> >>> If I think our users won't see any change for them due to Gradle build >>> (I think that most of our users will still use Maven with artifacts >>> provided by Gradle), I'm more concerned by the dev community and the >>> contribution. >>> >>> Maven is well known and straight forward for a large part of potential >>> contributors. I think we have to keep in mind that we still have to grow >>> up our contributors community. >>> >>> Today, maybe I'm wrong, but I have the feeling that gradle build is not >>> straight forward (build.gradle includes build_rules.gradle, gathering >>> all taks all together). >>> >>> I would like to add a task in the gradle "migration" process: simplify >>> the gradle structure and files, and document this. >>> >>> I know we already have a Jira about the documentation part, but I would >>> like to "polish" and use a clean structure for the Gradle resources. As >>> already quickly discussed, I think that having one gradle file per tasks >>> in the .gradle directory would be helpful. >>> >>> The goal is really to simplify the contribution. >>> >>> Do you agree if I add a Jira about "Gradle polish" ? >>> Thoughts ? >>> >>> Regards >>> JB >>> >>> On 07/04/2018 04:52, Scott Wegner wrote: >>> > Here's an end-of-day update on migration work: >>> > >>> > * Snapshot unsigned dailies and signed release builds are working (!!). >>> > PR/5048 [1] merges changes from Luke's branch >>> > * python precommit failing... will investigate python precommit >>> Monday >>> > * All Precommits are gradle only >>> > * All Postcommits except performance tests and Java_JDK_Versions_Test >>> > use gradle (after PR/5047 [2] merged) >>> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be >>> > merged before switching >>> > * ValidatesRunner_Spark failing consistently; investigating >>> > >>> > Thanks for another productive day of hacking. I'll pick up again on >>> Monday. >>> > >>> > [1] https://github.com/apache/beam/pull/5048 >>> > [2] https://github.com/apache/beam/pull/5047 >>> > >>> > >>> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau >>> > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> wrote: >>> > >>> > Why building a zip per runner which its stack and just pointing out >>> > on that zip and let beam lazy load the runner: >>> > >>> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or >>> > the fromSystemProperties() if it gets merged a day ;)) >>> > >>> > Le 6 avr. 2018 20:21, "Kenneth Knowles" <k...@google.com >>> > <mailto:k...@google.com>> a écrit : >>> > >>> > I'm working on finding a solution for launching the Nexmark >>> > suite with each runner. This doesn't have to be done via >>> Gradle, >>> > but we anyhow need built artifacts that don't require user >>> > classpath intervention. >>> > >>> > It looks to me like the examples are also missing this - they >>> > have separate configuration e.g. sparkRunnerPreCommit but that >>> > is overspecified compared to a free-form launching of a main() >>> > program with a runner profile. >>> > >>> > On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik <lc...@google.com >>> > <mailto:lc...@google.com>> wrote: >>> > >>> > Romain, are you talking about the profiles that exist as >>> > part of the archetype examples? >>> > >>> > If so, then those still exist and haven't been changed. If >>> > not, can you provide a link to the profile in a pom file to >>> > be clearer? >>> > >>> > On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau >>> > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> >>> wrote: >>> > >>> > Hi Scott, >>> > >>> > is it right that 2 doesn't handle the hierachy anymore >>> > and that it doesn't handle profiles for runners as it >>> is >>> > currently with maven? >>> > >>> > >>> > Romain Manni-Bucau >>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog >>> > <https://rmannibucau.metawerx.net/> | Old Blog >>> > <http://rmannibucau.wordpress.com> | Github >>> > <https://github.com/rmannibucau> | LinkedIn >>> > <https://www.linkedin.com/in/rmannibucau> | Book >>> > < >>> https://www.packtpub.com/application-development/java-ee-8-high-performance >>> > >>> > >>> > 2018-04-06 18:32 GMT+02:00 Scott Wegner >>> > <sweg...@google.com <mailto:sweg...@google.com>>: >>> > >>> > I wanted to start a thread to summarize the current >>> > state of Gradle migration. We've made lots of good >>> > progress so far this week. Here's the status from >>> > what I can tell-- please add or correct anything I >>> > missed: >>> > >>> > * Release artifacts can be built and published for >>> > Snapshot and officlal releases [1] >>> > * Gradle-generated releases have been validated >>> with >>> > the the Apache Beam archetype generation >>> quickstart; >>> > still needs additional validation. >>> > * Generated release pom files have correct project >>> > metadata [2] >>> > * The python pre-commits are now working in Gradle >>> [3] >>> > * Ismaël has started a collaborative doc of Gradle >>> > tips [4] as we all learn the new system-- please >>> add >>> > your own. This will eventually feed into official >>> > documentation on the website. >>> > * Łukasz Gajowy is working on migrating performance >>> > testing framework [5] >>> > * Daniel is working on updating documentation to >>> > refer to Gradle instead of maven >>> > >>> > If I missed anything, please add it to this thread. >>> > >>> > The general roadmap we're working towards is: >>> > (a) Publish release artifacts with Gradle (SNAPSHOT >>> > and signed releases) >>> > (b) Postcommits migrated to Gradle >>> > (c) Migrate documentation from maven to Gradle >>> > (d) Migrate perfkit suites to use Gradle >>> > >>> > For those of you that are hacking: thanks for your >>> > help so far! Progress is being roughly tracked on >>> > the Kanban [6]; please make sure the issues >>> assigned >>> > to you are up-to-date. Many of the changes are >>> > staged on lukecwik's local branch [7]; we'll work >>> on >>> > merging them back soon. >>> > >>> > >>> > [1] >>> https://github.com/lukecwik/incubator-beam/pull/7 >>> > [2] >>> https://github.com/lukecwik/incubator-beam/pull/3 >>> > [3] https://github.com/apache/beam/pull/5032 >>> > [4] >>> > >>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit >>> > [5] https://github.com/apache/beam/pull/5003 >>> > [6] >>> > >>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242 >>> > >>> > [7] >>> > >>> https://github.com/lukecwik/incubator-beam/tree/gradle >>> > -- >>> > >>> > >>> > Got feedback? http://go/swegner-feedback >>> > >>> > >>> > -- >>> > >>> > >>> > Got feedback? http://go/swegner-feedback >>> >>
