@Reuven: any idea what is missing? I don't expect it to be ready very quickly but having 2 repos does not hurt that much if both are working better than a single one so can be worth a try maybe?
Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> Le mer. 10 oct. 2018 à 08:31, Reuven Lax <re...@google.com> a écrit : > Unrelated to the Maven/Gradle discussion, but I do somewhat agree with > Romain that Beam could be split into separate repos, however I don't think > it's quite ready yet. AFAIK the portability interfaces are still being > modified, and until these interfaces are fixed it will be hard to split the > project. Once those interfaces are fixed and mature, I completely agree we > should then look into splitting the repos up. > > Reuven > > On Tue, Oct 9, 2018 at 10:35 PM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> >> >> >> Le mer. 10 oct. 2018 à 06:35, Kenneth Knowles <k...@apache.org> a écrit : >> >>> Here are some things I hear a lot: >>> >>> - Beam technical decision making is too Java-centric >>> - Gradle is a lot less Java-centric than Maven >>> >>> Hmm isnt it the opposite being groovy based? Maven tends to have the >> same set of plugins than gradle so it is a kind of status-quo if you check >> technically and are factual. >> >> >>> >>> - Gradle is still a Java-centric tool, but at least it isn't so >>> slow/wasteful >>> >>> >> Not what I experienced and it seems JB had the same experience. Gradle is >> comparable to maven in terms of full build and is way slower in Idea cause >> of the lack of integration and support (by design). >> >> >>> I can understand long-time Java devs wanting to have their familiar and >>> dominant toolchain, with good tooling and integrations (ignoring any >>> objective merits of Gradle for Java dev). But for non-Java devs you are >>> meeting in the middle. Actually, you are not even in the middle - you are >>> still in Java land. For example, setup.py can do arbitrary things, so we >>> could use it to do the Java build! What do you think of trying this out? >>> (j/k). >>> >>> What I would like is to better support language-native development >>> workflows, and make sure Gradle is lightweight glue that is easy to use. >>> Make the configurations as obvious to read as we can, with as little >>> hackery as possible. >>> >> >> Agree and think it can be time, now Beam python/go support become >> something, to split in N repos. Release lifecycles are different, codebase >> has no real link except the portability layer which can get its own repo so >> probably worth a PoC: >> >> 1. beam-portability >> 2. beam-java >> 3. beam-go >> 4. beam-python >> >> Side note: beam portability would be saner if added on top of others than >> the opposite which is done today. >> >> >>> >>> ---------- >>> >>> We can make Gradle a lot better (for the above, and for Java too). From >>> my work on the core of our Gradle support code, I can name a few issues >>> that I'm sure remain even after many months away from the code: >>> >>> 1. We were learning as we built it out. We don't always do things in the >>> natural way. >>> 2. We made our own abstractions to streamline converting lots of modules >>> quickly. Clarity and efficiency of the build were not the primary concerns. >>> (to be clear: it was an amazing undertaking to get this done and the >>> decision was right for the moment) >>> 2a. We tried to centralize a lot of policy, leading to the centralized >>> bits containing the union of all complexity of all modules (or maybe it is >>> multiplicative). >>> 3. We tried very hard to match the mvn build exactly, rather than doing >>> the best thing in Gradle. >>> 4. We've built a lot of imperative code for "telling Gradle what to do" >>> and that adds a lot of complexity compared with using Groovy as primarily a >>> configuration language. >>> 5. We have turned on things like "always rebuild" when we don't know the >>> dependencies, rather than putting in the work to get the dependencies right. >>> >>> A lot of the above also makes it hard for IDEs to grok the config, since >>> we deviate from the "golden path" a lot. >>> >>> It would be awesome if module owners took on the task of making their >>> modules have an awesome incremental Gradle build. >>> >> >> Well incremental build only matters when you run a full subpart of a job >> and is something very fragile - check how in months it didnt happen. In >> practise what's a dev workflow: >> >> 1. loop { dev in the IDE, run test } >> 2. run the full module or build to validate nothing has been broken >> 3. PR (+ back to 1 if comments) >> >> This means that incremental support is only relevant for jenkins where >> the perf diff is not significative and where you can't use incremental >> build cause you want to have a fully reliable build. >> So at the end the incremental build support is not that significative for >> end users and contibutors. The cost of not having the IDE support, however, >> is just a blocker. >> >> So my 2cts would be to stop trying to be good theorically at the cost of >> loosing the users and try to embrace a community driven choices approach. >> >> >>> >>> Kenn >>> >>> On Tue, Oct 9, 2018 at 3:38 AM Romain Manni-Bucau <rmannibu...@gmail.com> >>> wrote: >>> >>>> For me the vendoring issue is ok cause it should belong to another >>>> shade loduke released with beam when needed. It is not an uncommon >>>> practise. >>>> >>>> Now the lack of IDE integration for tests/debug (using gradle runner is >>>> a workaround and still hurts by its slowness compared to native run) is a >>>> clear showstopper for me. >>>> >>>> Also, from a community perspective, gradle adoption is far to be >>>> mainstream (even spark is built with maven) so does not serve beam at the >>>> end. >>>> >>>> Maven build didnt have any issue except the duration AFAIK, gradle has >>>> 2 blockers + several small drawbacks (custom build and no standard, no >>>> tooling without script execution, bad integration in enterprise chaines >>>> like security auditing etc). Overall gradle build is close to maven one - >>>> last time i tested it was within 15% so not worth it when you see the time >>>> you loose when developping anything. It is key to keep in mind jenkiks is >>>> cheaper than human time. >>>> >>>> Le mar. 9 oct. 2018 13:22, Robert Bradshaw <rober...@google.com> a >>>> écrit : >>>> >>>>> On Tue, Oct 9, 2018 at 10:04 AM Jean-Baptiste Onofré <j...@nanthrax.net> >>>>> wrote: >>>>> >>>>>> Hi guys, >>>>>> >>>>>> I know that's a hot topic, but I have to bring this discussion on the >>>>>> table. >>>>>> >>>>> >>>>> Thank you for bringing this up and revisiting it now that we have some >>>>> experience. >>>>> >>>>> >>>>>> Some months ago, we discussed about migrating our build from Maven to >>>>>> Gradle. One of the key expected improvement was the time to build. >>>>>> We proposed to do a PoC to evaluate the impacts and improvements, but >>>>>> this PoC was actually directly a migrate on master. >>>>>> >>>>>> Now, I would like to bring facts here: >>>>>> >>>>>> 1. Build time >>>>>> On my machine, the build time is roughly 1h15. It's pretty long, and >>>>>> regarding what the build is doing, I don't see huge improvement >>>>>> provided >>>>>> by Gradle. >>>>>> >>>>> >>>>> I rarely, if ever, build from scratch so perhaps I have not been >>>>> impacted by this nearly as much. (In particular, build and test times seem >>>>> to have gone way down for me, probably due to better incremental support, >>>>> but that's just anecdotal.) >>>>> >>>>> Is this worse than it was on maven, or just not as much better as was >>>>> hoped? >>>>> >>>>> >>>>>> 2. Build reliability >>>>>> Even worse, most of the time, we need to use --no-parallel and >>>>>> --no-daemon to have a reliable build (it's basically recommended for >>>>>> release). It has an impact on build time, and we loose part of Gradle >>>>>> benefits. >>>>>> >>>>> >>>>> I think this is a matter of incorrect dependency declarations (and is >>>>> not unique to gradle). I'd have loved to been able to go with a build >>>>> system that simply didn't let you have incorrect dependency declarations, >>>>> but that wasn't an option for other reasons. >>>>> >>>>> I wonder if there's some automatic tooling we could leverage to fix >>>>> (and keep fixed) this. Regardless, this is unfinished work that remains to >>>>> be done so we can realize the full benefits. >>>>> >>>>> >>>>>> 3. Release and repositories >>>>>> Even if couple of releases has been performed with Gradle, it's not >>>>>> obvious to see improvements around artifacts handling. I got my >>>>>> repository polluted twice (that's part of the trick Gradle is doing to >>>>>> speed up the build dealing around the repository). >>>>>> >>>>> >>>>> Could you clarify what improvements we were expecting here? I thought >>>>> the goal was that we could publish the same artifacts, with no regression. >>>>> >>>>> >>>>>> 4. IDE integration >>>>>> We already had some comments on the mailing lists about the IDE >>>>>> integration. Clearly, the situation is not good on that front too. The >>>>>> integration on IDE (especially IntelliJ) is not good enough right now. >>>>>> >>>>> >>>>> This is important. To be honest, I had also issues back in the day >>>>> getting the maven setup working well out of the box in IntelliJ and >>>>> Eclipse >>>>> (mostly with respect to things like shadowing and protobufs), so we >>>>> shouldn't fall prey to the golden age fallacy. >>>>> >>>>> It seems the recent move to vendoring has caused more issues here; I'm >>>>> not sure that would be fixed just moving back to maven (or how to resolve >>>>> it going forward). >>>>> >>>>> On the other hand, just last week I set up a new computer according to >>>>> https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips and >>>>> that seems to be working fine. >>>>> >>>>> >>>>>> We are working hard to grow up the community, and from a contributor >>>>>> perspective, our build system is not good today IMHO. >>>>>> As a contributor, I resumed my work on some PRs, and I'm spending so >>>>>> much time of the build, largely more than working on the PRs code >>>>>> itself. >>>>>> >>>>>> So, obviously, the situation is not perfect, at least from a >>>>>> contributor >>>>>> perspective. >>>>>> >>>>>> The purpose of this thread is not again to have a bunch of replied >>>>>> ending nowhere. I would like to be more "pushy" and let's try to be >>>>>> concrete. So basically, we only have two options: >>>>>> >>>>>> 1. Improve the build, working hard on Gradle front. Not sure if it >>>>>> makes >>>>>> such sense from a contributor perspective, as Maven is really well >>>>>> known >>>>>> from most of contributors (and easier to start with IMHO). >>>>>> 2. Back on Maven. That's clearly my preferred approach. IDE >>>>>> integration >>>>>> is better, Maven is well known from the contributors as already said. >>>>>> The effort is not so huge. We tried to use Gradle, we don't have the >>>>>> expected results now, that's not a problem, it's part of a project >>>>>> lifetime. >>>>>> >>>>>> Thoughts ? >>>>>> >>>>> >>>>> I'd like to add some perspective as a primarily non-Java contributor >>>>> (having contributed "only" a couple thousand lines to the java side over >>>>> the last couple of years) that maven feels much more java-centric (as is >>>>> our repo, but that's a separate issue). Also, as someone not coming from >>>>> with years of maven (or gradle for that matter) experience before working >>>>> on Beam, I have found gradle much more intuitive and easier to learn. >>>>> Especially when it comes to developing changes that span multiple modules >>>>> (which for some reason I guess I tend to do a lot of, but with so many of >>>>> our core sdk tests being validates runner that's likely to hit people just >>>>> starting out as well). Now of course I don't want to discount the Java >>>>> community, indeed it's arguably the largest and most important one at this >>>>> point, but I also think that Beam's ability to not be limited to that one >>>>> language (and ecosystem) is one of its huge selling points and >>>>> differentiation (see the whole portability effort, which is for both SDKs >>>>> and Runners). >>>>> >>>>> Even from a Java perspective, neither is so obscure as to be a >>>>> significant barrier, and both are customizable enough that the average new >>>>> developer is probably going to be looking at the documentation to see what >>>>> commands to run to build/test/validate this project. So this is probably >>>>> something documentation can address. >>>>> >>>>> But if we're going to reap the benefits and minimize the downsides of >>>>> this migration, we have to finish the job. >>>>> >>>>> - Robert >>>>> >>>>> >>>>>
