FWIW, hacked something as showcase for BEAM-4046 [1] This is miserably broken, but a
./gradlew projects or ./gradlew -p sdks/java build should work. Anything else is likely to cause issues. If u hit stack overflow exception, it's likely caused by https://github.com/gradle/gradle/issues/847 To continue here, lots of cleanup has to be done. We might also need to rename folders etc, do better reflect semantic intentions. [1] https://github.com/apache/beam/pull/8194 On Mon, Apr 1, 2019 at 11:56 PM Kenneth Knowles <[email protected]> wrote: > > > On Mon, Apr 1, 2019 at 2:20 PM Lukasz Cwik <[email protected]> wrote: > >> >> >> On Mon, Apr 1, 2019 at 2:00 PM Kenneth Knowles <[email protected]> wrote: >> >>> >>> As to building an aggregated "Java" project, I think the blocker will be >>> supporting conflicting deps. For IOs like ElasticSearch and runners like >>> Flink the conflict is essential and deliberate, to support multiple >>> versions of other services. And that is not even talking about transitive >>> dep conflicts. I think Python and Go don't have this issue simply because >>> they haven't tackled those problems. >>> >>> Are you talking about just a shortcut for building (super easy to just >>> add since we are using Gradle) or a new artifact that you want to >>> distribute? >>> >>> On Mon, Apr 1, 2019 at 10:01 AM Lukasz Cwik <[email protected]> wrote: >>> >>>> During the gradle migration, we used to have something like: >>>> >>>> include(":sdks:java:core") >>>> include(":sdks:java:extensions:sql") >>>> include(":sdks:python") >>>> >>>> Just to be super clear, this is Gradle default and is equivalent to >>> just leaving it blank. >>> >>> >>>> but we discovered the Maven module names that were used during >>>> publishing were "core" / "sql" / ... (effectively the directory name) >>>> instead of "beam-sdks-java-core". >>>> >>> >>> Isn't this managed by the publication plugin? >>> https://docs.gradle.org/current/userguide/publishing_maven.html#sec:identity_values_in_the_generated_pom >>> "overriding >>> the default identity values is easy: simply specify the groupId, artifactId >>> or version attributes when configuring the MavenPublication." >>> >> >> During the gradle migration this wasn't that easy. The new maven publish >> plugin improved a lot since then. >> >> >>> Using the default at the time also broke the artifact names for intra >>>> project dependencies that we generate[1]. Finally, we also ran into an >>>> issue because we had more then one Gradle project with the same directory >>>> name even though they were under a different parent folder (I think it was >>>> "core") and that was leading to some strange build time behavior. >>>> >>> >>> Weird. But I think the Jira should still stand as a move towards >>> simplifying our build and making it more discoverable for new contributors. >>> >> >> Agree on the JIRA makes sense, just calling out that there were other >> issues that this naming had caused in the past which should be checked >> before we call this done. >> > > Totally agree. It will be quite a large task with a lot of boilerplate > that might not be separable from technical blockers that come up as you go > through the boilerplate. > > Kenn > > > > Kenn >>> >>> >>>> We didn't migrate to a flat project structure where each project is a >>>> folder underneath the root project because of the existing Maven build >>>> rules that were being maintained in parallel and I'm not sure if people >>>> would want to have a flat project structure either. >>>> >>>> 1: >>>> https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1055 >>>> >>>> On Mon, Apr 1, 2019 at 9:49 AM Michael Luckey <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> although I did not yet manage to get deeper involved into actual >>>>> development, I think this ability would be a useful addition. >>>>> >>>>> But I would also like to point out, that this is kind of implicit, as >>>>> soon we get https://issues.apache.org/jira/browse/BEAM-4046 included. >>>>> >>>>> For instance, we would change the current setup from >>>>> >>>>> include "beam-sdks-java-core" >>>>> project(":beam-sdks-java-core").dir = file("sdks/java/core") >>>>> >>>>> to something like >>>>> >>>>> include(":sdks:java:core") >>>>> include(":sdks:java:extensions:sql") >>>>> include(":sdks:python") >>>>> >>>>> >>>>> With this in place a plain >>>>> >>>>> $ ./gradlew -p sdks/java build >>>>> >>>>> would exactly do what you want. And, of course, this will also work >>>>> for 'sdks/java/io', 'runners/' etc. Hope, you get the point. >>>>> >>>>> Currently, we deviate from gradle default convention and therefore >>>>> have to implement some quirks to restore default behaviour. And I somehow >>>>> dislike the structure introduced by parent/child folders, which will be >>>>> destroyed by our current project definitions. >>>>> >>>>> But, to be honest, although I have some clear understanding on how to >>>>> proceed here - especially regarding the requirement to keep the change >>>>> backwards compatible - we might decide not to switch. Because deeper >>>>> investigation might reveal issues, which I am currently not aware of. >>>>> >>>>> Best, >>>>> >>>>> michel >>>>> >>>>> On Mon, Apr 1, 2019 at 5:52 PM Jean-Baptiste Onofré <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi guys, >>>>>> >>>>>> I would like to introduce a Gradle "meta" project for the build: >>>>>> beam-sdks-java. >>>>>> >>>>>> The idea is to simply build all Java SDK related resources (core, IO, >>>>>> ...). >>>>>> >>>>>> The purpose is also to be aligned with the other SDKs which provide >>>>>> beam-sdks-go and beam-sdks-python. >>>>>> >>>>>> Thoughts ? >>>>>> >>>>>> Regards >>>>>> JB >>>>>> -- >>>>>> Jean-Baptiste Onofré >>>>>> [email protected] >>>>>> http://blog.nanthrax.net >>>>>> Talend - http://www.talend.com >>>>>> >>>>>
