Oh, yikes. It seems like https://github.com/gradle/gradle/issues/847 indicates that the feature to use the default names in Gradle is practically nonfunctional. If that bug is as severe as it looks, I have to retract my position. Like we could never have sdks/java/core and sdks/py/core, right?
Kenn On Mon, Apr 1, 2019 at 6:27 PM Michael Luckey <[email protected]> wrote: > FWIW, hacked something as showcase for BEAM-4046 [1] > > This is miserably broken, but a > > ./gradlew projects > > or > > ./gradlew -p sdks/java build > > should work. Anything else is likely to cause issues. If u hit stack > overflow exception, it's likely caused by > https://github.com/gradle/gradle/issues/847 > > To continue here, lots of cleanup has to be done. We might also need to > rename folders etc, do better reflect semantic intentions. > > [1] https://github.com/apache/beam/pull/8194 > > On Mon, Apr 1, 2019 at 11:56 PM Kenneth Knowles <[email protected]> wrote: > >> >> >> On Mon, Apr 1, 2019 at 2:20 PM Lukasz Cwik <[email protected]> wrote: >> >>> >>> >>> On Mon, Apr 1, 2019 at 2:00 PM Kenneth Knowles <[email protected]> wrote: >>> >>>> >>>> As to building an aggregated "Java" project, I think the blocker will >>>> be supporting conflicting deps. For IOs like ElasticSearch and runners like >>>> Flink the conflict is essential and deliberate, to support multiple >>>> versions of other services. And that is not even talking about transitive >>>> dep conflicts. I think Python and Go don't have this issue simply because >>>> they haven't tackled those problems. >>>> >>>> Are you talking about just a shortcut for building (super easy to just >>>> add since we are using Gradle) or a new artifact that you want to >>>> distribute? >>>> >>>> On Mon, Apr 1, 2019 at 10:01 AM Lukasz Cwik <[email protected]> wrote: >>>> >>>>> During the gradle migration, we used to have something like: >>>>> >>>>> include(":sdks:java:core") >>>>> include(":sdks:java:extensions:sql") >>>>> include(":sdks:python") >>>>> >>>>> Just to be super clear, this is Gradle default and is equivalent to >>>> just leaving it blank. >>>> >>>> >>>>> but we discovered the Maven module names that were used during >>>>> publishing were "core" / "sql" / ... (effectively the directory name) >>>>> instead of "beam-sdks-java-core". >>>>> >>>> >>>> Isn't this managed by the publication plugin? >>>> https://docs.gradle.org/current/userguide/publishing_maven.html#sec:identity_values_in_the_generated_pom >>>> "overriding >>>> the default identity values is easy: simply specify the groupId, artifactId >>>> or version attributes when configuring the MavenPublication." >>>> >>> >>> During the gradle migration this wasn't that easy. The new maven publish >>> plugin improved a lot since then. >>> >>> >>>> Using the default at the time also broke the artifact names for intra >>>>> project dependencies that we generate[1]. Finally, we also ran into an >>>>> issue because we had more then one Gradle project with the same directory >>>>> name even though they were under a different parent folder (I think it was >>>>> "core") and that was leading to some strange build time behavior. >>>>> >>>> >>>> Weird. But I think the Jira should still stand as a move towards >>>> simplifying our build and making it more discoverable for new contributors. >>>> >>> >>> Agree on the JIRA makes sense, just calling out that there were other >>> issues that this naming had caused in the past which should be checked >>> before we call this done. >>> >> >> Totally agree. It will be quite a large task with a lot of boilerplate >> that might not be separable from technical blockers that come up as you go >> through the boilerplate. >> >> Kenn >> >> >> >> Kenn >>>> >>>> >>>>> We didn't migrate to a flat project structure where each project is a >>>>> folder underneath the root project because of the existing Maven build >>>>> rules that were being maintained in parallel and I'm not sure if people >>>>> would want to have a flat project structure either. >>>>> >>>>> 1: >>>>> https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1055 >>>>> >>>>> On Mon, Apr 1, 2019 at 9:49 AM Michael Luckey <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> although I did not yet manage to get deeper involved into actual >>>>>> development, I think this ability would be a useful addition. >>>>>> >>>>>> But I would also like to point out, that this is kind of implicit, as >>>>>> soon we get https://issues.apache.org/jira/browse/BEAM-4046 included. >>>>>> >>>>>> For instance, we would change the current setup from >>>>>> >>>>>> include "beam-sdks-java-core" >>>>>> project(":beam-sdks-java-core").dir = file("sdks/java/core") >>>>>> >>>>>> to something like >>>>>> >>>>>> include(":sdks:java:core") >>>>>> include(":sdks:java:extensions:sql") >>>>>> include(":sdks:python") >>>>>> >>>>>> >>>>>> With this in place a plain >>>>>> >>>>>> $ ./gradlew -p sdks/java build >>>>>> >>>>>> would exactly do what you want. And, of course, this will also work >>>>>> for 'sdks/java/io', 'runners/' etc. Hope, you get the point. >>>>>> >>>>>> Currently, we deviate from gradle default convention and therefore >>>>>> have to implement some quirks to restore default behaviour. And I somehow >>>>>> dislike the structure introduced by parent/child folders, which will be >>>>>> destroyed by our current project definitions. >>>>>> >>>>>> But, to be honest, although I have some clear understanding on how to >>>>>> proceed here - especially regarding the requirement to keep the change >>>>>> backwards compatible - we might decide not to switch. Because deeper >>>>>> investigation might reveal issues, which I am currently not aware of. >>>>>> >>>>>> Best, >>>>>> >>>>>> michel >>>>>> >>>>>> On Mon, Apr 1, 2019 at 5:52 PM Jean-Baptiste Onofré <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi guys, >>>>>>> >>>>>>> I would like to introduce a Gradle "meta" project for the build: >>>>>>> beam-sdks-java. >>>>>>> >>>>>>> The idea is to simply build all Java SDK related resources (core, >>>>>>> IO, ...). >>>>>>> >>>>>>> The purpose is also to be aligned with the other SDKs which provide >>>>>>> beam-sdks-go and beam-sdks-python. >>>>>>> >>>>>>> Thoughts ? >>>>>>> >>>>>>> Regards >>>>>>> JB >>>>>>> -- >>>>>>> Jean-Baptiste Onofré >>>>>>> [email protected] >>>>>>> http://blog.nanthrax.net >>>>>>> Talend - http://www.talend.com >>>>>>> >>>>>>
